Pushmi-pullyu

It turns out that the infrastructure supporting our push process is also very handy for pulling data to a central location. The next couple posts are going to talk about a reporting process I've built for monitoring general utilization of boxes in the environment. Central to the process is the ability to reach out to all of the boxes and run commands, piping the data from those commands back to the central server. For the reports I'm creating, that data is the output of sar (the System Activity Reporter).

The raw cpu data looks something like this:

SunOS hostname.domain.com 5.9 Generic_122300-12 sun4u    04/11/2008
00:00:00    %usr    %sys    %wio   %idle
01:40:00       0       2      24      74
01:45:00       0       2      24      74
01:50:00       0       2      24      74
01:55:00       0       2      24      74
02:00:00       0       2      24      74
02:05:00       1       1      26      72
02:10:00       0       1      25      74
02:15:00       0       2      24      74
02:20:00       0       2      24      74
02:25:00       0       2      24      74
02:30:00       0       2      24      74

Raw memory data looks something like this:

SunOS hostname.domain.com 5.9 Generic_122300-12 sun4u    04/11/2008
00:00:00 freemem freeswap
01:40:00  948072 15057081
01:45:00  948002 15055725
01:50:00  948028 15056531
01:55:00  948021 15056523
02:00:00  947853 15052417
02:05:00  947898 15052517
02:10:00  947826 15051640
02:15:00  947744 15049178
02:20:00  947905 15052163
02:25:00  947799 15050548
02:30:00  947914 15053483

That is only moderately useful, though. Far more pleasant to view are some simple graphs like these:

The next few posts will show you how to go from cpu percentages and counts of free memory pages to graphs of those values. They'll also talk about how doing that can (and does) regularly update a reporting website so you can provide up to date utilization graphs. Most important, all of this complies with my general philosophy -- automate everything that you have to do repeatedly.