I prefer looking at output of "sar" which shows you nicely, in 10-minute increments, how idle the system was today (and I think usually this data captured is rotated daily for a month), and gives you also a good idea of whether you have processes waiting excessively for IO.
It also has a bunch of other options.
On my own system, I also generally run "ps auwx" every 15 minutes, together with a scan of what queries Postgres servers are doing and dump of web request activity (read last 10k lines from access logs, find out how long ago the first request was to determine rough hits per second and ares of application they hit). That way when someone says "hey, the system was slow around this time" I can go back and find out that some cron job had a dozen processes taking up tons of memory or blocking on IO.
Some of those statistics also go into some RRD-based system which makes it easier to follow e.g. number of users logged in or number of Apache children based on weekday/time of day.
I'm a big fan of 'sar' as well. It's nice that it can also show you i/o wait for the sampled period. I also love vmstat, as you can use it to see everything that is happening with the system sampled every second if you like. The first two columns will show you the number of processes in the run queue as well as number of processes blocked on i/o.
We must be running different versions of sar then, as "sar" by itself here (RHEL 5) shows information about the time split between user/system/waitIO/idle -- that certainly does not come from /proc/loadavg.
If you run "sar -q" you could get the load average information, but that's not particularly useful, as you can't see whether the 20 load avg an hour ago was caused by heavy disk IO or a dozen CPU bound processes.
Nope, we're likely running the same version. The particular info you mentioned comes from /proc/stat (you're right that it's a different file), but again it's the same sources as top:
# lsb_release -d
Description: Red Hat Enterprise Linux Server release 5.3 (Tikanga)
# rpm -qf $(which sar)
sysstat-7.0.2-3.el5
# strace /usr/lib64/sa/sa1 1 1 &> results
# grep open results
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib64/libtermcap.so.2", O_RDONLY) = 3
open("/lib64/libdl.so.2", O_RDONLY) = 3
open("/lib64/libc.so.6", O_RDONLY) = 3
open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
open("/proc/meminfo", O_RDONLY) = 3
open("/usr/lib64/sa/sa1", O_RDONLY) = 3
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib64/libc.so.6", O_RDONLY) = 3
open("/etc/localtime", O_RDONLY) = 3
open("/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
open("/proc/tty/driver/serial", O_RDONLY) = 3
open("/proc/interrupts", O_RDONLY) = 3
open("/proc/net/dev", O_RDONLY) = 3
open("/proc/diskstats", O_RDONLY) = 3
open("/var/log/sa/sa06", O_RDWR|O_APPEND) = 3
open("/proc/stat", O_RDONLY) = 4
open("/proc/meminfo", O_RDONLY) = 4
open("/proc/loadavg", O_RDONLY) = 4
open("/proc/vmstat", O_RDONLY) = 4
open("/proc/sys/fs/dentry-state", O_RDONLY) = 4
open("/proc/sys/fs/file-nr", O_RDONLY) = 4
open("/proc/sys/fs/inode-state", O_RDONLY) = 4
open("/proc/sys/fs/super-max", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/sys/fs/dquot-max", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/sys/kernel/rtsig-max", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/net/sockstat", O_RDONLY) = 4
open("/proc/net/rpc/nfs", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/net/rpc/nfsd", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/diskstats", O_RDONLY) = 4
open("/proc/tty/driver/serial", O_RDONLY) = 4
open("/proc/interrupts", O_RDONLY) = 4
open("/proc/net/dev", O_RDONLY) = 4
# strace top -n -b 1 &> results
# grep open results
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib64/libproc-3.2.7.so", O_RDONLY) = 3
open("/usr/lib64/libncurses.so.5", O_RDONLY) = 3
open("/lib64/libc.so.6", O_RDONLY) = 3
open("/lib64/libdl.so.2", O_RDONLY) = 3
open("/proc/stat", O_RDONLY) = 3
open("/proc/sys/kernel/pid_max", O_RDONLY) = 3
open("/etc/toprc", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/root/.toprc", O_RDONLY) = -1 ENOENT (No such file or directory)
If you don't already know about it, dstat is a good one too, and lightweight enough that it can be run on heavily loaded production systems w/out as much "top problem" going on.
It also has a bunch of other options.
On my own system, I also generally run "ps auwx" every 15 minutes, together with a scan of what queries Postgres servers are doing and dump of web request activity (read last 10k lines from access logs, find out how long ago the first request was to determine rough hits per second and ares of application they hit). That way when someone says "hey, the system was slow around this time" I can go back and find out that some cron job had a dozen processes taking up tons of memory or blocking on IO.
Some of those statistics also go into some RRD-based system which makes it easier to follow e.g. number of users logged in or number of Apache children based on weekday/time of day.