Unix: Viewing your processes through the eyes of /proc

http://www.itworld.com/operating-systems/432024/unix-viewing-your-processes-through-eyes-proc

The /proc file system brings the processes on your Unix systems into view in some very useful ways, but only if you take the time to cd over to /proc and see all it can tell you.

The /proc virtual file system has been available on Unix systems for going on 20 years now.
I think it made its appearance on Solaris in 1996.
Providing access to information previously available only in the Unix kernel or through
a particular set of commands, /proc made it much easier to access and use information about the system and running processes. Even so, some aspects of /proc are easy to understand while others are something of a challenge to grasp.
The /proc file system is itself a virtual file system. The files are not "real" files like those
we're used to, associated with inodes and having space allocated to them on our disks.
If you look at the files in /proc, one of the first things you notice is that the files and the
directories all show 0 as their size.

$ cd /proc
$ ls -l
total 0
dr-xr-xr-x  6 root      root              0 Jul 28 13:39 1
dr-xr-xr-x  6 root      root              0 Jul 28 13:39 10
dr-xr-xr-x  6 root      root              0 Jul 28 13:39 1020
dr-xr-xr-x  6 root      root              0 Jul 28 13:39 11
dr-xr-xr-x  6 root      root              0 Jul 28 13:39 111
...
-r--------  1 root      root              0 Aug 17 14:02 vmcore
-r--r--r--  1 root      root              0 Aug 17 14:02 vmstat
-r--r--r--  1 root      root              0 Aug 17 14:02 zoneinfo

These "empty" directories and files provide views into our processes that can be very hard to derive through other means.
When you cd to /proc and list its content, you will notice that many of the directories
have names that are simply numbers. Each of these corresponds to a process currently running on your system. Counting the files with numeric names and the number of processes running, we should yield the same result or something very close (the commands you use will be in both lists).

$ cd /proc
$ ls | grep '[0-9]' | wc -l
164
$ ps -ef | wc -l
164

To see an example of what /proc knows about a process, pick out a process from your ps -ef output and then move to the corresponding /proc directory.

$ ps -ef | grep httpd
root      2278     1  0 Jul23 ?        00:00:01 /usr/sbin/httpd
apache    3770  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3771  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3772  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3773  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3774  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3775  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3776  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
apache    3777  2278  0 04:02 ?        00:00:00 /usr/sbin/httpd
$ ls /proc/2278
ls: cannot read symbolic link /proc/2278/cwd: Permission denied
ls: cannot read symbolic link /proc/2278/root: Permission denied
ls: cannot read symbolic link /proc/2278/exe: Permission denied
attr             cpuset   fd      loginuid  mountstats  schedstat  status
auxv             cwd      fdinfo  maps      oom_adj     smaps      task
cmdline          environ  io      mem       oom_score   stat       wchan
coredump_filter  exe      limits  mounts    root        statm

Using your personal credentials, you won't be able to look at all the files in /proc (as you can see from the display above) but, using sudo or switching to root, you can look at any file for any process.
The 2278 directory is just one of many, of course.

$ ls /proc
1      1607  199   2462  2844   31293  516        devices      mounts
10     17    2     2478  2899   31294  6          diskstats    mtrr
1020   1772  201   2479  29     31295  6428       dma          net
11     18    202   2486  2900   31298  6430       driver       partitions
111    1836  203   2491  2929   359    6486       execdomains  schedstat
112    1842  204   2497  2932   3770   6642       fb           scsi
113    1843  205   25    2933   3771   7          filesystems  self
114    1844  2154  2506  2934   3772   7730       fs           slabinfo
117    1845  2156  2559  2935   3773   7838       ide          stat
119    1863  2186  2580  2936   3774   7966       interrupts   swaps
12     1881  2189  2585  2938   3775   7970       iomem        sys
13     1882  2240  26    2961   3776   7972       ioports      sysrq-trigger
14     1883  2241  2602  2963   3777   8          irq          sysvipc
15     1888  2242  2615  3      4      8006       kallsyms     tty
15393  1895  2243  2631  31261  406    8008       kcore        uptime
1567   1896  2265  2647  31264  412    8009       keys         version
1568   1898  2278  27    31268  413    8059       key-users    vmcore
1570   1899  2312  2704  31271  414    9          kmsg         vmstat
1571   19    2313  2716  31272  415    acpi       loadavg      zoneinfo
1572   1907  2314  2719  31274  416    buddyinfo  locks
16     1927  2315  2746  31285  437    bus        mdstat
1601   1933  2361  2765  31287  458    cmdline    meminfo
1603   1934  2391  2775  31288  483    cpuinfo    misc
1605   198   2440  28    31289  5      crypto     modules

Let's say you want to look at what /proc can tell you about your login shell. The first thing you probably want to do is display your shell's process ID.

$ echo $$
8009

OK, so let's look at its representation in /proc. First, here's the directory:

$ cd /proc
$ ls -ld 8009
dr-xr-xr-x 6 shs staff   0 Aug 17 12:31 8009

If we move into the directory, we can see all of the files that provide some information on the process. Notice that most of these files provide read access to anyone on the system. Notice also that they all have the same creation date and time -- when you logged in.

$ cd 8009
$ ls -l
total 0
dr-xr-xr-x 2 shs staff   0 Aug 17 12:52 attr
-r-------- 1 shs staff   0 Aug 17 12:52 auxv
-r--r--r-- 1 shs staff   0 Aug 17 12:52 cmdline
-rw-r--r-- 1 shs staff   0 Aug 17 12:52 coredump_filter
-r--r--r-- 1 shs staff   0 Aug 17 12:52 cpuset
lrwxrwxrwx 1 shs staff   0 Aug 17 12:52 cwd -> /proc/8009
-r-------- 1 shs staff   0 Aug 17 12:52 environ
lrwxrwxrwx 1 shs staff   0 Aug 17 12:52 exe -> /bin/bash
dr-x------ 2 shs staff   0 Aug 17 12:52 fd
dr-x------ 2 shs staff   0 Aug 17 12:52 fdinfo
-r-------- 1 shs staff   0 Aug 17 12:52 io
-r--r--r-- 1 shs staff   0 Aug 17 12:52 limits
-rw-r--r-- 1 shs staff   0 Aug 17 12:52 loginuid
-r--r--r-- 1 shs staff   0 Aug 17 12:52 maps
-rw------- 1 shs staff   0 Aug 17 12:52 mem
-r--r--r-- 1 shs staff   0 Aug 17 12:52 mounts
-r-------- 1 shs staff   0 Aug 17 12:52 mountstats
-rw-r--r-- 1 shs staff   0 Aug 17 12:52 oom_adj
-r--r--r-- 1 shs staff   0 Aug 17 12:52 oom_score
lrwxrwxrwx 1 shs staff   0 Aug 17 12:52 root -> /
-r--r--r-- 1 shs staff   0 Aug 17 12:52 schedstat
-r--r--r-- 1 shs staff   0 Aug 17 12:52 smaps
-r--r--r-- 1 shs staff   0 Aug 17 12:52 stat
-r--r--r-- 1 shs staff   0 Aug 17 12:52 statm
-r--r--r-- 1 shs staff   0 Aug 17 12:52 status
dr-xr-xr-x 3 shs staff   0 Aug 17 12:52 task
-r--r--r-- 1 shs staff   0 Aug 17 12:52 wchan

Some of these files are easy to use. Others require a lot more effort. The stat file provides information about a process' status, but represents the information in this format:

$ cat stat
8009 (bash) S 8008 8009 8009 34816 9706 4194304 5516 51685 0 1 6 8 37 47 15 0 1 
0 1831964390 4898816 370 4294967295 134508544 135222164 3215953072 3215951
908 10765314 0 65536 3686404 1266761467 3225587569 0 0 17 2 0 0 0

The status file is much easier to use. It provides the same kind of information as stat but in a friendlier format.

$ cat status
Name:   bash
State:  S (sleeping)
SleepAVG:       98%
Tgid:   8009
Pid:    8009
PPid:   8008
TracerPid:      0
Uid:    263     263     263     263
Gid:    100     100     100     100
FDSize: 256
Groups: 100
VmPeak:     4784 kB
VmSize:     4784 kB
VmLck:         0 kB
VmHWM:      1476 kB
VmRSS:      1476 kB
VmData:      316 kB
VmStk:        88 kB
VmExe:       700 kB
VmLib:      1544 kB
VmPTE:        28 kB
StaBrk: 08851000 kB
Brk:    08893000 kB
StaStk: bfaf8cb0 kB
ExecLim:        080f6000
Threads:        1
SigQ:   0/32375
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000010000
SigIgn: 0000000000384004
SigCgt: 000000004b813efb
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed:   0000000f
Mems_allowed:   1

Some of this information is very straightforward -- the name of the process (bash), the process ID and parent process ID. You should be easily able to pick out the UID and GID.
It should come as no surprise that our process is sleeping. At any time, most processes are sleeping and, though we're obviously using the shell when we run this command, we're running another process within the shell. We can also see that our shell is sleeping 98% of the time.
The TracerPid variable set to 0 simply tells you that the process is not being traced.
A lot of the other variables -- those beginning with "Vm" -- relate to memory while those that start with "Sig" tell you how signals are being handled. Some may be blocked, others ignored, and still others caught. One thing to keep in mind while looking at all the 000000000038400 type values is that these are bit maps, but they are expressed in hexadecimal. So, every digit represents four bits in the overall value.
Take the SigBlk (blocked signals) value as an example. If set to 10000 sh shown above, being hex, that's 10000000000000000 in binary. Bit number 17 (counting from the right) is set and signal 17 is SIGCHLD. Thus, we can tell which signals are being blocked.
Our signals caught variable has a lot more bits set, but we can map them out if we're curious like so:

000000004b813efb ==> 0100 1011 1000 0001 0011 1110 1111 1011
                      |   | || |       |   || |||  |||| | ||
                      |   | || |       |   || |||  |||| | |+-  1 = SIGHUP
                      |   | || |       |   || |||  |||| | +--  2 = SIGINT
                      |   | || |       |   || |||  |||| +----  4 = SIGILL
                      |   | || |       |   || |||  |||+------  5 = SIGTRAP
                      |   | || |       |   || |||  ||+-------  6 = SIGABRT
                      |   | || |       |   || |||  |+--------  7 = SIGBUS
                      |   | || |       |   || |||  +---------  8 = SIGFPE
                      |   | || |       |   || ||+------------ 10 = SIGUSR1
                      |   | || |       |   || |+------------- 11 = SIGSEGV
                      |   | || |       |   || +-------------- 12 = SIGUSR2
                      |   | || |       |   |+---------------- 13 = SIGPIPE
                      |   | || |       |   +----------------- 14 = SIGALRM
                      |   | || |       +--------------------- 17 = SIGCHLD
                      |   | || +----------------------------- 24 = SIGXCPU
                      |   | |+------------------------------- 25 = SIGXFSZ
                      |   | +-------------------------------- 26 = SIGVTALRM
                      |   +---------------------------------- 28 = SIGWINCH
                      +-------------------------------------- 31 = SIGSYS

The FDSize variable may be set to 256, but we can see that the only file descriptors in use are 0, 1, 2 and 255. Just examine the contents of the fd directory.

$ ls fd
0  1  2  255

We can also look at the limits file to see what limits may be placed on our shell.

$ cat limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             50                   50                   processes
Max open files            1024                 1024                 files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       32375                32375                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

The environ file shows some details about our operational environment -- such as our search path.

$ cat environ
USER=shsLOGNAME=shsHOME=/home/staff  /shsPATH=/usr/local/bin:/bin:/usr/binMAIL=
/var/mail/shsSHELL=/bin/bashSSH_CLIENT=10.20.30.111 8506 22SSH_CONNECTION=10.20.
30.111 8506 192.168.0.12 22SSH_TTY=/dev/pts/0TERM=xterm

Another interesting file to look at is the io file. As you can see, it's reporting on characters read and written. Note the changes between the first and second running.

$ cat io
rchar: 1953940
wchar: 57247
syscr: 1791
syscw: 917
read_bytes: 8192
write_bytes: 8192
cancelled_write_bytes: 4096
$ cat io
rchar: 1955293
wchar: 57370
syscr: 1804
syscw: 921
read_bytes: 8192
write_bytes: 8192
cancelled_write_bytes: 4096

• rchar: the number of bytes the process read from files, pipes, etc.
• wchar: the number of bytes the process wrote
• syscr: the number of read-like system call invocations.
• syscr: the number of write-like system call invocations
• read_bytes: the number of bytes the read from disk
• write_bytes: number of bytes to be written to the disk
• cancelled_write_bytes: the number of bytes that the process caused to not happen (yeah, this one's hard to follow!)

The maps file shows regions in virtual memory that the process is using.

$ cat maps
00572000-0058d000 r-xp 00000000 68:02 4723528    /lib/ld-2.5.so
0058d000-0058e000 r-xp 0001a000 68:02 4723528    /lib/ld-2.5.so
0058e000-0058f000 rwxp 0001b000 68:02 4723528    /lib/ld-2.5.so
00591000-006e8000 r-xp 00000000 68:02 4723531    /lib/libc-2.5.so
006e8000-006ea000 r-xp 00157000 68:02 4723531    /lib/libc-2.5.so
006ea000-006eb000 rwxp 00159000 68:02 4723531    /lib/libc-2.5.so
006eb000-006ee000 rwxp 006eb000 00:00 0
006f0000-006f3000 r-xp 00000000 68:02 4723597    /lib/libdl-2.5.so
006f3000-006f4000 r-xp 00002000 68:02 4723597    /lib/libdl-2.5.so
006f4000-006f5000 rwxp 00003000 68:02 4723597    /lib/libdl-2.5.so
006f7000-006fa000 r-xp 00000000 68:02 4724408    /lib/libtermcap.so.2.0.8
006fa000-006fb000 rwxp 00002000 68:02 4724408    /lib/libtermcap.so.2.0.8
00a44000-00a45000 r-xp 00a44000 00:00 0          [vdso]
00c7e000-00c88000 r-xp 00000000 68:02 4723790    /lib/libnss_files-2.5.so
00c88000-00c89000 r-xp 00009000 68:02 4723790    /lib/libnss_files-2.5.so
00c89000-00c8a000 rwxp 0000a000 68:02 4723790    /lib/libnss_files-2.5.so
08047000-080f6000 r-xp 00000000 68:02 4756156    /bin/bash
080f6000-080fb000 rw-p 000ae000 68:02 4756156    /bin/bash
080fb000-08100000 rw-p 080fb000 00:00 0
08851000-08893000 rw-p 08851000 00:00 0          [heap]
b7d28000-b7f28000 r--p 00000000 68:07 5052872    /usr/lib/locale/locale-archive
b7f28000-b7f2a000 rw-p b7f28000 00:00 0
b7f30000-b7f32000 rw-p b7f30000 00:00 0
b7f32000-b7f39000 r--s 00000000 68:07 5116679    /usr/lib/gconv/gconv-modules.cache
bfae4000-bfaf9000 rw-p bffe9000 00:00 0          [stack]

There's also an uptime file, though you're probably never going to choose it over the uptime command.

$ cat uptime
18321661.93 1308472.21
$ uptime
 13:04:14 up 212 days,  1:21,  1 user,  load average: 0.00, 0.00, 0.00

The loadavg file, on the other hand, is easy to digest. Yes, the system this was run on is not a very busy one!

$ cat loadavg
0.00 0.00 0.00 1/175 8336

The /proc file system has a lot to tell you, but it's probably going to bemost useful when you use it fairly frequently and get used to what your processes look like from the kernel's point of view.