http://linux-server-security.com/linux_servers_howtos/linux_process_table.html
A fundamental design feature of Unix-like Operating Systems is that many of a system’s resources are accessible via the filesystem, as a file. For example the “procfs” pseudo-filesystem offers us access to all kinds of valuable treasures.
By querying files present in “/proc” you can quickly find out the intricacies of your network connections, the server’s CPU vendor and look up a mountain of useful information such as the command line parameters passed to your latest-and-greatest application when it fired up. This is thanks to the fact that many of the functions of a server, such as a network connection are really just another stream of data and in most cases can be presented as a file on your filesystem.
Let’s jump ahead of ourselves for a moment in case you’re not too familiar with the “/proc” filesystem. If you knew that your Process ID was “16651” for example then you could run this command in order to find out what was sent to the Puppet process to start it up:
# xargs -0 < /proc/16551/cmdline
The output from that command is:
/usr/bin/ruby /usr/bin/puppet agent
As you can see Puppet’s agent is using the Ruby programming language in this case and the binary “/usr/bin/puppet” is passed the parameter “agent” to run it as an “agent” and not a “master”.
The “Everything Is A File” philosophy makes total sense if you think about it. The power harnessed within Unix’s standard tools (usually used for manipulating data held in the more common text files) such as “grep”, “awk” and “sed” are a major strength of the Operating System. But most importantly you can have a system’s components integrate very efficiently and easily if many things are simply files.
If you have you ever tried to look into a process running on a Unix-like machine then you’ll know that if anything the abundance of information adds confusion, rather than assists, if you don’t know where to look. There are all sorts of things to consider when you are eagerly trying to track down a rogue process on production machine.
In this chapter I will attempt to offer a broad insight into how the Process Table can be accessed by the ps command and in combination with “/proc” and “/dev” how it can help you manipulate your systems.
Legacy
There are a few legacy stumbling blocks when it comes to looking up a process on different types of Unix boxes but thankfully we can rely on the trusty "ps" command to mitigate some of these headaches automatically for us.
For example, Unix used the "ps" command by grouping its parameters together and prepending a hyphen. Whereas BSD enjoyed grouping switches together but, for one reason or another, fell out with the hyphen entirely.
Throwing another spanner in the works however was good, old GNU's preference; where its long options used two dashes. Now that you've fully committed those confusing differences to memory let's assume that the ps command does as much as it can for us by mixing and matching the aforementioned options in an attempt to keep everyone happy.
Be warned that occasionally sometimes oddities can occur so keep an eye out for them just in case. I’ll try and offer alternative commands as we continue to act as a reminder that not all is to be taken exactly as read. For example a very common use of the ps command is:
# ps -aux
It should be noted however that this is indeed different to:
# ps aux
You might suspect, and would be forgiven for thinking as much, that this is purely to keep everyone on their toes. However according to the ps command’s manual this is apparently because POSIX and UNIX insist that they should cater for processes owned by a user called "x". However, if I’m reading the information correctly, then should the user "x" not exist then "ps aux" is run. I love the last sentence of the manual’s definition and draw your attention to it as a gentle warning: “It is fragile, subject to change, and thus should not be relied upon.”
Process Tree
Enough eye-strain for a moment and let’s begin by looking at the ps command and what it is capable of helping with in relation to querying the Process Table.
For starters (and I won’t be betting the ranch on this statement) it's relatively safe to assume that upper and lower case mean the same thing.
If you’ve never seen the output of a Process Tree then it might help with understanding “child” threads which live under a “parent” process. In this case the command is simply:
# pstree
The not-so-impossible-to-decipher output from that command is shown in Listing One (on a server not running “systemd” but good, old “init” (which is always Process ID (PID) number one, as an aside):
init-+-auditd---{auditd}
|-certmonger
|-crond
|-dbus-daemon---{dbus-daemon}
|-hald---hald-runner-+-hald-addon-acpi
| `-hald-addon-inpu
|-apache2---8*[apache2]
|-master-+-pickup
| `-qmgr
|-6*[mingetty]
|-oddjobd
|-rpcbind
|-rsyslogd---3*[{rsyslogd}]
|-sshd---sshd---sshd---bash---pstree
|-udevd---2*[udevd]
Listing One: Output from the “pstree” command showing parent processes and their children
You can make your screen output much messier by adding the “-a” switch. Doing so will add command line arguments (pulled from the /proc filesystem in the same way that our example did earlier). This is very useful but you might want to do something like “grep” a specific process name from the output, as follows:
# pstree -ap | grep ssh
|-sshd,29076
| `-sshd,32365
| `-sshd,32370
| |-grep,8401 ssh
| |-sssd_ssh,1143 --debug-to-files
Listing Two: The command “pstree” showing only SSH processes with command line arguments and PIDs
As we can see from Listing Two, the command which we are querying with is also shown (starting with “grep”) in the output so try not to let that trip you up on your travels. I’ve added the “-p” switch to display the PIDs too for ease.
One final look at that example is seen in Listing Three. Here the all-pervasive “-Z” switch offers us any SELinux config associated with the parent and child detail displayed in our process table tree. That command for reference was:
# pstree -aZp | grep ssh
|-sshd,29076,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'
| `-sshd,32365,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'
| `-sshd,32370,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023'
| |-grep,8406,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023' ssh
| |-sssd_ssh,1143,`system_u:system_r:sssd_t:s0' --debug-to-files
Listing Three: The output now includes SELinux detail, command line arguments and PIDs
Alternative View
Back to school for a moment and we’ll pick up some of the basics that will help us increase our knowledge later. We will (very loosely) mimic the Process Tree output we saw earlier with just the ps command now. You might stumble across “forest” puns in manuals with regards to “trees” in case it’s confusing.
The “-e” switch is also equal to “-A” and it will dutifully display ALL of the processes. We’re going to combine that with “-j” which should give us a “jobs format”. I’m sorry to potentially give you more headaches and I encourage to try a few of these of alternative options but one small caveat would be that running “j” and “-j” gives different levels of information. The non-hyphenated version giving more output in this case.
We will also add the “-H” to display the hierarchical output and we can achieve this by running:
# ps -ejH
Or, in a parallel universe, alternatively you can also use for vaguely similar results:
# ps axjf
Try it yourself. It’s far from identical to the “pstree” command but there are similarities in the output due to the hierarchies I’m sure that you will agree.
Day-To-Day
It is easy to get overwhelmed by the level of detail that the ps command can provide. My favourite process command is:
# ps -ef
With “-e” this gives us a display of all the processes in the Process Table and the newly added “-f” switch gives us what’s known as full-format listing. Apparently we can add “-L” to this concoction in order to offer us process thread information. I find this command very easy to remember around Christmas time.
# ps -eLf
Et voilà, as requested two new columns NLWP (number of threads) and LWP (thread ID) are now added to the now busying display.
If you only wanted to print the name of a process (which might be ideal for scripting) then you can sort the wheat from the chaff by using this command for process number "37":
# ps -p 37 -o comm=
An option that I've used in the past a few times is checking what command my terminal is currently
responsible for. This is useful if you're stressing your server or worrying your workstation with lots of extra load. The simple (non-hyphenated) "T" switch lets us view this.
# ps T
You can test this by running something unusual, like a “for” loop with a pause in it using“sleep”, or anything odd that stands out such as this command.
# echo {1..999999} &
This simply runs a counter process in the background and when we run “ps T” we can see processes associated with the terminal in question.
Built-In Features
Let’s look at a few other built-in basics of the pliable ps command.
You can reverse the output of a command with a non-hyphenated “N” switch, which stands for negate. Or more precisely from the manual “Select all processes except those that fulfill the specified conditions. (negates the selection) Identical to --deselect.” All is revealed in Listing Four. As we can see, there isn’t any mention of “ntp” except from our ps command.
# ps N ntp
PID TTY STAT TIME COMMAND
1414 tty1 Ss+ 0:00 /sbin/mingetty /dev/tty1
1416 tty2 Ss+ 0:00 /sbin/mingetty /dev/tty2
1418 tty3 Ss+ 0:00 /sbin/mingetty /dev/tty3
1420 tty4 Ss+ 0:00 /sbin/mingetty /dev/tty4
1426 tty5 Ss+ 0:00 /sbin/mingetty /dev/tty5
1430 tty6 Ss+ 0:00 /sbin/mingetty /dev/tty6
9896 pts/1 S 0:00 sudo -i
9899 pts/1 S 0:00 -bash
10040 pts/1 R+ 0:00 ps N ntp
Listing Four: Demonstrates running processes without including “ntp” (note our “ps” command again being seen)
Imagine that you again wanted to see what SSH activity was going on and also the Hardware Abstraction Layer daemon, “hald”. Hardly related I agree but you can never account for strange scenarios when it comes to computers.
The command which follows is a way of searching for a list of processes with a certain name. Separated without a space and just a comma in this case.
# ps -C sshd,hald
If you need to check any processes run by a particular system group you can achieve this with the
following command:
# ps -G ntp
Compatability
Although not always the case the modern ps command cleverly mitigates our migraine-inducing compatibility headaches by letting us run a simple command in a number of ways where possible.
If you wanted to select a list of processes which belonged to the superuser, "root", then you can achieve this with the following three commands (which admittedly display ever-so-slightly different outputs):
# ps -u root
# ps U root
# ps --user root
The above commands dutifully offer us what's known as the "EUID" or “effective ID” of a user but
not the "real user ID". In reality, no pun intended, every process actually has two user IDs; just to keep things simple. This also applies to groups too but let's not worry about that.
Apparently the kernel is most concerned with the "effective user ID" for activities such as writing to a file and if a user is allowed to complete a request to do something that requires a privilege.
And, although that’s required for much of the time, there's an important scenario within which the "real user ID" needs to be paid attention to. If someone or something wants to alter the owner of "effective user ID" on an already running process then the kernel needs to check up on both the "real user ID" and the "effective user ID".
Changing the ownership of a process is particularly useful if a new user wants to do essentially the same thing (like write to a file) as the existing owner does. Rather than duplicating the process (adding extra strain to the system and potentially introducing more security aspects to consider) we can avoid duplicating the process and simply reassign it.
What about after a user is finished with their short task? The answer is that we only temporarily give access and then swap it back again to its original owner. If you want to reveal the somewhat elusive "real user IDs" then you can do so with system groups like this (which is the same as "-G"):
# ps --Group ntp
And, not surprisingly, we can do exactly the same thing for users as follows (we do this with "-U"):
# ps --User chrisbinnie
If you want to query a very specific Process ID (because you've spotted it in "top" or a script
has complained about it) then to all intents and purposes these commands all the do the same. I have included some output for comparison as it’s short and easy to read in this case:
# ps -p 1248
PID TTY TIME CMD
1248 ? 00:00:08 ntpd
# ps p 1248
PID TTY STAT TIME COMMAND
1248 ? Ss 0:08 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
# ps --pid 1248
PID TTY TIME CMD
1248 ? 00:00:08 ntpd
If you’ve ever wondered about the efficiencies of a system, here’s something interesting. The kernel has to be able to tidy up when a user logs out (otherwise there would be a plethora of useless processes clogging up the pipes) so Unix-like systems dutifully groups processes into “sessions”. You can test for session IDs by using "--sid" or as below with “-s”:
# ps -s 1526
If it’s of interest a session can have an associated terminal (of "tty" or "Teletype" varieties) controlling it however only one process can be at running in the foreground. All these components are given numbers to keep the system nice and orderly. A a result we have thread IDs, process IDs, process group IDs and session IDs. Here was you thinking that the ps command didn't have much to do.
If you’re interested in reading a little more about sessions this book’s excerpt is intriguing with sentences such as “Consequently, sessions are largely the business of shells. In fact, nothing else really cares about them.”:
Children
Now picture the scene where we want to check for the parents of a process (we'll look at this in more detail in a second). You can achieve this by using this command:
# ps --ppid 21201
This shows us the processes with a parent process of that ID. In other words we can pinpoint processes that are children of process "21201"in this case.
Having said earlier that usually case-sensitivity shouldn't cause too many headaches I'm going to completely contradict myself with a few examples of why that statement isn’t always true.
Try running my favourite ps command again, its abbreviated output is shown underneath:
# ps -ef
UID PID PPID C STIME TTY TIME CMD
apache 23026 22856 0 Feb26 ? 00:00:00 /usr/sbin/apache2
Now try running the full fat version by using an uppercase "F":
# ps -eF
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
apache 23026 22856 0 44482 3116 0 Feb26 ? 00:00:00 /usr/sbin/apache2
The differences are that the latter includes SZ, RSS and PSR fields. The first two are memory related whereas PSR shows which CPU the process is using. For more information there’s lots more in the manual:
# man ps
Onwards we can look at another alternative to the “-Z” option which we briefly touched on before:
# ps -efM
unconfined_u:system_r:apache2_t:s0 apache 23031 22856 0 Feb26 ? 00:00:00 /usr/sbin/apache2
A useful BSD throwback I quite like the look of is possibly one of the shortest commands known to
mankind. Have a look at Listing Five.
# ps l
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 0 1414 1 20 0 4064 584 n_tty_ Ss+ tty1 0:00 /sbin/mingetty /dev/tty1
4 0 1416 1 20 0 4064 588 n_tty_ Ss+ tty2 0:00 /sbin/mingetty /dev/tty2
4 0 1418 1 20 0 4064 584 n_tty_ Ss+ tty3 0:00 /sbin/mingetty /dev/tty3
4 0 1420 1 20 0 4064 580 n_tty_ Ss+ tty4 0:00 /sbin/mingetty /dev/tty4
4 0 1426 1 20 0 4064 584 n_tty_ Ss+ tty5 0:00 /sbin/mingetty /dev/tty5
4 0 1430 1 20 0 4064 588 n_tty_ Ss+ tty6 0:00 /sbin/mingetty /dev/tty6
4 0 9896 9558 20 0 191392 2740 poll_s S pts/1 0:00 sudo -i
4 0 9899 9896 20 0 110496 1960 wait S pts/1 0:00 -bash
4 0 10776 9899 20 0 108104 980 - R+ pts/1 0:00 ps l
Listing Five:Shows us the “long formatted” output which can be embellished with other options, harking from BSD origins
Clarity
Sometimes even the mighty ps command struggles to precisely refine its output. Imagine a scenario where Java processes are filling up the process table and all you want to do is find their parent so that you can stop (or "kill") the process abruptly. To summarise your information you can use the non-hyphenated “S” switch:
# ps S
This helps to find a parent when its child processes only live for a short period of time.
What about when your Process Table is brimming with processes and you need to list a number of process PIDs at once? As you’d expect there are different ways to achieve this as Listing Six shows when we run the following command:
# ps -p "1 2" -p 3,4
PID TTY TIME CMD
1 ? 00:00:03 init
2 ? 00:00:01 kthreadd
3 ? 00:00:01 migration/0
4 ? 00:00:20 ksoftirqd/0
Listing Six: We can pick and choose the PIDs that we view in a number of ways
Everything Is A File
As we mentioned earlier the well-considered principle of basing as much as possible on Unix-like systems around files is a well-advised approach. It could be said that this also extends to the Process Table; consider the treasure trove of gems to be found if you delve deeply into the “procfs” pseudo-filesystem, located in root level “/proc” on your filesystem.
Elements of the innards of /proc can only be read from and not written to. The key file here is
“/etc/sysctl.conf” where you can also change many tunable kernel settings so that they persist following a reboot. One not-so-trivial caveat is that, almost magically, any freshly entered parameters into “/proc” are usually set live instantly so be careful!
Clearly this has a number of advantages. There's no messing about with stopping and starting daemons but be warned that if you're in the slightest bit unsure of making a change (especially to servers) then take a deep breath before doing so. Rest assured that a reboot will revert any changes that you make if they are not entered into the file “/etc/sysctl.conf”.
There are zillions of hidden corridors and secret rooms to explore inside /proc and sadly we will only be able to look at a tiny percentage of them. Needless to say that on a test Virtual Machine or development machine you should spend a long time tweaking, fiddling and breaking your current kernel’s procfs settings. If you’re like me then you might even find it vaguely cathartic and it’s safe to say that the immediacy of the changes really appeal to the impatient.
You can look further into a particular process which you've found using the excellent ps command as we've already seen. The path of Process ID “23022” for example is simply “/proc/23022” in relation to /proc.
If we enter that directory then we are shown (after some complaints that we don't have access to parts of the directory structure if we’re not logged in as root) the contents is presented as follows in Listing Seven:
dr-xr-xr-x. 8 apache apache 0 Feb 26 03:15 .
dr-xr-xr-x. 144 root root 0 Feb 11 13:31 ..
dr-xr-xr-x. 2 apache apache 0 Feb 26 04:03 attr
-rw-r--r--. 1 root root 0 Feb 28 08:25 autogroup
-r--------. 1 root root 0 Feb 28 08:25 auxv
-r--r--r--. 1 root root 0 Feb 28 08:25 cgroup
--w-------. 1 root root 0 Feb 28 08:25 clear_refs
-r--r--r--. 1 root root 0 Feb 26 04:03 cmdline
-rw-r--r--. 1 root root 0 Feb 28 08:25 comm
-rw-r--r--. 1 root root 0 Feb 28 08:25 coredump_filter
-r--r--r--. 1 root root 0 Feb 28 08:25 cpuset
lrwxrwxrwx. 1 root root 0 Feb 28 08:25 cwd -> /
-r--------. 1 root root 0 Feb 27 14:01 environ
lrwxrwxrwx. 1 root root 0 Feb 28 08:25 exe -> /usr/sbin/apache2
dr-x------. 2 root root 0 Feb 26 04:03 fd
dr-x------. 2 root root 0 Feb 28 08:25 fdinfo
-r--------. 1 root root 0 Feb 28 08:25 io
-rw-------. 1 root root 0 Feb 28 08:25 limits
-rw-r--r--. 1 root root 0 Feb 28 08:25 loginuid
-r--r--r--. 1 root root 0 Feb 28 08:25 maps
-rw-------. 1 root root 0 Feb 28 08:25 mem
-r--r--r--. 1 root root 0 Feb 28 08:25 mountinfo
-r--r--r--. 1 root root 0 Feb 28 08:25 mounts
-r--------. 1 root root 0 Feb 28 08:25 mountstats
dr-xr-xr-x. 4 apache apache 0 Feb 28 08:25 net
dr-x--x--x. 2 root root 0 Feb 28 08:25 ns
-r--r--r--. 1 root root 0 Feb 28 08:25 numa_maps
-rw-r--r--. 1 root root 0 Feb 28 08:25 oom_adj
-r--r--r--. 1 root root 0 Feb 28 08:25 oom_score
-rw-r--r--. 1 root root 0 Feb 28 08:25 oom_score_adj
-r--r--r--. 1 root root 0 Feb 28 08:25 pagemap
-r--r--r--. 1 root root 0 Feb 28 08:25 personality
lrwxrwxrwx. 1 root root 0 Feb 28 08:25 root -> /
-rw-r--r--. 1 root root 0 Feb 28 08:25 sched
-r--r--r--. 1 root root 0 Feb 28 08:25 schedstat
-r--r--r--. 1 root root 0 Feb 28 08:25 sessionid
-r--r--r--. 1 root root 0 Feb 28 07:52 smaps
-r--r--r--. 1 root root 0 Feb 28 08:25 stack
-r--r--r--. 1 root root 0 Feb 26 03:15 stat
-r--r--r--. 1 root root 0 Feb 26 03:15 statm
-r--r--r--. 1 root root 0 Feb 26 04:03 status
-r--r--r--. 1 root root 0 Feb 28 08:25 syscall
dr-xr-xr-x. 3 apache apache 0 Feb 27 11:41 task
-r--r--r--. 1 root root 0 Feb 28 08:25 wchan
Listing Seven: Inside “/proc/23022” we can see a number of pseudo files and directories for our Web server
You might want to think of this content as belonging to runtime system information. It has been said that /proc is a centralised config system for the kernel and it’s easy to see that directory is full of a mountain of information for just one process. As suggested rummaging through these directories and looking up which file does what when you have a spare moment might be described as therapeutic. It’s well worth the effort.
Pseudo Filesystems
It’s hard to dismiss the power which /proc wields. Be wary though that there’s a lot going on inside your server when it is running, even if no-one is hitting your website. As a result wouldn’t it be sensible to separate the tricksy hardware settings aside from the kernel settings and Process Table?
Continuing onwards with our “Everything Is A File” mantra that’s exactly what Unix-type Operating Systems do. Step forward “/dev”.
When dealing with physical devices, whether connected to the machine or not, we turn to “/dev” and not “/proc”.
Here’s an abbreviated directory listing of “/dev” as shown in Listing Eight.
drwxr-xr-x. 2 root root 740 Feb 11 13:31 block
drwxr-xr-x. 2 root root 80 Feb 11 13:31 bsg
lrwxrwxrwx. 1 root root 3 Feb 11 13:31 cdrom -> sr0
lrwxrwxrwx. 1 root root 3 Feb 11 13:31 cdrw -> sr0
drwxr-xr-x. 2 root root 2.5K Feb 11 13:31 char
crw-------. 1 root root 5,1 Feb 11 13:31 console
lrwxrwxrwx. 1 root root 11 Feb 11 13:31 core -> /proc/kcore
drwxr-xr-x. 4 root root 80 Feb 11 13:31 cpu
crw-rw----. 1 root root 10, 61 Feb 11 13:31 cpu_dma_latency
crw-rw----. 1 root root 10, 62 Feb 11 13:31 crash
drwxr-xr-x. 5 root root 100 Feb 11 13:31 disk
Listing Eight: We can see an abbreviated list of some of the devices which “/dev” deals with
What about another example of what “/dev” can do for us? Lets take an example of the superb “lsof” utility. If you're not familiar with “lsof” then it's unquestionably worth a look at, I’m a big fan. The abbreviation “lsof” stands for “list open files” and its seemingly endless functionality is exceptionally useful.
Listing Nine shows us output from “lsof” when looking up information relating to the directory “/var/log“. We can display this information having run this command:
# lsof +D /var/log/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1103 root 1w REG 253,4 2743 19 /var/log/messages
rsyslogd 1103 root 2w REG 253,4 1906 17 /var/log/cron
rsyslogd 1103 root 4w REG 253,4 747 18 /var/log/maillog
rsyslogd 1103 root 5w REG 253,4 1753 27 /var/log/secure
apache2 22856 root 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 22856 root 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23022 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23022 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23024 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23024 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23026 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23026 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23027 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23027 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23028 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23028 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23029 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23029 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23030 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23030 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23031 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23031 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
Listing Nine: The output from the mighty “lsof” looks much like that from the ps command
I am using this “lsof” example because it highlights how a system weaves in and out referencing data from “/proc” and “/dev”. I won’t pretend to understand the nuances.
From its manual the versatile “lsof” transparently informs us of how it gathered such information about that directory, by telling us which files it references:
“/dev/kmem” - the kernel virtual memory device
“/dev/mem” - the physical memory device
“/dev/swap” - the system paging device
From what I can gather these files change between varying Unix versions but hopefully they at least give you a taste of which file is responsible for which task.
As we can see “/dev and “/proc” are useful for all sorts of things whether they be network related, devices (real or virtual), disks (loop disks and physical drives) and much more.
Counting Processes
Now we’ve whet our whistles with a quick look at the Process Table and pseudo filesystems, “/dev” and “/proc”, let’s explore a few (hopefully useful) unrelated command lines which may save the day at some point in the future.
You might be concerned that a single member of a cluster of identical machines is beginning to give up the ghost. You could check how many processes the system has created since its last reboot by using this command:
# grep processes /proc/stat
Shovel at the ready now look inside the file “/proc/$PID/cmdline” (replacing “$PID” for your Process ID) and there you will find some buried goodies. This file retains the entire and complete command line for a process (almost always at least, zombie processes being one of the exceptions so please don’t fret if your mileage varies).
Sitting alongside that file, in the PID’s directory, is the “cwd” symbolic link (or symlink) to the current working directory of that process.You may need to be logged in as “root”.
In order to discover the current-working-directory of that process run this command:
# cd /proc/$PID/cwd; /bin/pwd
We probably need to be “root” sometimes because the symlink is hidden to normal users. I won't claim to understand all of the intricacies of these pseudo files but if you use the commands “cat” or “less” to view some of these files usually then a little more light is shed on their raison d’ etre.
One such pseudo file (which is less mysterious thanks to its name) is the disk input/output statistics file, named “io”. By running this command and seeing its output in Listing Ten:
# cat io
rchar: 0
wchar: 0
syscr: 0
syscw: 0
read_bytes: 342
write_bytes: 11
cancelled_write_bytes: 0
Listing Ten: Here we can see what this process has been up to in relation to disk activities
Amongst many others there's another useful addition which is the "maps" file. In Listing Eleven we can see the memory regions and their access permissions for a process by using:
# cat maps
7eff839c7000-7eff839de000 r-xp 00000000 fd:01 3221 /lib64/libpthread-2.12.so
7eff839de000-7eff83bde000 ---p 00017000 fd:01 3221 /lib64/libpthread-2.12.so
7eff83bde000-7eff83bdf000 r--p 00017000 fd:01 3221 /lib64/libpthread-2.12.so
7eff83bdf000-7eff83be0000 rw-p 00018000 fd:01 3221 /lib64/libpthread-2.12.so
7eff843ac000-7eff843b3000 r-xp 00000000 fd:01 3201 /lib64/libcrypt-2.12.so
7eff843b3000-7eff845b3000 ---p 00007000 fd:01 3201 /lib64/libcrypt-2.12.so
7eff845b3000-7eff845b4000 r--p 00007000 fd:01 3201 /lib64/libcrypt-2.12.so
7eff845b4000-7eff845b5000 rw-p 00008000 fd:01 3201 /lib64/libcrypt-2.12.so
7eff82fb4000-7eff83025000 r-xp 00000000 fd:01 3478 /lib64/libfreebl3.so
7eff83025000-7eff83224000 ---p 00071000 fd:01 3478 /lib64/libfreebl3.so
7eff83224000-7eff83226000 r--p 00070000 fd:01 3478 /lib64/libfreebl3.so
7eff83226000-7eff83227000 rw-p 00072000 fd:01 3478 /lib64/libfreebl3.so
Listing Eleven: An extremely abbreviated sample of the “maps” file for a process
Apparently the legend for the permissions is as follows:
r = read
w = write
x = execute
s = shared
p = private (copy on write)
The "maps" file can be useful to see how a process is interacting with the system’s files. Or maybe you’re curious as to which libraries a process needs and you have forgotten the correct options to add to a tool like the super-duper “lsof”.
Any eagle-eyed console champions will spot that the file sizes, having used "ls" to list the
files in the directory, all appear as zero bytes. Clearly these pseudo files are different animals than we’re used to getting our fingers nipped by.
Kernel Support
Let's now move onto other benefits of /proc and not just on a per-process basis.
For example you might look at the filesystems which were compiled into the kernel by checking out "/proc/filesystems".
In Listing Twelve you can see (an abbreviated list) of what filesystems our kernel supports and even as standard it’s a sizeable list.
nodev sysfs
nodev rootfs
nodev bdev
nodev proc
nodev cgroup
nodev cpuset
nodev tmpfs
nodev devtmpfs
nodev binfmt_misc
nodev debugfs
nodev securityfs
nodev sockfs
nodev usbfs
nodev pipefs
nodev anon_inodefs
nodev inotifyfs
nodev devpts
nodev ramfs
nodev hugetlbfs
nodev pstore
nodev mqueue
Listing Twelve: Here we can see an abbreviated list of the types of file systems supported by the kernel without having to make any tricky changes
You may have heard of an excellent lightweight utility called "vmstat" which reports back dutifully with screeds of useful memory statistics.
You may not, at this juncture, fall off your seat if I tell you that in order to retrieve this useful
information the following files are used. Note the asterisk for all PIDs etc.
/proc/meminfo
/proc/stat
/proc/*/stat
Another aspect that the kernel deals with on a system is the hardware ports, you know the kind that accept a keyboard or a mouse. Have a peek at Listing Thirteen having looked into the file "/proc/ioports", again abbreviated for ease.
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0070-0071 : rtc0
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
Listing Thirteen: Displaying an abbreviated output from the file "/proc/ioports"
As expected we can also look at network information. I’ll leave you to trying some of these yourself. You might want to begin by trying this:
# ls /proc/net
Or for live ARP information (the Address Resolution Protocol) use the “cat” command in front of this:
# cat proc/net/arp
If that output looks familiar then think of how many utilities which you’ve used in the past that refer to this file.
Why stop with ARP? SNMP statistics are also readily available:
# cat /proc/net/snmp
There's a whole heap (no memory puns intended) of other options which we could explore but instead what about a look at changing some of the "sysctl" settings I mentioned earlier too.
Sadly the majority of the powerful changes you can make to a running system's kernel are for another day (there's simply too many to cover) but let's take the popular "/proc/sys" directory for a quick look before we finish.
What about if the system-wide limit of the number of open files for every process on the system is holding you up when you’re trying to launch your latest ground-breaking application?
This setting can cause all sorts of system headaches if it’s abused (and it appears that not only the “root” user can change it on occasion). Warnings aside it’s very useful to arm yourself with this quick fix if it's ever necessary.
Try this command yourself on a test box:
# echo 999999 > /proc/sys/fs/file-max
Once you've conquered that particular outage-inducing issue, taking all the glory as you go, how about checking that the new setting has stuck? Treat it like any other file:
# cat /proc/sys/fs/file-max
I should have of course mentioned that you should check the original value before making any changes (so you can revert the value if need be). My computer used "6815744".
What about another setting? This usually uses a start/stop daemon to make the change, on some systems, but let's do it on the fly:
# echo "ChrisLinux"> /proc/sys/kernel/hostname
In case that you’re wondering you will probably struggle to edit these files with an editor (you may get “file has changed” errors) hence using “echo” as above.
On other distributions you can affect the same change with:
# hostname "ChrisLinux"
I'm sure you get the idea; it’s hardly Computer Science after all.
The End
We have looked in some detail (but not nearly enough) at the very clever Unix-like process table, procfs and “/dev” briefly. We covered how to read and sometimes manipulate tunable settings and how they integrate with components running on Unix-like systems.
Hopefully this starter-for-ten chapter will give you enough of an insight to know what to look if you encounter issues in the future. Above everything else however it’s important for you to find the time to fire up a test machine, dig into otherwise buried file paths and tweak the available settings to further increase your knowledge.
Incidentally should you ever find yourself at a loss on a rainy afternoon then try running this command:
# /sbin/sysctl -a
That should definitely offer you some food for thought. It helpfully lists all of the tunable kernel parameters on a system, each of which can be altered with "echo", simply read the “dots” as “slashes” to find them inside /proc. Just don’t make ill-advised, arbitrary kernel changes on a production machine or you will likely rue the day!
If you weren’t aware of some of the settings that we’ve covered then I hope you will enjoy your newly gained Sysadmin knowledge and use it creatively in the future.
Linux Books
If you've enjoyed reading the technical content on this site then please have a look at my
Linux books which were both published in 2016 and some of my articles in
Linux Magazine and
Admin Magazine are available on their respective websites.
