Have you ever wondered how to patch your systems, reboot, and continue working?
If so, you'll be interested in Ansible, a simple configuration management tool that can make some of the hardest work easy. For example, system administration tasks that can be complicated, take hours to complete, or have complex requirements for security. In my experience, one of the hardest parts of being a sysadmin is patching systems. Every time you get a Common Vulnerabilities and Exposure (CVE) notification or Information Assurance Vulnerability Alert (IAVA) mandated by security, you have to kick into high gear to close the security gaps. (And, believe me, your security officer will hunt you down unless the vulnerabilities are patched.)
Ansible can reduce the time it takes to patch systems by running packaging modules. To demonstrate, let's use the yum module to update the system. Ansible can install, update, remove, or install from another location (e.g., rpmbuild from continuous integration/continuous development). Here is the task for updating the system:

  - name: update the system

    yum:

      name: "*"

      state: latest

In the first line, we give the task a meaningful name so we know what Ansible is doing. In the next line, the yum module updates the CentOS virtual machine (VM), then name: "*" tells yum to update everything, and, finally, state: latest updates to the latest RPM.
After updating the system, we need to restart and reconnect:

  - name: restart system to reboot to newest kernel

    shell: "sleep 5 && reboot"

    async: 1

    poll: 0



  - name: wait for 10 seconds

    pause:

      seconds: 10



  - name: wait for the system to reboot

    wait_for_connection:

      connect_timeout: 20

      sleep: 5

      delay: 5

      timeout: 60



  - name: install epel-release

    yum:

      name: epel-release

      state: latest

The shell module puts the system to sleep for 5 seconds then reboots. We use sleep to prevent the connection from breaking, async to avoid timeout, and poll to fire & forget. We pause for 10 seconds to wait for the VM to come back and use wait_for_connection to connect back to the VM as soon as it can make a connection. Then we install epel-release to test the RPM installation. You can run this playbook multiple times to show the idempotent, and the only task that will show as changed is the reboot since we are using the shell module. You can use changed_when: False to ignore the change when using the shell module if you expect no actual changes.
So far we've learned how to update a system, restart the VM, reconnect, and install a RPM. Next we will install NGINX using the role in Ansible Lightbulb.

  - name: Ensure nginx packages are present

    yum:

      name: nginx, python-pip, python-devel, devel

      state: present

    notify: restart-nginx-service



  - name: Ensure uwsgi package is present

    pip:

      name: uwsgi

      state: present

    notify: restart-nginx-service



  - name: Ensure latest default.conf is present

    template:

      src: templates/nginx.conf.j2

      dest: /etc/nginx/nginx.conf

      backup: yes

    notify: restart-nginx-service



  - name: Ensure latest index.html is present

    template:

      src: templates/index.html.j2

      dest: /usr/share/nginx/html/index.html



  - name: Ensure nginx service is started and enabled

    service:

      name: nginx

      state: started

      enabled: yes



  - name: Ensure proper response from localhost can be received

    uri:

      url: "http://localhost:80/"

      return_content: yes

    register: response

    until: 'nginx_test_message in response.content'

    retries: 10

    delay: 1

And the handler that restarts the nginx service:

# handlers file for nginx-example

  - name: restart-nginx-service

    service:

      name: nginx

      state: restarted

In this role, we install the RPMs nginx, python-pip, python-devel, and devel and install uwsgi with PIP. Next, we use the template module to copy over the nginx.conf and index.html for the page to display. After that, we make sure the service is enabled on boot and started. Then we use the uri module to check the connection to the page.
Here is a playbook showing an example of updating, restarting, and installing an RPM. Then continue installing nginx. This can be done with any other roles/applications you want.

  - hosts: all

    roles:

      - centos-update

      - nginx-simple

Watch this demo video for more insight on the process.
This was just a simple example of how to update, reboot, and continue. For simplicity, I added the packages without variables. Once you start working with a large number of hosts, you will need to change a few settings:

• async & poll
• serial
• forks

This is because on your production environment you might want to update one system at a time (not fire & forget) and actually wait a longer time for your system to reboot and continue.
For more ways to automate your work with this tool, take a look at the other Ansible articles on Opensource.com.

In my previous article, I discussed how to use Ansible to patch systems and install applications. In this article, I'll show you how to do other things with Ansible that will make your life as a sysadmin easier. First, though, I want to share why I came to Ansible.
I started using Ansible because it made patching systems easier. I could run some ad-hoc commands here and there and some playbooks someone else wrote. I didn't get very in depth, though, because the playbook I was running used a lot of lineinfile modules, and, to be honest, my regex techniques were nonexistent. I was also limited in my capacity due to my management's direction and instructions: "You can run this playbook only and that's all you can do." After leaving that job, I started working on a team where most of the infrastructure was in the cloud. After getting used to the team and learning how everything works, I started trying to find ways to automate more things. We were spending two to three months deploying virtual machines in large numbers—doing all the work manually, including the lifecycle of each virtual machine, from provision to decommission. Our work often got behind schedule, as we spent a lot of time doing maintenance. When folks went on vacation, others had to take over with little knowledge of the tasks they were doing.

Diving deeper into Ansible

Sharing ideas about how to resolve issues is one of the best things we can do in the IT and open source world, so I went looking for help by submitting issues in Ansible and asking questions in roles others created.
Reading the documentation (including the following topics) is the best way to get started learning Ansible.

• Getting started
• Best practices
• Ansible Lightbulb
• Ansible FAQ

If you are trying to figure out what you can do with Ansible, take a moment and think about the daily activities you do, the ones that take a lot of time that would be better spent on other things. Here are some examples:

• Managing accounts in systems: Creating users, adding them to the correct groups, and adding the SSH keys… these are things that used to take me days when we had a large number of systems to build. Even using a shell script, this process was very time-consuming.
• Maintaining lists of required packages: This could be part of your security posture and include the packages required for your applications.
• Installing applications: You can use your current documentation and convert application installs into tasks by finding the correct module for the job.
• Configuring systems and applications: You might want to change /etc/ssh/sshd_config for different environments (e.g., production vs. development) by adding a line or two, or maybe you want a file to look a specific way in every system you're managing.
• Provisioning a VM in the cloud: This is great when you need to launch a few virtual machines that are similar for your applications and you are tired of using the UI.

Now let's look at how to use Ansible to automate some of these repetitive tasks.

Managing users

If you need to create a large list of users and groups with the users spread among the different groups, you can use loops. Let's start by creating the groups:

- name: create user groups

  group:

    name: "{{ item }}"

  loop:

    - postgresql

    - nginx-test

    - admin

    - dbadmin

    - hadoop

You can create users with specific parameters like this:

- name: all users in the department

  user:

    name:  "{{ item.name }}"

    group: "{{ item.group }}"

    groups: "{{ item.groups }}"

    uid: "{{ item.uid }}"

    state: "{{ item.state }}"

  loop:

    - { name: 'admin1', group: 'admin', groups: 'nginx', uid: '1234', state: 'present' }

    - { name: 'dbadmin1', group: 'dbadmin', groups: 'postgres', uid: '4321', state: 'present' }

    - { name: 'user1', group: 'hadoop', groups: 'wheel', uid: '1067', state: 'present' }

    - { name: 'jose', group: 'admin', groups: 'wheel', uid: '9000', state: 'absent' }

Looking at the user jose, you may recognize that state: 'absent' deletes this user account, and you may be wondering why you need to include all the other parameters when you're just removing him. It's because this is a good place to keep documentation of important changes for audits or security compliance. By storing the roles in Git as your source of truth, you can go back and look at the old versions in Git if you later need to answer questions about why changes were made.
To deploy SSH keys for some of the users, you can use the same type of looping as in the last example.

- name: copy admin1 and dbadmin ssh keys

  authorized_key:

    user: "{{ item.user }}"

    key: "{{ item.key }}"

    state: "{{ item.state }}"

    comment: "{{ item.comment }}"

  loop:

    - { user: 'admin1', key: "{{ lookup('file', '/data/test_temp_key.pub'), state: 'present', comment: 'admin1 key' }

    - { user: 'dbadmin', key: "{{ lookup('file', '/data/vm_temp_key.pub'), state: 'absent', comment: 'dbadmin key' }

Here, we specify the user, how to find the key by using lookup, the state, and a comment describing the purpose of the key.

Installing packages

Package installation can vary depending on the packaging system you are using. You can use Ansible facts to determine which module to use. Ansible does offer a generic module called package that uses ansible_pkg_mgr and calls the proper package manager for the system. For example, if you're using Fedora, the package module will call the DNF package manager.
The package module will work if you're doing a simple installation of packages. If you're doing more complex work, you will have to use the correct module for your system. For example, if you want to ignore GPG keys and install all the security packages on a RHEL-based system, you need to use the yum module. You will have different options depending on your packaging module, but they usually offer more parameters than Ansible's generic package module.
Here is an example using the package module:

  - name: install a package

    package:

      name: nginx

      state: installed

The following uses the yum module to install NGINX, disable gpg_check from the repo, ignore the repository's certificates, and skip any broken packages that might show up.

  - name: install a package

    yum:

      name: nginx

      state: installed

      disable_gpg_check: yes

      validate_certs: no

      skip_broken: yes

Here is an example using Apt. The Apt module tells Ansible to uninstall NGINX and not update the cache:

  - name: install a package

    apt:

      name: nginx

      state: absent

      update_cache: no

You can use loop when installing packages, but they are processed individually if you pass a list:

  - name:

      - nginx

      - postgresql-server

      - ansible

      - httpd

NOTE: Make sure you know the correct name of the package you want in the package manager you're using. Some names change depending on the package manager.

Starting services

Much like packages, Ansible has different modules to start services. Like in our previous example, where we used the package module to do a general installation of packages, the service module does similar work with services, including with systemd and Upstart. (Check the module's documentation for a complete list.) Here is an example:

  - name: start nginx

    service: 

      name: nginx

      state: started

You can use Ansible's service module if you are just starting and stopping applications and don't need anything more sophisticated. But, like with the yum module, if you need more options, you will need to use the systemd module. For example, if you modify systemd files, then you need to do a daemon-reload, the service module won't work for that; you will have to use the systemd module.

  - name: reload postgresql for new configuration and reload daemon

    systemd:

      name: postgresql

      state: reload

      daemon-reload: yes

This is a great starting point, but it can become cumbersome because the service will always reload/restart. This a good place to use a handler.
If you used best practices and created your role using ansible-galaxy init "role name", then you should have the full directory structure. You can include the code above inside the handlers/main.yml and call it when you make a change with the application. For example:

handlers/main.yml



  - name: reload postgresql for new configuration and reload daemon

    systemd:

      name: postgresql

      state: reload

      daemon-reload: yes

This is the task that calls the handler:

  - name: configure postgresql

    template:

      src: postgresql.service.j2

      dest: /usr/lib/systemd/system/postgresql.service

    notify: reload postgresql for new configuration and reload daemon

It configures PostgreSQL by changing the systemd file, but instead of defining the restart in the tasks (like before), it calls the handler to do the restart at the end of the run. This is a good way to configure your application and keep it idempotent since the handler only runs when a task changes—not in the middle of your configuration.
The previous example uses the template module and a Jinja2 file. One of the most wonderful things about configuring applications with Ansible is using templates. You can configure a whole file like postgresql.service with the full configuration you require. But, instead of changing every line, you can use variables and define the options somewhere else. This will let you change any variable at any time and be more versatile. For example:

[database]

DB_TYPE  = "{{ gitea_db }}"

HOST     = "{{ ansible_fqdn}}:3306"

NAME     = gitea

USER     = gitea

PASSWD   = "{{ gitea_db_passwd }}"

SSL_MODE = disable

PATH     = "{{ gitea_db_dir }}/gitea.db

This configures the database options on the file app.ini for Gitea. This is similar to writing Ansible tasks, even though it is a configuration file, and makes it easy to define variables and make changes. This can be expanded further if you are using group_vars, which allows you to define variables for all systems and specific groups (e.g., production vs. development). This makes it easier to manage variables, and you don't have to specify the same ones in every role.

Provisioning a system

We've gone over several things you can do with Ansible on your system, but we haven't yet discussed how to provision a system. Here's an example of provisioning a virtual machine (VM) with the OpenStack cloud solution.

  - name: create a VM in openstack

    osp_server:

      name: cloudera-namenode

      state: present

      cloud: openstack

      region_name: andromeda

      image: 923569a-c777-4g52-t3y9-cxvhl86zx345

      flavor_ram: 20146

      flavor: big

      auto_ip: yes

      volumes: cloudera-namenode

All OpenStack modules start with os, which makes it easier to find them. The above configuration uses the osp-server module, which lets you add or remove an instance. It includes the name of the VM, its state, its cloud options, and how it authenticates to the API. More information about cloud.yml is available in the OpenStack docs, but if you don't want to use cloud.yml, you can use a dictionary that lists your credentials using the auth option. If you want to delete the VM, just change state: to absent.
Say you have a list of servers you shut down because you couldn't figure out how to get the applications working, and you want to start them again. You can use os_server_action to restart them (or rebuild them if you want to start from scratch).
Here is an example that starts the server and tells the modules the name of the instance:

  - name: restart some servers

    os_server_action:

      action: start

      cloud: openstack

      region_name: andromeda

      server: cloudera-namenode

Most OpenStack modules use similar options. Therefore, to rebuild the server, we can use the same options but change the action to rebuild and add the image we want it to use:

  os_server_action:

    action: rebuild

    image: 923569a-c777-4g52-t3y9-cxvhl86zx345

Doing other things

There are modules for a lot of system admin tasks, but what should you do if there isn't one for what you are trying to do? Use the shell and command modules, which allow you to run any command just like you do on the command line. Here's an example using the OpenStack CLI:

  - name: run an opencli command

    command: "openstack hypervisor list"

---

They are so many ways you can do daily sysadmin tasks with Ansible. Using this automation tool can transform your hardest task into a simple solution, save you time, and make your work days shorter and more relaxed.

You probably know the feeling: You ask for data and get a positive response, only to open the email and find a whole bunch of PDFs attached. Data, interrupted.
We understand your frustration, and we’ve done something about it: Introducing Textricator, our first open source product.
We’re Measures for Justice, a criminal justice research and transparency organization. Our mission is to provide data transparency for the entire justice system, from arrest to post-conviction. We do this by producing a series of up to 32 performance measures covering the entire criminal justice system, county by county. We get our data in many ways—all legal, of course—and while many state and county agencies are data-savvy, giving us quality, formatted data in CSVs, the data is often bundled inside software with no simple way to get it out. PDF reports are the best they can offer.
Developers Joe Hale and Stephen Byrne have spent the past two years developing Textricator to extract tens of thousands of pages of data for our internal use. Textricator can process just about any text-based PDF format—not just tables, but complex reports with wrapping text and detail sections generated from tools like Crystal Reports. Simply tell Textricator the attributes of the fields you want to collect, and it chomps through the document, collecting and writing out your records.
Not a software engineer? Textricator doesn’t require programming skills; rather, the user describes the structure of the PDF and Textricator handles the rest. Most users run it via the command line; however, a browser-based GUI is available.
We evaluated other great open source solutions like Tabula, but they just couldn’t handle the structure of some of the PDFs we needed to scrape. “Textricator is both flexible and powerful and has cut the time we spend to process large datasets from days to hours,” says Andrew Branch, director of technology.
At MFJ, we’re committed to transparency and knowledge-sharing, which includes making our software available to anyone, especially those trying to free and share data publicly. Textricator is available on GitHub and released under GNU Affero General Public License Version 3.
You can see the results of our work, including data processed via Textricator, on our free online data portal. Textricator is an essential part of our process and we hope civic tech and government organizations alike can unlock more data with this new tool.
If you use Textricator, let us know how it helped solve your data problem. Want to improve it? Submit a pull request.

In this three-part series, I'll explain how to set up a simple, useful NAS (network attached storage) system. I use this kind of setup to store my files on a central system, creating incremental backups automatically every night. To mount the disk on devices that are located in the same network, NFS is installed. To access files offline and share them with friends, I use Nextcloud.
This article will cover the basic setup of software and hardware to mount the data disk on a remote device. In the second article, I will discuss a backup strategy and set up a cron job to create daily backups. In the third and last article, we will install Nextcloud, a tool for easy file access to devices synced offline as well as online using a web interface. It supports multiple users and public file-sharing so you can share pictures with friends, for example, by sending a password-protected link.
The target architecture of our system looks like this:

Hardware

Let's get started with the hardware you need. You might come up with a different shopping list, so consider this one an example.
The computing power is delivered by a Raspberry Pi 3, which comes with a quad-core CPU, a gigabyte of RAM, and (somewhat) fast ethernet. Data will be stored on two USB hard drives (I use 1-TB disks); one is used for the everyday traffic, the other is used to store backups. Be sure to use either active USB hard drives or a USB hub with an additional power supply, as the Raspberry Pi will not be able to power two USB drives.

Software

The operating system with the highest visibility in the community is Raspbian, which is excellent for custom projects. There are plenty of guides that explain how to install Raspbian on a Raspberry Pi, so I won't go into details here. The latest official supported version at the time of this writing is Raspbian Stretch, which worked fine for me. At this point, I will assume you have configured your basic Raspbian and are able to connect to the Raspberry Pi by ssh.

Prepare the USB drives

To achieve good performance reading from and writing to the USB hard drives, I recommend formatting them with ext4. To do so, you must first find out which disks are attached to the Raspberry Pi. You can find the disk devices in /dev/sd/. Using the command fdisk -l, you can find out which two USB drives you just attached. Please note that all data on the USB drives will be lost as soon as you follow these steps.

pi@raspberrypi:~ $ sudofdisk-l



<...>



Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors

Units: sectors of 1*512 = 512 bytes

Sector size(logical/physical): 512 bytes /512 bytes

I/O size(minimum/optimal): 512 bytes /512 bytes

Disklabel type: dos

Disk identifier: 0xe8900690



Device     Boot Start        End    Sectors   Size Id Type

/dev/sda1        204819535251671953523120 931.5G 83 Linux





Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors

Units: sectors of 1*512 = 512 bytes

Sector size(logical/physical): 512 bytes /512 bytes

I/O size(minimum/optimal): 512 bytes /512 bytes

Disklabel type: dos

Disk identifier: 0x6aa4f598



Device     Boot Start        End    Sectors   Size Id Type

/dev/sdb1  *    204819535216631953519616 931.5G  83 Linux

As those devices are the only 1TB disks attached to the Raspberry Pi, we can easily see that /dev/sda and /dev/sdb are the two USB drives. The partition table at the end of each disk shows how it should look after the following steps, which create the partition table and format the disks. To do this, repeat the following steps for each of the two devices by replacing sda with sdb the second time (assuming your devices are also listed as /dev/sda and /dev/sdb in fdisk).
First, delete the partition table of the disk and create a new one containing only one partition. In fdisk, you can use interactive one-letter commands to tell the program what to do. Simply insert them after the prompt Command (m for help): as follows (you can also use the m command anytime to get more information):

pi@raspberrypi:~ $ sudofdisk/dev/sda



Welcome to fdisk(util-linux 2.29.2).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.





Command (m forhelp): o

Created a new DOS disklabel with disk identifier 0x9c310964.



Command (m forhelp): n

Partition type

   p   primary (0 primary, 0 extended, 4free)

   e   extended (container for logical partitions)

Select (default p): p

Partition number (1-4, default 1):

First sector (2048-1953525167, default 2048):

Last sector, +sectors or +size{K,M,G,T,P}(2048-1953525167, default 1953525167):



Created a new partition 1 of type'Linux' and of size931.5 GiB.



Command (m forhelp): p



Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors

Units: sectors of 1*512 = 512 bytes

Sector size(logical/physical): 512 bytes /512 bytes

I/O size(minimum/optimal): 512 bytes /512 bytes

Disklabel type: dos

Disk identifier: 0x9c310964



Device     Boot Start        End    Sectors   Size Id Type

/dev/sda1        204819535251671953523120 931.5G 83 Linux



Command (m forhelp): w

The partition table has been altered.

Syncing disks.

Now we will format the newly created partition /dev/sda1 using the ext4 filesystem:

pi@raspberrypi:~ $ sudo mkfs.ext4 /dev/sda1

mke2fs 1.43.4 (31-Jan-2017)

Discarding device blocks: done



<...>



Allocating group tables: done

Writing inode tables: done

Creating journal (1024 blocks): done

Writing superblocks and filesystem accounting information: done

After repeating the above steps, let's label the new partitions according to their usage in your system:

pi@raspberrypi:~ $ sudo e2label /dev/sda1 data

pi@raspberrypi:~ $ sudo e2label /dev/sdb1 backup 

Now let's get those disks mounted to store some data. My experience, based on running this setup for over a year now, is that USB drives are not always available to get mounted when the Raspberry Pi boots up (for example, after a power outage), so I recommend using autofs to mount them when needed.
First install autofs and create the mount point for the storage:

pi@raspberrypi:~ $ sudo apt install autofs

pi@raspberrypi:~ $ sudomkdir/nas

Then mount the devices by adding the following line to /etc/auto.master:

/nas    /etc/auto.usb

Create the file /etc/auto.usb if not existing with the following content, and restart the autofs service:

data -fstype=ext4,rw :/dev/disk/by-label/data

backup -fstype=ext4,rw :/dev/disk/by-label/backup

pi@raspberrypi3:~ $ sudo service autofs restart

Now you should be able to access the disks at /nas/data and /nas/backup, respectively. Clearly, the content will not be too thrilling, as you just erased all the data from the disks. Nevertheless, you should be able to verify the devices are mounted by executing the following commands:

pi@raspberrypi3:~ $ cd/nas/data

pi@raspberrypi3:/nas/data $ cd/nas/backup

pi@raspberrypi3:/nas/backup $ mount

<...>

/etc/auto.usb on /nas type autofs (rw,relatime,fd=6,pgrp=463,timeout=300,minproto=5,maxproto=5,indirect)

<...>

/dev/sda1 on /nas/data type ext4 (rw,relatime,data=ordered)

/dev/sdb1 on /nas/backup type ext4 (rw,relatime,data=ordered)

First move into the directories to make sure autofs mounts the devices. Autofs tracks access to the filesystems and mounts the needed devices on the go. Then the mount command shows that the two devices are actually mounted where we wanted them.
Setting up autofs is a bit fault-prone, so do not get frustrated if mounting doesn't work on the first try. Give it another chance, search for more detailed resources (there is plenty of documentation online), or leave a comment.

Mount network storage

Now that you have set up the basic network storage, we want it to be mounted on a remote Linux machine. We will use the network file system (NFS) for this. First, install the NFS server on the Raspberry Pi:

pi@raspberrypi:~ $ sudo apt install nfs-kernel-server

Next we need to tell the NFS server to expose the /nas/data directory, which will be the only device accessible from outside the Raspberry Pi (the other one will be used for backups only). To export the directory, edit the file /etc/exports and add the following line to allow all devices with access to the NAS to mount your storage:

/nas/data *(rw,sync,no_subtree_check)

For more information about restricting the mount to single devices and so on, refer to man exports. In the configuration above, anyone will be able to mount your data as long as they have access to the ports needed by NFS: 111 and 2049. I use the configuration above and allow access to my home network only for ports 22 and 443 using the routers firewall. That way, only devices in the home network can reach the NFS server.
To mount the storage on a Linux computer, run the commands:

you@desktop:~ $ sudomkdir/nas/data

you@desktop:~ $ sudomount-t nfs <raspberry-pi-hostname-or-ip>:/nas/data /nas/data

Again, I recommend using autofs to mount this network device. For extra help, check out How to use autofs to mount NFS shares.
Now you are able to access files stored on your own RaspberryPi-powered NAS from remote devices using the NFS mount. In the next part of this series, I will cover how to automatically back up your data to the second hard drive using rsync. To save space on the device while still doing daily backups, you will learn how to create incremental backups with rsync.

Every computerized device uses some form of software to perform its intended tasks. In the early days of software, products were stringently tested for bugs and other defects. For the last decade or so, software has been released via the internet with the intent that any bugs would be fixed by applying new versions of the software. In some cases, each individual application has its own updater. In others, it is left up to the user to figure out how to obtain and upgrade software.
Linux adopted early the practice of maintaining a centralized location where users could find and install software. In this article, I'll discuss the history of software installation on Linux and how modern operating systems are kept up to date against the never-ending torrent of CVEs.

How was software on Linux installed before package managers?

Historically, software was provided either via FTP or mailing lists (eventually this distribution would grow to include basic websites). Only a few small files contained the instructions to create a binary (normally in a tarfile). You would untar the files, read the readme, and as long as you had GCC or some other form of C compiler, you would then typically run a ./configure script with some list of attributes, such as pathing to library files, location to create new binaries, etc. In addition, the configure process would check your system for application dependencies. If any major requirements were missing, the configure script would exit and you could not proceed with the installation until all the dependencies were met. If the configure script completed successfully, a Makefile would be created.
Once a Makefile existed, you would then proceed to run the make command (this command is provided by whichever compiler you were using). The make command has a number of options called make flags, which help optimize the resulting binaries for your system. In the earlier days of computing, this was very important because hardware struggled to keep up with modern software demands. Today, compilation options can be much more generic as most hardware is more than adequate for modern software.
Finally, after the make process had been completed, you would need to run make install (or sudo make install) in order to actually install the software. As you can imagine, doing this for every single piece of software was time-consuming and tedious—not to mention the fact that updating software was a complicated and potentially very involved process.

What is a package?

Packages were invented to combat this complexity. Packages collect multiple data files together into a single archive file for easier portability and storage, or simply compress files to reduce storage space. The binaries included in a package are precompiled with according to the sane defaults the developer chosen. Packages also contain metadata, such as the software's name, a description of its purpose, a version number, and a list of dependencies necessary for the software to run properly.
Several flavors of Linux have created their own package formats. Some of the most commonly used package formats include:

• .deb: This package format is used by Debian, Ubuntu, Linux Mint, and several other derivatives. It was the first package type to be created.
• .rpm: This package format was originally called Red Hat Package Manager. It is used by Red Hat, Fedora, SUSE, and several other smaller distributions.
• .tar.xz: While it is just a compressed tarball, this is the format that Arch Linux uses.

While packages themselves don't manage dependencies directly, they represented a huge step forward in Linux software management.

What is a software repository?

A few years ago, before the proliferation of smartphones, the idea of a software repository was difficult for many users to grasp if they were not involved in the Linux ecosystem. To this day, most Windows users still seem to be hardwired to open a web browser to search for and install new software. However, those with smartphones have gotten used to the idea of a software "store." The way smartphone users obtain software and the way package managers work are not dissimilar. While there have been several attempts at making an attractive UI for software repositories, the vast majority of Linux users still use the command line to install packages. Software repositories are a centralized listing of all of the available software for any repository the system has been configured to use. Below are some examples of searching a repository for a specifc package (note that these have been truncated for brevity):
Arch Linux with aurman

user@arch ~ $  aurman -Ss kate



extra/kate 18.04.2-2 (kde-applications kdebase)

    Advanced Text Editor

aur/kate-root 18.04.0-1 (11, 1.139399)

    Advanced Text Editor, patched to be able to run as root

aur/kate-git r15288.15d26a7-1 (1, 1e-06)

    An advanced editor component which is used in numerous KDE applications requiring a text editing component

CentOS 7 using YUM

[user@centos ~]$ yum search kate



kate-devel.x86_64 : Development files for kate

kate-libs.x86_64 : Runtime files for kate

kate-part.x86_64 : Kate kpart plugin

Ubuntu using APT

user@ubuntu ~ $ apt search kate

Sorting... Done

Full Text Search... Done



kate/xenial 4:15.12.3-0ubuntu2 amd64

  powerful text editor



kate-data/xenial,xenial 4:4.14.3-0ubuntu4 all

  shared data files for Kate text editor



kate-dbg/xenial 4:15.12.3-0ubuntu2 amd64

  debugging symbols for Kate



kate5-data/xenial,xenial 4:15.12.3-0ubuntu2 all

  shared data files for Kate text editor

What are the most prominent package managers?

As suggested in the above output, package managers are used to interact with software repositories. The following is a brief overview of some of the most prominent package managers.

RPM-based package managers

Updating RPM-based systems, particularly those based on Red Hat technologies, has a very interesting and detailed history. In fact, the current versions of yum (for enterprise distributions) and DNF (for community) combine several open source projects to provide their current functionality.
Initially, Red Hat used a package manager called RPM (Red Hat Package Manager), which is still in use today. However, its primary use is to install RPMs, which you have locally, not to search software repositories. The package manager named up2date was created to inform users of updates to packages and enable them to search remote repositories and easily install dependencies. While it served its purpose, some community members felt that up2date had some significant shortcomings.
The current incantation of yum came from several different community efforts. Yellowdog Updater (YUP) was developed in 1999-2001 by folks at Terra Soft Solutions as a back-end engine for a graphical installer of Yellow Dog Linux. Duke University liked the idea of YUP and decided to improve upon it. They created Yellowdog Updater, Modified (yum) which was eventually adapted to help manage the university's Red Hat Linux systems. Yum grew in popularity, and by 2005 it was estimated to be used by more than half of the Linux market. Today, almost every distribution of Linux that uses RPMs uses yum for package management (with a few notable exceptions).

Working with yum

In order for yum to download and install packages out of an internet repository, files must be located in /etc/yum.repos.d/ and they must have the extension .repo. Here is an example repo file:

[local_base]

name=Base CentOS  (local)

baseurl=http://7-repo.apps.home.local/yum-repo/7/

enabled=1

gpgcheck=0

This is for one of my local repositories, which explains why the GPG check is off. If this check was on, each package would need to be signed with a cryptographic key and a corresponding key would need to be imported into the system receiving the updates. Because I maintain this repository myself, I trust the packages and do not bother signing them.
Once a repository file is in place, you can start installing packages from the remote repository. The most basic command is yum update, which will update every package currently installed. This does not require a specific step to refresh the information about repositories; this is done automatically. A sample of the command is shown below:

[user@centos ~]$ sudo yum update

Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager

local_base                             | 3.6 kB  00:00:00     

local_epel                             | 2.9 kB  00:00:00     

local_rpm_forge                        | 1.9 kB  00:00:00     

local_updates                          | 3.4 kB  00:00:00     

spideroak-one-stable                   | 2.9 kB  00:00:00     

zfs                                    | 2.9 kB  00:00:00     

(1/6): local_base/group_gz             | 166 kB  00:00:00     

(2/6): local_updates/primary_db        | 2.7 MB  00:00:00     

(3/6): local_base/primary_db           | 5.9 MB  00:00:00     

(4/6): spideroak-one-stable/primary_db |  12 kB  00:00:00     

(5/6): local_epel/primary_db           | 6.3 MB  00:00:00     

(6/6): zfs/x86_64/primary_db           |  78 kB  00:00:00     

local_rpm_forge/primary_db             | 125 kB  00:00:00     

Determining fastest mirrors

Resolving Dependencies

--> Running transaction check

If you are sure you want yum to execute any command without stopping for input, you can put the -y flag in the command, such as yum update -y.
Installing a new package is just as easy. First, search for the name of the package with yum search:

[user@centos ~]$ yum search kate



artwiz-aleczapka-kates-fonts.noarch : Kates font in Artwiz family

ghc-highlighting-kate-devel.x86_64 : Haskell highlighting-kate library development files

kate-devel.i686 : Development files for kate

kate-devel.x86_64 : Development files for kate

kate-libs.i686 : Runtime files for kate

kate-libs.x86_64 : Runtime files for kate

kate-part.i686 : Kate kpart plugin

Once you have the name of the package, you can simply install the package with sudo yum install kate-devel -y. If you installed a package you no longer need, you can remove it with sudo yum remove kate-devel -y. By default, yum will remove the package plus its dependencies.
There may be times when you do not know the name of the package, but you know the name of the utility. For example, suppose you are looking for the utility updatedb, which creates/updates the database used by the locate command. Attempting to install updatedb returns the following results:

[user@centos ~]$ sudo yum install updatedb

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile

No package updatedb available.

Error: Nothing to do

You can find out what package the utility comes from by running:

[user@centos ~]$ yum whatprovides *updatedb

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile



bacula-director-5.2.13-23.1.el7.x86_64 : Bacula Director files

Repo        : local_base

Matched from:

Filename    : /usr/share/doc/bacula-director-5.2.13/updatedb



mlocate-0.26-8.el7.x86_64 : An utility for finding files by name

Repo        : local_base

Matched from:

Filename    : /usr/bin/updatedb

The reason I have used an asterisk * in front of the command is because yum whatprovides uses the path to the file in order to make a match. Since I was not sure where the file was located, I used an asterisk to indicate any path.
There are, of course, many more options available to yum. I encourage you to view the man page for yum for additional options.
Dandified Yum (DNF) is a newer iteration on yum. Introduced in Fedora 18, it has not yet been adopted in the enterprise distributions, and as such is predominantly used in Fedora (and derivatives). Its usage is almost exactly the same as that of yum, but it was built to address poor performance, undocumented APIs, slow/broken dependency resolution, and occasional high memory usage. DNF is meant as a drop-in replacement for yum, and therefore I won't repeat the commands—wherever you would use yum, simply substitute dnf.

Working with Zypper

Zypper is another package manager meant to help manage RPMs. This package manager is most commonly associated with SUSE (and openSUSE) but has also seen adoption by MeeGo, Sailfish OS, and Tizen. It was originally introduced in 2006 and has been iterated upon ever since. There is not a whole lot to say other than Zypper is used as the back end for the system administration tool YaST and some users find it to be faster than yum.
Zypper's usage is very similar to that of yum. To search for, update, install or remove a package, simply use the following:

zypper search kate

zypper update

zypper install kate

zypper remove kate

Some major differences come into play in how repositories are added to the system with zypper. Unlike the package managers discussed above, zypper adds repositories using the package manager itself. The most common way is via a URL, but zypper also supports importing from repo files.

suse:~ # zypper addrepo http://download.videolan.org/pub/vlc/SuSE/15.0 vlc

Adding repository 'vlc' [done]

Repository 'vlc' successfully added



Enabled     : Yes

Autorefresh : No

GPG Check   : Yes

URI         : http://download.videolan.org/pub/vlc/SuSE/15.0

Priority    : 99

You remove repositories in a similar manner:

suse:~ # zypper removerepo vlc

Removing repository 'vlc' ...................................[done]

Repository 'vlc' has been removed.

Use the zypper repos command to see what the status of repositories are on your system:

suse:~ # zypper repos

Repository priorities are without effect. All enabled repositories share the same priority.



#  | Alias                     | Name                                    | Enabled | GPG Check | Refresh

---+---------------------------+-----------------------------------------+---------+-----------+--------

 1 | repo-debug                | openSUSE-Leap-15.0-Debug                | No      | ----      | ----   

 2 | repo-debug-non-oss        | openSUSE-Leap-15.0-Debug-Non-Oss        | No      | ----      | ----   

 3 | repo-debug-update         | openSUSE-Leap-15.0-Update-Debug         | No      | ----      | ----   

 4 | repo-debug-update-non-oss | openSUSE-Leap-15.0-Update-Debug-Non-Oss | No      | ----      | ----   

 5 | repo-non-oss              | openSUSE-Leap-15.0-Non-Oss              | Yes     | ( p) Yes  | Yes    

 6 | repo-oss                  | openSUSE-Leap-15.0-Oss                  | Yes     | ( p) Yes  | Yes     

zypper even has a similar ability to determine what package name contains files or binaries. Unlike YUM, it uses a hyphen in the command (although this method of searching is deprecated):

localhost:~ # zypper what-provides kate

Command 'what-provides' is replaced by 'search --provides --match-exact'.

See 'help search' for all available options.

Loading repository data...

Reading installed packages...



S  | Name | Summary              | Type       

---+------+----------------------+------------

i+ | Kate | Advanced Text Editor | application

i  | kate | Advanced Text Editor | package  

As with YUM and DNF, Zypper has a much richer feature set than covered here. Please consult with the official documentation for more in-depth information.

Debian-based package managers

One of the oldest Linux distributions currently maintained, Debian's system is very similar to RPM-based systems. They use .deb packages, which can be managed by a tool called dpkg. dpkg is very similar to rpm in that it was designed to manage packages that are available locally. It does no dependency resolution (although it does dependency checking), and has no reliable way to interact with remote repositories. In order to improve the user experience and ease of use, the Debian project commissioned a project called Deity. This codename was eventually abandoned and changed to Advanced Package Tool (APT).
Released as test builds in 1998 (before making an appearance in Debian 2.1 in 1999), many users consider APT one of the defining features of Debian-based systems. It makes use of repositories in a similar fashion to RPM-based systems, but instead of individual .repo files that yum uses, apt has historically used /etc/apt/sources.list to manage repositories. More recently, it also ingests files from /etc/apt/sources.d/. Following the examples in the RPM-based package managers, to accomplish the same thing on Debian-based distributions you have a few options. You can edit/create the files manually in the aforementioned locations from the terminal, or in some cases, you can use a UI front end (such as Software & Updates provided by Ubuntu et al.). To provide the same treatment to all distributions, I will cover only the command-line options. To add a repository without directly editing a file, you can do something like this:

user@ubuntu:~$ sudo apt-add-repository "deb http://APT.spideroak.com/ubuntu-spideroak-hardy/ release restricted"

This will create a spideroakone.list file in /etc/apt/sources.list.d. Obviously, these lines change depending on the repository being added. If you are adding a Personal Package Archive (PPA), you can do this:

user@ubuntu:~$ sudo apt-add-repository ppa:gnome-desktop

NOTE: Debian does not support PPAs natively.
After a repository has been added, Debian-based systems need to be made aware that there is a new location to search for packages. This is done via the apt-get update command:

user@ubuntu:~$ sudo apt-get update

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]

Hit:2 http://APT.spideroak.com/ubuntu-spideroak-hardy release InRelease

Hit:3 http://ca.archive.ubuntu.com/ubuntu xenial InRelease

Get:4 http://ca.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]              

Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [517 kB]

Get:6 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [455 kB]      

Get:7 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [221 kB]     

...



Fetched 6,399 kB in 3s (2,017 kB/s)                                           

Reading package lists... Done

Now that the new repository is added and updated, you can search for a package using the apt-cache command:

user@ubuntu:~$ apt-cache search kate

aterm-ml - Afterstep XVT - a VT102 emulator for the X window system

frescobaldi - Qt4 LilyPond sheet music editor

gitit - Wiki engine backed by a git or darcs filestore

jedit - Plugin-based editor for programmers

kate - powerful text editor

kate-data - shared data files for Kate text editor

kate-dbg - debugging symbols for Kate

katepart - embeddable text editor component

To install kate, simply run the corresponding install command:

user@ubuntu:~$ sudo apt-get install kate

To remove a package, use apt-get remove:

user@ubuntu:~$ sudo apt-get remove kate

When it comes to package discovery, APT does not provide any functionality that is similar to yum whatprovides. There are a few ways to get this information if you are trying to find where a specific file on disk has come from.
Using dpkg

user@ubuntu:~$ dpkg -S /bin/ls

coreutils: /bin/ls

Using apt-file

user@ubuntu:~$ sudo apt-get install apt-file -y 



user@ubuntu:~$ sudo apt-file update



user@ubuntu:~$ apt-file search kate

The problem with apt-file search is that it, unlike yum whatprovides, it is overly verbose unless you know the exact path, and it automatically adds a wildcard search so that you end up with results for anything with the word kate in it:

kate: /usr/bin/kate

kate: /usr/lib/x86_64-linux-gnu/qt5/plugins/ktexteditor/katebacktracebrowserplugin.so

kate: /usr/lib/x86_64-linux-gnu/qt5/plugins/ktexteditor/katebuildplugin.so

kate: /usr/lib/x86_64-linux-gnu/qt5/plugins/ktexteditor/katecloseexceptplugin.so

kate: /usr/lib/x86_64-linux-gnu/qt5/plugins/ktexteditor/katectagsplugin.so

Most of these examples have used apt-get. Note that most of the current tutorials for Ubuntu specifically have taken to simply using apt. The single apt command was designed to implement only the most commonly used commands in the APT arsenal. Since functionality is split between apt-get, apt-cache, and other commands, apt looks to unify these into a single command. It also adds some niceties such as colorization, progress bars, and other odds and ends. Most of the commands noted above can be replaced with apt,  but not all Debian-based distributions currently receiving security patches support using apt by default, so you may need to install additional packages.

Arch-based package managers

Arch Linux uses a package manager called pacman. Unlike .deb or .rpm files, pacman uses a more traditional tarball with the LZMA2 compression (.tar.xz). This enables Arch Linux packages to be much smaller than other forms of compressed archives (such as gzip). Initially released in 2002, pacman has been steadily iterated and improved. One of the major benefits of pacman is that it supports the Arch Build System, a system for building packages from source. The build system ingests a file called a PKGBUILD, which contains metadata (such as version numbers, revisions, dependencies, etc.) as well as a shell script with the required flags for compiling a package conforming to the Arch Linux requirements. The resulting binaries are then packaged into the aforementioned .tar.xz file for consumption by pacman.
This system led to the creation of the Arch User Repository (AUR) which is a community-driven repository containing PKGBUILD files and supporting patches or scripts. This allows for a virtually endless amount of software to be available in Arch. The obvious advantage of this system is that if a user (or maintainer) wishes to make software available to the public, they do not have to go through official channels to get it accepted in the main repositories. The downside is that it relies on community curation similar to Docker Hub, Canonical's Snap packages, or other similar mechanisms. There are numerous AUR-specific package managers that can be used to download, compile, and install from the PKGBUILD files in the AUR (we will look at this later).

Working with pacman and official repositories

Arch's main package manager, pacman, uses flags instead of command words like yum and apt. For example, to search for a package, you would use pacman -Ss. As with most commands on Linux, you can find both a manpage and inline help. Most of the commands for pacman use the sync (-S) flag. For example:

user@arch ~ $ pacman -Ss kate



extra/kate 18.04.2-2 (kde-applications kdebase)

    Advanced Text Editor

extra/libkate 0.4.1-6 [installed]

    A karaoke and text codec for embedding in ogg

extra/libtiger 0.3.4-5 [installed]

    A rendering library for Kate streams using Pango and Cairo

extra/ttf-cheapskate 2.0-12

    TTFonts collection from dustimo.com

community/haskell-cheapskate 0.1.1-100

    Experimental markdown processor.

Arch also uses repositories similar to other package managers. In the output above, search results are prefixed with the repository they are found in (extra/ and community/ in this case). Similar to both Red Hat and Debian-based systems, Arch relies on the user to add the repository information into a specific file. The location for these repositories is /etc/pacman.conf. The example below is fairly close to a stock system. I have enabled the [multilib] repository for Steam support:

[options]

Architecture = auto



Color

CheckSpace



SigLevel    = Required DatabaseOptional

LocalFileSigLevel = Optional



[core]

Include = /etc/pacman.d/mirrorlist



[extra]

Include = /etc/pacman.d/mirrorlist



[community]

Include = /etc/pacman.d/mirrorlist



[multilib]

Include = /etc/pacman.d/mirrorlist

It is possible to specify a specific URL in pacman.conf. This functionality can be used to make sure all packages come from a specific point in time. If, for example, a package has a bug that affects you severely and it has several dependencies, you can roll back to a specific point in time by adding a specific URL into your pacman.conf and then running the commands to downgrade the system:

[core]

Server=https://archive.archlinux.org/repos/2017/12/22/$repo/os/$arch

Like Debian-based systems, Arch does not update its local repository information until you tell it to do so. You can refresh the package database by issuing the following command:

user@arch ~ $ sudo pacman -Sy 



:: Synchronizing package databases...

 core                                                                    
 130.2 KiB   851K/s 00:00 
[##########################################################] 100%

 extra                                                                  
 1645.3 KiB  2.69M/s 00:01 
[##########################################################] 100%

 community                                                              
   4.5 MiB  2.27M/s 00:02 
[##########################################################] 100%

 multilib is up to date

As you can see in the above output, pacman thinks that the multilib package database is up to date. You can force a refresh if you think this is incorrect by running pacman -Syy. If you want to update your entire system (excluding packages installed from the AUR), you can run pacman -Syu:

user@arch ~ $ sudo pacman -Syu



:: Synchronizing package databases...

 core is up to date

 extra is up to date

 community is up to date

 multilib is up to date

:: Starting full system upgrade...

resolving dependencies...

looking for conflicting packages...



Packages (45) ceph-13.2.0-2  ceph-libs-13.2.0-2  debootstrap-1.0.105-1 
 guile-2.2.4-1  harfbuzz-1.8.2-1  harfbuzz-icu-1.8.2-1 
 haskell-aeson-1.3.1.1-20

              haskell-attoparsec-0.13.2.2-24  haskell-tagged-0.8.6-1 
 imagemagick-7.0.8.4-1  lib32-harfbuzz-1.8.2-1  lib32-libgusb-0.3.0-1 
 lib32-systemd-239.0-1

              libgit2-1:0.27.2-1  libinput-1.11.2-1  libmagick-7.0.8.4-1
 libmagick6-6.9.10.4-1  libopenshot-0.2.0-1  libopenshot-audio-0.1.6-1 
 libosinfo-1.2.0-1

              libxfce4util-4.13.2-1  minetest-0.4.17.1-1 
 minetest-common-0.4.17.1-1  mlt-6.10.0-1  mlt-python-bindings-6.10.0-1 
 ndctl-61.1-1  netctl-1.17-1

              nodejs-10.6.0-1  



Total Download Size:      2.66 MiB

Total Installed Size:   879.15 MiB

Net Upgrade Size:      -365.27 MiB



:: Proceed with installation? [Y/n] 

In the scenario mentioned earlier regarding downgrading a system, you can force a downgrade by issuing pacman -Syyuu. It is important to note that this should not be undertaken lightly. This should not cause a problem in most cases; however, there is a chance that downgrading of a package or several packages will cause a cascading failure and leave your system in an inconsistent state. USE WITH CAUTION!
To install a package, simply use pacman -S kate:

user@arch ~ $ sudo pacman -S kate



resolving dependencies...

looking for conflicting packages...



Packages (7) editorconfig-core-c-0.12.2-1  kactivities-5.47.0-1 
 kparts-5.47.0-1  ktexteditor-5.47.0-2  syntax-highlighting-5.47.0-1 
 threadweaver-5.47.0-1

             kate-18.04.2-2



Total Download Size:   10.94 MiB

Total Installed Size:  38.91 MiB



:: Proceed with installation? [Y/n] 

To remove a package, you can run pacman -R kate. This removes only the package and not its dependencies:

user@arch ~ $ sudo pacman -S kate



checking dependencies...



Packages (1) kate-18.04.2-2



Total Removed Size:  20.30 MiB



:: Do you want to remove these packages? [Y/n]

If you want to remove the dependencies that are not required by other packages, you can run pacman -Rs:

user@arch ~ $ sudo pacman -Rs kate



checking dependencies...



Packages (7) editorconfig-core-c-0.12.2-1  kactivities-5.47.0-1 
 kparts-5.47.0-1  ktexteditor-5.47.0-2  syntax-highlighting-5.47.0-1 
 threadweaver-5.47.0-1

             kate-18.04.2-2



Total Removed Size:  38.91 MiB



:: Do you want to remove these packages? [Y/n] 

Pacman, in my opinion, offers the most succinct way of searching for the name of a package for a given utility. As shown above, yum and apt both rely on pathing in order to find useful results. Pacman makes some intelligent guesses as to which package you are most likely looking for:

user@arch ~ $ sudo pacman -Fs updatedb

core/mlocate 0.26.git.20170220-1

    usr/bin/updatedb



user@arch ~ $ sudo pacman -Fs kate

extra/kate 18.04.2-2

    usr/bin/kate

Working with the AUR

There are several popular AUR package manager helpers. Of these, yaourt and pacaur are fairly prolific. However, both projects are listed as discontinued or problematic on the Arch Wiki. For that reason, I will discuss aurman. It works almost exactly like pacman, except it searches the AUR and includes some helpful, albeit potentially dangerous, options. Installing a package from the AUR will initiate use of the package maintainer's build scripts. You will be prompted several times for permission to continue (I have truncated the output for brevity):

aurman -S telegram-desktop-bin

~~ initializing aurman...

~~ the following packages are neither in known repos nor in the aur

...

~~ calculating solutions...



:: The following 1 package(s) are getting updated:

   aur/telegram-desktop-bin  1.3.0-1  ->  1.3.9-1



?? Do you want to continue? Y/n: Y



~~ looking for new pkgbuilds and fetching them...

Cloning into 'telegram-desktop-bin'...



remote: Counting objects: 301, done.

remote: Compressing objects: 100% (152/152), done.

remote: Total 301 (delta 161), reused 286 (delta 147)

Receiving objects: 100% (301/301), 76.17 KiB | 639.00 KiB/s, done.

Resolving deltas: 100% (161/161), done.

?? Do you want to see the changes of telegram-desktop-bin? N/y: N



[sudo] password for user:



...

==> Leaving fakeroot environment.

==> Finished making: telegram-desktop-bin 1.3.9-1 (Thu 05 Jul 2018 11:22:02 AM EDT)

==> Cleaning up...

loading packages...

resolving dependencies...

looking for conflicting packages...



Packages (1) telegram-desktop-bin-1.3.9-1



Total Installed Size:  88.81 MiB

Net Upgrade Size:       5.33 MiB



:: Proceed with installation? [Y/n]

Sometimes you will be prompted for more input, depending on the complexity of the package you are installing. To avoid this tedium, aurman allows you to pass both the --noconfirm and --noedit options. This is equivalent to saying "accept all of the defaults, and trust that the package maintainers scripts will not be malicious."USE THIS OPTION WITH EXTREME CAUTION! While these options are unlikely to break your system on their own, you should never blindly accept someone else's scripts.

Conclusion

This article, of course, only scratches the surface of what package managers can do. There are also many other package managers available that I could not cover in this space. Some distributions, such as Ubuntu or Elementary OS, have gone to great lengths to provide a graphical approach to package management.
If you are interested in some of the more advanced functions of package managers, please post your questions or comments below and I would be glad to write a follow-up article.

Appendix

# search for packages

yum search 

dnf search 

zypper search 

apt-cache search 

apt search 

pacman -Ss 



# install packages

yum install 

dnf install 

zypper install 

apt-get install 

apt install 

pacman -Ss 



# update package database, not required by yum, dnf and zypper

apt-get update

apt update

pacman -Sy



# update all system packages

yum update

dnf update

zypper update

apt-get upgrade

apt upgrade

pacman -Su



# remove an installed package

yum remove 

dnf remove 

apt-get remove 

apt remove 

pacman -R 

pacman -Rs 



# search for the package name containing specific file or folder

yum whatprovides *

dnf whatprovides *

zypper what-provides 

zypper search --provides 

apt-file search 

pacman -Sf 

Modern computers are ever increasing in performance and capacity. This matters little if that increasing capacity is not well utilized. Following is a description of the motivation and work behind "curt," a new tool for Linux systems for measuring and breaking down system utilization by process, by task, and by CPU using the perf command's Python scripting capabilities.
I had the privilege of presenting this topic at Texas Linux Fest 2018, and here I've gone a bit deeper into the details, included links to further information, and expanded the scope of my talk.

System utilization

In discussing computation, let's begin with some assertions:

1. Every computational system is equally fast at doing nothing.
2. Computational systems were created to do things.
3. A computational system is better at doing things when it is doing something than when it is doing nothing.

Modern computational systems have many streams of execution:

• Often, very large systems are created by literally wiring together smaller systems. At IBM, these smaller systems are sometimes called CECs (short for Central Electronics Complexes and pronounced "keks").
• There are multiple sockets for processor modules in each system.
• There are sometimes multiple chips per socket (in the form of dual-chip modules—DCMs—or multi-chip modules—MCMs).
• There are multiple cores per chip.
• There are multiple threads per core.

In sum, there are potentially thousands of execution threads across a single computational system.
Ideally, all these execution streams are 100% busy doing useful work. One measure of utilization for an individual execution stream (CPU thread) is the percentage of time that thread has tasks scheduled and running. (Note that I didn't say "doing useful work." Creating a tool that measures useful work is left as an exercise for the reader.) By extension, system utilization is the overall percentage of time that all execution streams of a system have tasks scheduled and running. Similarly, utilization can be defined with respect to an individual task. Task utilization is the percentage of the lifetime of the task that was spent actively running on any CPU thread. By extension, process utilization is the collective utilization of its tasks.

Utilization measurement tools

There are tools that measure system utilization: uptime, vmstat, mpstat, nmon, etc. There are tools that measure individual process utilization: time. There are not many tools that measure system-wide per-process and per-task utilization. One such command is curt on AIX. According to IBM's Knowledge Center: "The curt command takes an AIX trace file as input and produces a number of statistics related to processor (CPU) utilization and process/thread/pthread activity."
The AIX curt command reports system-wide, per-processor, per-process, and per-task statistics for application processing (user time), system calls (system time), hypervisor calls, kernel threads, interrupts, and idle time.
This seems like a good model for a similar command for a Linux system.

Utilization data

Before starting to create any tools for utilization analysis, it is important to know what data is required. Since utilization is directly related to whether a task is actively running or not, related scheduling events are required: When is the task made to run, and when is it paused? Tracking on which CPU the task runs is important, so migration events are required for implicit migrations. There are also certain system calls that force explicit migrations. Creation and deletion of tasks are obviously important. Since we want to understand user time, system time, hypervisor time, and interrupt time, events that show the transitions between those task states are required.
The Linux kernel contains "tracepoints" for all those events. It is possible to enable tracing for those events directly in the kernel's debugfs filesystem, usually mounted at /sys/kernel/debug, in the tracing directory (/sys/kernel/debug/tracing).
An easier way to record tracing data is with the Linux perf command.

The perf command

perf is a very powerful userspace command for tracing or counting both hardware and software events.
Software events are predefined in the kernel, can be predefined in userspace code, and can be dynamically created (as "probes") in kernel or userspace code.
perf can do much more than just trace and count, though.

perf stat

The stat subcommand of perf will run a command, count some events commonly found interesting, and produce a simple report:

Performance counter stats for './load 100000':

 

      90537.006424      task-clock:u (msec)       #    1.000 CPUs utilized          

                 0      context-switches:u        #    0.000 K/sec                  

                 0      cpu-migrations:u          #    0.000 K/sec                  

               915      page-faults:u             #    0.010 K/sec                  

   386,836,206,133      cycles:u                  #    4.273 GHz                      (66.67%)

     3,488,523,420      stalled-cycles-frontend:u #    0.90% frontend cycles idle     (50.00%)

   287,222,191,827      stalled-cycles-backend:u  #   74.25% backend cycles idle      (50.00%)

   291,102,378,513      instructions:u            #    0.75  insn per cycle         

                                                  #    0.99  stalled cycles per insn  (66.67%)

    43,730,320,236      branches:u                #  483.010 M/sec                    (50.00%)

       822,030,340      branch-misses:u           #    1.88% of all branches          (50.00%)

 

      90.539972837 seconds time elapsed

perf record, perf report, and perf annotate

For much more interesting analysis, the perf command can also be used to record events and information associated with the task state at the time the event occurred:

$ perf record ./some-command

[ perf record: Woken up 55 times to write data ]

[ perf record: Captured and wrote 13.973 MB perf.data (366158 samples) ]

$ perf report --stdio --show-nr-samples --percent-limit 4

# Samples: 366K of event 'cycles:u'

# Event count (approx.): 388851358382

#

# Overhead       Samples  Command  Shared Object      Symbol                                          

# ........  ............  .......  .................  ................................................

#

    62.31%        228162  load     load               [.] main

    19.29%         70607  load     load               [.] sum_add

    18.33%         67117  load     load               [.] sum_sub

This example shows a program that spends about 60% of its running time in the function main and about 20% each in subfunctions sum_sub and sum_add. Note that the default event used by perf record is "cycles." Later examples will show how to use perf record with other events.
perf report can further report runtime statistics by source code line (if the compilation was performed with the -g flag to produce debug information):

$ perf report --stdio --show-nr-samples --percent-limit 4 --sort=srcline

# Samples: 366K of event 'cycles:u'

# Event count (approx.): 388851358382

#

# Overhead       Samples  Source:Line                        

# ........  ............  ...................................

#

    19.40%         71031  load.c:58

    16.16%         59168  load.c:18

    15.11%         55319  load.c:14

    13.30%         48690  load.c:66

    13.23%         48434  load.c:70

     4.58%         16767  load.c:62

     4.01%         14677  load.c:56

Further, perf annotate can show statistics for each instruction of the program:

$ perf annotate --stdio

Percent |      Source code & Disassembly of load for cycles:u (70607 samples)

------------------------------------------------------------------------------

         :      0000000010000774 :

         :      int sum_add(int sum, int value) {

   12.60 :        10000774:   std     r31,-8(r1)

    0.02 :        10000778:   stdu    r1,-64(r1)

    0.00 :        1000077c:   mr      r31,r1

   41.90 :        10000780:   mr      r10,r3

    0.00 :        10000784:   mr      r9,r4

    0.05 :        10000788:   stw     r10,32(r31)

   23.78 :        1000078c:   stw     r9,36(r31)

         :              return (sum + value);

    0.76 :        10000790:   lwz     r10,32(r31)

    0.00 :        10000794:   lwz     r9,36(r31)

   14.75 :        10000798:   add     r9,r10,r9

    0.00 :        1000079c:   extsw   r9,r9

         :      }

    6.09 :        100007a0:   mr      r3,r9

    0.02 :        100007a4:   addi    r1,r31,64

    0.03 :        100007a8:   ld      r31,-8(r1)

    0.00 :        100007ac:   blr

(Note: this code is not optimized.)

perf top

Similar to the top command, which displays (at a regular update interval) the processes using the most CPU time, perf top will display the functions using the most CPU time among all processes on the system, a nice leap in granularity.

perf list

The examples thus far have used the default event, run cycles. There are hundreds and perhaps thousands of events of different types. perf list will show them all. Following are just a few examples:

$ perf list

  instructions                                       [Hardware event]

  context-switches OR cs                             [Software event]

  L1-icache-loads                                    [Hardware cache event]

  mem_access OR cpu/mem_access/                      [Kernel PMU event]

cache:

  pm_data_from_l2                                   

       [The processor's data cache was reloaded from local core's L2 due to a demand load]

floating point:

  pm_fxu_busy                                       

       [fxu0 busy and fxu1 busy]

frontend:

  pm_br_mpred_cmpl                                  

       [Number of Branch Mispredicts]

memory:

  pm_data_from_dmem                                 

       [The processor's data cache was reloaded from another chip's 
memory on the same Node or Group (Distant) due to a demand load]

  pm_data_from_lmem                                 

       [The processor's data cache was reloaded from the local chip's Memory due to a demand load]

  rNNN                                               [Raw hardware event descriptor]

  raw_syscalls:sys_enter                             [Tracepoint event]

  syscalls:sys_enter_chmod                           [Tracepoint event]

  sdt_libpthread:pthread_create                      [SDT event]

Events labeled as Hardware event, Hardware cache event, Kernel PMU event, and most (if not all) of the events under the categories like cache, floating point, frontend, and memory are hardware events counted by the hardware and triggered each time a certain count is reached. Once triggered, an entry is made into the kernel trace buffer with the current state of the associated task. Raw hardware event codes are alphanumeric encodings of the hardware events. These are mostly needed when the hardware is newer than the kernel and the user needs to enable events that are new for that hardware. Users will rarely, if ever, need to use raw event codes.
Events labeled Tracepoint event are embedded in the kernel. These are triggered when that section of code is executed by the kernel. There are "syscalls" events for every system call supported by the kernel. raw_syscalls events are triggered for every system call. Since there is a limit to the number of events being actively traced, the raw_syscalls events may be more practical when a large number of system calls need to be traced.
Events labeled SDT event are for software-defined tracepoints (SDTs). These can be embedded in application or library code and enabled as needed. When enabled, they behave just like other events: When that section of code is executed (by any task being traced on the system), an entry is made in the kernel trace buffer with the current state of the associated task. This is a very powerful capability that can prove very useful.

perf buildid-cache and perf probe

Enabling SDTs is easy. First, make the SDTs for a certain library known to perf:

$ perf buildid-cache -v --add /lib/powerpc64le-linux-gnu/libpthread.so.0

$ perf list | grep libpthread

[…]

  sdt_libpthread:pthread_create                      [SDT event]

[…]

Then, turn SDT definitions into available tracepoints:

$ /usr/bin/sudo perf probe sdt_libpthread:pthread_create

Added new event:

  sdt_libpthread:pthread_create (on %pthread_create in /lib/powerpc64le-linux-gnu/libpthread-2.27.so)

You can now use it in all perf tools, such as:

    perf record -e sdt_libpthread:pthread_create -aR sleep 1

$ perf record -a -e sdt_libpthread:pthread_create ./test

[ perf record: Woken up 1 times to write data ]

[ perf record: Captured and wrote 0.199 MB perf.data (9 samples) ]

Note that any location in an application or library can be made into a tracepoint. To find functions in an application that can be made into tracepoints, use perf probe with –funcs:

$ perf probe –x ./load --funcs

[…]

main

sum_add

sum_sub

To enable the function main of the ./load application as a tracepoint:

/usr/bin/sudo perf probe –x ./load main

Added new event:

  probe_load:main      (on main in /home/pc/projects/load-2.1pc/load)

You can now use it in all perf tools, such as:

    perf record –e probe_load:main –aR sleep 1

$ perf list | grep load:main

  probe_load:main                                     [Tracepoint event]

$ perf record –e probe_load:main ./load

[ perf record: Woken up 1 times to write data ]

[ perf record: Captured and wrote 0.024 MB perf.data (1 samples) ]

perf script

Continuing the previous example, perf script can be used to walk through the perf.data file and output the contents of each record:

$ perf script

            Load 16356 [004] 80526.760310: probe_load:main: (4006a2)

Processing perf trace data

The preceding discussion and examples show that perf can collect the data required for system utilization analysis. However, how can that data be processed to produce the desired results?

perf eBPF

A relatively new and emerging technology with perf is called eBPF. BPF is an acronym for Berkeley Packet Filter, and it is a C-like language originally for, not surprisingly, network packet filtering in the kernel. eBPF is an acronym for extended BPF, a similar, but more robust C-like language based on BPF.
Recent versions of perf can be used to incorporate compiled eBPF code into the kernel to securely and intelligently handle events for any number of purposes, with some limitations.
The capability is very powerful and quite useful for real-time, continuous updates of event-related data and statistics.
However, as this capability is emerging, support is mixed on current releases of Linux distributions. It's a bit complicated (or, put differently, I have not figured it out yet). It's also only for online use; there is no offline capability. For these reasons, I won't cover it further here.

perf data file

perf record produces a perf.data file. The file is a structured binary file, is not particularly well documented, has no programming interface for access, and is unclear on what compatibility guarantees exist. For these reasons, I chose not to directly use the perf.data file.

perf script

One of the last examples above showed how perf script is used for walking through the perf.data file and emitting basic information about each record there. This is an appropriate model for what would be needed to process the file and track the state changes and compute the statistics required for system utilization analysis.
perf script has several modes of operation, including several higher-level scripts that come with perf that produce statistics based on the trace data in a perf.data file.

$ perf script -l

List of available trace scripts:

  rw-by-pid                            system-wide r/w activity

  rwtop [interval]                     system-wide r/w top

  wakeup-latency                       system-wide min/max/avg wakeup latency

  failed-syscalls [comm]               system-wide failed syscalls

  rw-by-file                    r/w activity for a program, by file

  failed-syscalls-by-pid [comm]        system-wide failed syscalls, by pid

  intel-pt-events                      print Intel PT Power Events and PTWRITE

  syscall-counts-by-pid [comm]         system-wide syscall counts, by pid

  export-to-sqlite [database name] [columns] [calls] export perf data to a sqlite3 database

  futex-contention                     futext contention measurement

  sctop [comm] [interval]              syscall top

  event_analyzing_sample               analyze all perf samples

  net_dropmonitor                      display a table of dropped frames

  compaction-times [-h] [-u] [-p|-pv] [-t | [-m] [-fs] [-ms]] [pid|pid-range|comm-regex] display time taken by mm compaction

  export-to-postgresql [database name] [columns] [calls] export perf data to a postgresql database

  stackcollapse                        produce callgraphs in short form for scripting use

  netdev-times [tx] [rx] [dev=] [debug] display a process of packet and processing time

  syscall-counts [comm]                system-wide syscall counts

  sched-migration                      sched migration overview

$ perf script failed-syscalls-by-pid /bin/ls

 

syscall errors:

 

comm [pid]                           count

------------------------------  ----------

 

ls [18683]

  syscall: access          

    err = ENOENT                         1

  syscall: statfs          

    err = ENOENT                         1

  syscall: ioctl           

    err = ENOTTY                         3

What do these scripts look like? Let's find out.

$ locate failed-syscalls-by-pid

/usr/libexec/perf-core/scripts/python/failed-syscalls-by-pid.py

[…]

$ rpm –qf /usr/libexec/perf-core/scripts/python/failed-syscalls-by-pid.py

perf-4.14.0-46.el7a.x86_64

$ $ ls /usr/libexec/perf-core/scripts

perl  python

$ perf script -s lang

 

Scripting language extensions (used in perf script -s [spec:]script.[spec]):

 

  Perl                                       [Perl]

  pl                                         [Perl]

  Python                                     [Python]

  py                                         [Python]

So, these scripts come with perf, and both Python and Perl are supported languages.
Note that for the entirety of this content, I will refer exclusively to Python.

perf scripts

How do these scripts do what they do? Here are important extracts from /usr/libexec/perf-core/scripts/python/failed-syscalls-by-pid.py:

def raw_syscalls__sys_exit(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain,id, ret):

[…]

        if ret <0:

[…]

                        syscalls[common_comm][common_pid][id][ret] +=1

The function raw_syscalls__sys_exit has parameters for all the data for the associated event. The rest of the function only increments a counter associated with the command, process ID, and system call. The rest of the code doesn't do that much. Most of the complexity is in the function signature for the event-handling routine.
Fortunately, perf makes it easy to figure out the proper signatures for various tracepoint event-handling functions.

perf script –gen-script

For the raw_syscalls events, we can generate a trace containing just those events:

$ perf list | grep raw_syscalls

  raw_syscalls:sys_enter                             [Tracepoint event]

  raw_syscalls:sys_exit                              [Tracepoint event]

$ perf record -e 'raw_syscalls:*' /bin/ls >/dev/null

[ perf record: Woken up 1 times to write data ]

[ perf record: Captured and wrote 0.025 MB perf.data (176 samples) ]

We can then have perf generate a script that contains sample implementations of event-handling functions for the events in the perf.data file:

$ perf script --gen-script python

generated Python script: perf-script.py

What do we find in the script?

def raw_syscalls__sys_exit(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain,id, ret):

[…]

def raw_syscalls__sys_enter(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain,id, args):

Both event-handling functions are specified with their signatures. Nice!
Note that this script works with perf script –s:

$ perf script -s ./perf-script.py

in trace_begin

raw_syscalls__sys_exit     7 94571.445908134    21117 ls                    id=0, ret=0

raw_syscalls__sys_enter     7 94571.445942946    21117 ls                    id=45, args=���?bc���?�

[…]

Now we have a template on which to base writing a Python script to parse the events of interest for reporting system utilization.

perf scripting

The Python scripts generated by perf script –gen-script are not directly executable. They must be invoked by perf:

$ perf script –s ./perf-script.py

What's really going on here?

1. First, perf starts. The script subcommand's -s option indicates that an external script will be used.
2. perf establishes a Python runtime environment.
3. perf loads the specified script.
4. perf runs the script. The script can perform normal initialization and even handle command line arguments, although passing the arguments is slightly awkward, requiring a -- separator between the arguments for perf and for the script:

   $ perf script -s ./perf-script.py -- --script-arg1 [...]

5. perf processes each record of the trace file, calling the appropriate event-handling function in the script. Those event-handling functions can do whatever they need to do.

Utilization

It appears that perf scripting has sufficient capabilities for a workable solution. What sort of information is required to generate the statistics for system utilization?

• Task creation (fork, pthread_create)
• Task termination (exit)
• Task replacement (exec)
• Task migration, explicit or implicit, and current CPU
• Task scheduling
• System calls
• Hypervisor calls
• Interrupts

It can be helpful to understand what portion of time a task spends in various system calls, handling interrupts, or making explicit calls out to the hypervisor. Each of these categories of time can be considered a "state" for the task, and the methods of transitioning from one state to another need to be tracked:
The most important point of the diagram is that there are events for each state transition.

• Task creation: clone system call
• Task termination: sched:sched_process_exit
• Task replacement: sched:sched_process_exec
• Task migration: sched_setaffinity system call (explicit), sched:sched_migrate_task (implicit)
• Task scheduling: sched:sched_switch
• System calls: raw_syscalls:sys_enter, raw_syscalls:sys_exit
• Hypervisor calls: (POWER-specific) powerpc:hcall_entry, powerpc:hcall_exit
• Interrupts: irq:irq_handler_entry, irq:irq_handler_exit

The curt command for Linux

perf provides a suitable infrastructure with which to capture the necessary data for system utilization. There are a sufficient set of events available for tracing in the Linux kernel. The Python scripting capabilities permit a powerful and flexible means of processing the trace data. It's time to write the tool.

High-level design

In processing each event, the relevant state of the affected tasks must be updated:

• New task? Create and initialize data structures to track the task's state
  • Command
  • Process ID
  • Task ID
  • Migration count (0)
  • Current CPU
• New CPU for this task? Create and initialize data structures for CPU-specific data
  • User time (0)
  • System time (0)
  • Hypervisor time (0)
  • Interrupt time (0)
  • Idle time (0)
• New transaction for this task? Create and initialize data structures for transaction-specific data
  • Elapsed time (0)
  • Count (0)
  • Minimum (maxint), maximum (0)
• Existing task?
  • Accumulate time for the previous state
  • Transaction ending? Accumulate time for the transaction, adjust minimum, maximum values
• Set new state
• Save current time (time current state entered)
• Migration? Increment migration count

High-level example

For a raw_syscalls:sys_enter event:

• If this task has not been seen before, allocate and initialize a new task data structure
• If the CPU is new for this task, allocate and initialize a new CPU data structure
• If this system call is new for this task, allocate and initialize a new call data structure
• In the task data structure:
  • Accumulate the time since the last state change in a bucket for the current state ("user")
  • Set the new state ("system")
  • Save the current timestamp as the start of this time period for the new state

Edge cases

sys_exit as a task's first event

If the first event in the trace for a task is raw_syscalls:sys_exit:

• There is no matching raw_syscalls:sys_enter with which to determine the start time of this system call.
• The accumulated time since the start of the trace was all spent in the system call and needs to be added to the overall elapsed time spent in all calls to this system call.
• The elapsed time of this system call is unknown.
• It would be inaccurate to account for this elapsed time in the average, minimum, or maximum statistics for this system call.

In this case, the tool creates a separate bucket called "pending" for time spent in the system call that cannot be accounted for in the average, minimum, or maximum.
A "pending" bucket is required for all transactional events (system calls, hypervisor calls, and interrupts).

sys_enter as a task's last event

Similarly, If the last event in the trace for a task is raw_syscalls:sys_enter:

• There is no matching raw_syscalls:sys_exit with which to determine the end time of this system call.
• The accumulated time from the start of the system call to the end of the trace was all spent in the system call and needs to be added to the overall elapsed time spent in all calls to this system call.
• The elapsed time of this system call is unknown.
• It would be inaccurate to account for this elapsed time in the average, minimum, or maximum statistics for this system call.

This elapsed time is also accumulated in the "pending" bucket.
A "pending" bucket is required for all transactional events (system calls, hypervisor calls, and interrupts).
Since this condition can only be discovered at the end of the trace, a final "wrap-up" step is required in the tool where the statistics for all known tasks are completed based on their final states.

Indeterminable state

It is possible that a very busy task (or a short trace) will never see an event for a task from which the task's state can be determined. For example, if only sched:sched_switch or sched:sched_task_migrate events are seen for a task, it is impossible to determine that task's state. However, the task is known to exist and to be running.
Since the actual state cannot be determined, the runtime for the task is accumulated in a separate bucket, arbitrarily called "busy-unknown." For completeness, this time is also displayed in the final report.

Invisible tasks

For very, very busy tasks (or a short trace), it is possible that a task was actively running during the entire time the trace was being collected, but no events for that task appear in the trace. It was never migrated, paused, or forced to wait.
Such tasks cannot be known to exist by the tool and will not appear in the report.

curt.py Python classes

Task

• One per task
• Holds all task-specific data (command, process ID, state, CPU, list of CPU data structures [see below], migration count, lists of per-call data structures [see below])
• Maintains task state

Call

• One per unique transaction, per task (for example, one for the "open" system call, one for the "close" system call, one for IRQ 27, etc.)
• Holds call-specific data (e.g., start timestamp, count, elapsed time, minimum, maximum)
• Allocated as needed (lazy allocation)
• Stored within a task in a Python dictionary indexed by the unique identifier of the call (e.g., system call code, IRQ number, etc.)

CPU

• One per CPU on which this task has been observed to be running
• Holds per-CPU task data (e.g., user time, system time, hypervisor call time, interrupt time)
• Allocated as needed (lazy allocation)
• Stored within a task in a Python dictionary indexed by the CPU number

curt.py event processing example

As previously discussed, perf script will iterate over all events in the trace and call the appropriate event-handling function for each event.
A first attempt at an event-handling function for sys_exit, given the high-level example above, might be:

tasks ={}



def raw_syscalls__sys_enter(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, args):

 

  # convert the multiple timestamp values into a single value

  timestamp = nsecs(common_secs, common_nsecs)



  # find this task's data structure

  try:

    task = tasks[common_pid]

  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task



  if common_cpu notin task.cpus:

    # new CPU!

    task.cpu= common_cpu

    task.cpus[common_cpu]= CPU()



  # compute time spent in the previous state ('user')

  delta = timestamp – task.timestamp

  # accumulate 'user' time for this task/CPU

  task.cpus[task.cpu].user += delta

  ifidnotin task.syscalls:

    # new system call for this task!

    task.syscalls[id]= Call()



  # change task's state

  task.mode='sys'



  # save the timestamp for the last event (this one) for this task

  task.timestamp= timestamp



def raw_syscalls__sys_exit(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, ret):



  # convert the multiple timestamp values into a single value

  timestamp = nsecs(common_secs, common_nsecs)



  # get the task data structure

  task = tasks[common_pid]



  # compute elapsed time for this system call

  delta = task.timestamp - timestamp



  # accumulate time for this task/system call

  task.syscalls[id].elapsed += delta

  # increment the tally for this task/system call

  task.syscalls[id].count +=1

  # adjust statistics

  if delta < task.syscalls[id].min:

    task.syscalls[id].min= delta

  if delta > task.syscalls[id].max:

    task.syscalls[id].max= delta



  # accumulate time for this task's state on this CPU

  task.cpus[common_cpu].system += delta



  # change task's state

  task.mode='user'



  # save the timestamp for the last event (this one) for this task

  task.timestamp= timestamp

Handling the edge cases

Following are some of the edge cases that are possible and must be handled.

Sys_exit as first event

As a system-wide trace can be started at an arbitrary time, it is certainly possible that the first event for a task is raw_syscalls:sys_exit. This requires adding the same code for new task discovery from the event-handling function for raw_syscalls:sys_enter to the handler for raw_syscalls:sys_exit. This:

  # get the task data structure

  task = tasks[common_pid]

becomes this:

  # find this task's data structure

  try:

    task = tasks[common_pid]

  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task

Another issue is that it is impossible to properly accumulate the data for this system call since there is no timestamp for the start of the system call. The time from the start of the trace until this event has been spent by this task in the system call. It would be inaccurate to ignore this time. It would also be inaccurate to incorporate this time such that it is used to compute the average, minimum, or maximum. The only reasonable option is to accumulate this separately, calling it "pending" system time. To accurately compute this time, the timestamp of the first event of the trace must be known. Since any event could be the first event in the trace, every event must conditionally save its timestamp if it is the first event. A global variable is required:

start_timestamp =0

And every event-handling function must conditionally save its timestamp:

  # convert the multiple timestamp values into a single value

  timestamp = nsecs(common_secs, common_nsecs)



  If start_timestamp =0:

    start_timestamp = timestamp

So, the event-handling function for raw_syscalls:sys_exit becomes:

def raw_syscalls__sys_exit(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, ret):



  # convert the multiple timestamp values into a single value

  timestamp = nsecs(common_secs, common_nsecs)



  If start_timestamp =0:

    start_timestamp = timestamp



  # find this task's data structure

  try:

    task = tasks[common_pid]



    # compute elapsed time for this system call

    delta = task.timestamp - timestamp



    # accumulate time for this task/system call

    task.syscalls[id].elapsed += delta

    # increment the tally for this task/system call

    task.syscalls[id].count +=1

    # adjust statistics

    if delta < task.syscalls[id].min:

      task.syscalls[id].min= delta

    if delta > task.syscalls[id].max:

      task.syscalls[id].max= delta



  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task



    # compute elapsed time for this system call

    delta = start_timestamp - timestamp



    # accumulate time for this task/system call

    task.syscalls[id].pending += delta



  # accumulate time for this task's state on this CPU

  task.cpus[common_cpu].system += delta



  # change task's state

  task.mode='user'



  # save the timestamp for the last event (this one) for this task

  task.timestamp= timestamp

Sys_enter as last event

A similar issue to having sys_exit as the first event for a task is when sys_enter is the last event seen for a task. The time spent in the system call must be accumulated for completeness but can't accurately impact the average, minimum, nor maximum. This time will also be accumulated in for a separate "pending" state.
To accurately determine the elapsed time of the pending system call, from sys_entry to the end of the trace period, the timestamp of the final event in the trace file is required. Unfortunately, there is no way to know which event is the last event until that event has already been processed. So, all events must save their respective timestamps in a global variable.
It may be that many tasks are in the state where the last event seen for them was sys_enter. Thus, after the last event is processed, a final "wrap up" step is required to complete the statistics for those tasks. Fortunately, there is a trace_end function which is called by perf after the final event has been processed.
Last, we need to save the id of the system call in everysys_enter.

curr_timestamp =0



def raw_syscalls__sys_enter(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, args):



  # convert the multiple timestamp values into a single value 

  curr_timestamp = nsecs(common_secs, common_nsecs)

[…]

  task.syscall=id

[…]



def trace_end(): 

        for tid in tasks.keys(): 

                task = tasks[tid]

                # if this task ended while executing a system call

                if task.mode=='sys':

                        # compute the time from the entry to the system call to the end of the trace period

                        delta = curr_timestamp - task.timestamp

                        # accumulate the elapsed time for this system call

                        task.syscalls[task.syscall].pending += delta

                        # accumulate the system time for this task/CPU

                        task.cpus[task.cpu].sys += delta

Migrations

A task migration is when a task running on one CPU is moved to another CPU. This can happen by either:

1. Explicit request (e.g., a call to sched_setaffinity), or
2. Implicitly by the kernel (e.g., load balancing or vacating a CPU being taken offline)

When detected:

• The migration count for the task should be incremented
• The statistics for the previous CPU should be updated
• A new CPU data structure may need to be updated and initialized if the CPU is new for the task
• The task's current CPU is set to the new CPU

For accurate statistics, task migrations must be detected as soon as possible. The first case, explicit request, happens within a system call and can be detected in the sys_exit event for that system call. The second case has its own event, sched:sched_migrate_task, so it will need a new event-handling function.

def raw_syscalls__sys_exit(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, ret):



  # convert the multiple timestamp values into a single value

  timestamp = nsecs(common_secs, common_nsecs)



  If start_timestamp =0:

    start_timestamp = timestamp



  # find this task's data structure

  try:

    task = tasks[common_pid]



    # compute elapsed time for this system call

    delta = task.timestamp - timestamp



    # accumulate time for this task/system call

    task.syscalls[id].elapsed += delta

    # increment the tally for this task/system call

    task.syscalls[id].count +=1

    # adjust statistics

    if delta < task.syscalls[id].min:

      task.syscalls[id].min= delta

    if delta > task.syscalls[id].max:

      task.syscalls[id].max= delta



  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task



    task.cpu= common_cpu



    # compute elapsed time for this system call

    delta = start_timestamp - timestamp



    # accumulate time for this task/system call

    task.syscalls[id].pending += delta



  If common_cpu != task.cpu:

    task.migrations +=1

    # divide the time spent in this syscall in half...

    delta /=2

    # and give have to the previous CPU, below, and half to the new CPU, later

    task.cpus[task.cpu].system += delta



  # accumulate time for this task's state on this CPU

  task.cpus[common_cpu].system += delta



  # change task's state

  task.mode='user'



  # save the timestamp for the last event (this one) for this task

  task.timestamp= timestamp



def sched__sched_migrate_task(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain, comm, pid, prio, orig_cpu,

        dest_cpu, perf_sample_dict):



  If start_timestamp =0:

    start_timestamp = timestamp



  # find this task's data structure

  try:

    task = tasks[common_pid]

  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task



    task.cpu= common_cpu



    If common_cpu notin task.cpus:

      task.cpus[common_cpu]= CPU()



    task.migrations +=1

Task creation

To accurately collect statistics for a task, it is essential to know when the task is created. Tasks can be created with fork(), which creates a new process, or pthread_create(), which creates a new task within the same process. Fortunately, both are manifested by a clone system call and made evident by a sched:sched_process_fork event. The lifetime of the task starts at the sched_process_fork event. The edge case that arises is that the first likely events for the new task are:

1. sched_switch when the new task starts running. The new task should be considered idle at creation until this event occurs
2. sys_exit for the clone system call. The initial state of the new task needs to be based on the state of the task that creates it, including being within the clone system call.

One edge case that must be handled is if the creating task (parent) is not yet known, it must be created and initialized, and the presumption is that it has been actively running since the start of the trace.

def sched__sched_process_fork(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain, parent_comm, parent_pid, child_comm, child_pid):

  global start_timestamp, curr_timestamp

  curr_timestamp =self.timestamp

  if(start_timestamp ==0):

    start_timestamp = curr_timestamp

  # find this task's data structure

  try:

    task = tasks[common_pid]

  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task

  try:

    parent = tasks[self.parent_tid]

  except:

    # need to create parent task here!

    parent = Task(start_timestamp,self.command,'sys',self.pid)

    parent.sched_stat=True# ?

    parent.cpu=self.cpu

    parent.cpus[parent.cpu]= CPU()

    tasks[self.parent_tid]= parent

 

    task.resume_mode= parent.mode

    task.syscall= parent.syscall

    task.syscalls[task.syscall]= Call()

    task.syscalls[task.syscall].timestamp=self.timestamp

Task exit

Similarly, for complete and accurate task statistics, it is essential to know when a task has terminated. There's an event for that: sched:sched_process_exit. This one is pretty easy to handle, in that the effort is just to close out the statistics and set the mode appropriately, so any end-of-trace processing will not think the task is still active:

def sched__sched_process_exit_old(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain, comm, pid, prio):

  global start_timestamp, curr_timestamp

  curr_timestamp =self.timestamp

  if(start_timestamp ==0):

    start_timestamp = curr_timestamp



  # find this task's data structure

  try:

    task = tasks[common_pid]

  except:

    # new task!

    task = Task()

    # save the command string

    task.comm= common_comm

    task.timestamp= curr_timestamp

    # save the new task in the global list (dictionary) of tasks

    tasks[common_pid]= task



  delta = timestamp – task.timestamp

  task.sys += delta

  task.mode='exit'

Output

What follows is an example of the report displayed by curt, slightly reformatted to fit on a narrower page width and with the idle-time classification data (which makes the output very wide) removed, and for brevity. Seen are two processes, 1497 and 2857. Process 1497 has two tasks, 1497 and 1523. Each task has a per-CPU summary and system-wide ("ALL" CPUs) summary. Each task's data is followed by the system call data for that task (if any), hypervisor call data (if any), and interrupt data (if any). After each process's respective tasks is a per-process summary.  Process 2857 has a task 2857-0 that is the previous task image before an exec() system call replaced the process image. After all processes is a system-wide summary.

1497:

-- [  task] command     cpu      user       sys       irq        hv      busy      idle |  util% moves

   [  1497] X             2  0.076354  0.019563  0.000000  0.000000  0.000000 15.818719 |   0.6%

   [  1497] X           ALL  0.076354  0.019563  0.000000  0.000000  0.000000 15.818719 |   0.6%     0

 

  -- ( ID)name             count   elapsed      pending      average      minimum      maximum

     (  0)read                 2  0.004699     0.000000     0.002350     0.002130     0.002569

     (232)epoll_wait           1  9.968375     5.865208     9.968375     9.968375     9.968375

 

-- [  task] command     cpu      user       sys       irq        hv      busy      idle |  util% moves

   [  1523] InputThread   1  0.052598  0.037073  0.000000  0.000000  0.000000 15.824965 |   0.6%

   [  1523] InputThread ALL  0.052598  0.037073  0.000000  0.000000  0.000000 15.824965 |   0.6%     0

 

  -- ( ID)name             count   elapsed      pending      average      minimum      maximum

     (  0)read                14  0.011773     0.000000     0.000841     0.000509     0.002185

     (  1)write                2  0.010763     0.000000     0.005381     0.004974     0.005789

     (232)epoll_wait           1  9.966649     5.872853     9.966649     9.966649     9.966649

 

-- [  task] command     cpu      user       sys       irq        hv      busy      idle |  util% moves

   [   ALL]             ALL  0.128952  0.056636  0.000000  0.000000  0.000000 31.643684 |   0.6%     0

 

2857:

-- [  task] command     cpu      user       sys       irq        hv      busy      idle |  util% moves

   [  2857] execs.sh      1  0.257617  0.249685  0.000000  0.000000  0.000000  0.266200 |  65.6%

   [  2857] execs.sh      2  0.000000  0.023951  0.000000  0.000000  0.000000  0.005728 |  80.7%

   [  2857] execs.sh      5  0.313509  0.062271  0.000000  0.000000  0.000000  0.344279 |  52.2%

   [  2857] execs.sh      6  0.136623  0.128883  0.000000  0.000000  0.000000  0.533263 |  33.2%

   [  2857] execs.sh      7  0.527347  0.194014  0.000000  0.000000  0.000000  0.990625 |  42.1%

   [  2857] execs.sh    ALL  1.235096  0.658804  0.000000  0.000000  0.000000  2.140095 |  46.9%     4

 

  -- ( ID)name             count   elapsed      pending      average      minimum      maximum

     (  9)mmap                15  0.059388     0.000000     0.003959     0.001704     0.017919

     ( 14)rt_sigprocmask      12  0.006391     0.000000     0.000533     0.000431     0.000711

     (  2)open                 9  2.253509     0.000000     0.250390     0.008589     0.511953

     (  3)close                9  0.017771     0.000000     0.001975     0.000681     0.005245

     (  5)fstat                9  0.007911     0.000000     0.000879     0.000683     0.001182

     ( 10)mprotect             8  0.052198     0.000000     0.006525     0.003913     0.018073

     ( 13)rt_sigaction         8  0.004281     0.000000     0.000535     0.000458     0.000751

     (  0)read                 7  0.197772     0.000000     0.028253     0.000790     0.191028

     ( 12)brk                  5  0.003766     0.000000     0.000753     0.000425     0.001618

     (  8)lseek                3  0.001766     0.000000     0.000589     0.000469     0.000818

 

-- [  task] command     cpu      user       sys       irq        hv      busy      idle |  util% moves

   [2857-0] perf          6  0.053925  0.191898  0.000000  0.000000  0.000000  0.827263 |  22.9%

   [2857-0] perf          7  0.000000  0.656423  0.000000  0.000000  0.000000  0.484107 |  57.6%

   [2857-0] perf        ALL  0.053925  0.848321  0.000000  0.000000  0.000000  1.311370 |  40.8%     1

 

  -- ( ID)name             count   elapsed      pending      average      minimum      maximum

     (  0)read                 0  0.000000     0.167845           --           --           --

     ( 59)execve               0  0.000000     0.000000           --           --           --

 

ALL:

-- [  task] command     cpu      user       sys       irq        hv      busy      idle |  util% moves

   [   ALL]             ALL 10.790803 29.633170  0.160165  0.000000  0.137747 54.449823 |   7.4%    50

 

  -- ( ID)name             count   elapsed      pending      average      minimum      maximum

     (  1)write             2896  1.623985     0.000000     0.004014     0.002364     0.041399

     (102)getuid            2081  3.523861     0.000000     0.001693     0.000488     0.025157

     (142)sched_setparam     691  7.222906    32.012841     0.024925     0.002024     0.662975

     ( 13)rt_sigaction       383  0.235087     0.000000     0.000614     0.000434     0.014402

     (  8)lseek              281  0.169157     0.000000     0.000602     0.000452     0.013404

     (  0)read               133  2.782795     0.167845     0.020923     0.000509     1.864439

     (  7)poll                96  8.583354   131.889895     0.193577     0.000626     4.596280

     (  4)stat                93  7.036355     1.058719     0.183187     0.000981     3.661659

     ( 47)recvmsg             85  0.146644     0.000000     0.001725     0.000646     0.019067

     (  3)close               79  0.171046     0.000000     0.002165     0.000428     0.020659

     (  9)mmap                78  0.311233     0.000000     0.003990     0.001613     0.017919

     (186)gettid              74  0.067315     0.000000     0.000910     0.000403     0.014075

     (  2)open                71  3.081589     0.213059     0.184248     0.001921     0.937946

     (202)futex               62  5.145112   164.286154     0.405566     0.000597    11.587437

 

  -- ( ID)name             count   elapsed      pending      average      minimum      maximum

     ( 12)i8042               10  0.160165     0.000000     0.016016     0.010920     0.032805

 

Total Trace Time: 15.914636 ms

Hurdles and issues

Following are some of the issues encountered in the development of curt.

Out-of-order events

One of the more challenging issues is the discovery that events in a perf.data file can be out of time order. For a program trying to monitor state transitions carefully, this is a serious issue. For example, a trace could include the following sequence of events, displayed as they appear in the trace file:

time 0000:  sys_enter syscall1

time 0007:  sys_enter syscall2

time 0006:  sys_exit syscall1

time 0009:  sys_exit syscall2

Just blindly processing these events in the order they are presented to their respective event-handling functions (in the wrong time order) will result in incorrect statistics (or worse).
The most user-friendly ways to handle out-of-order events include:

• Prevent traces from having out-of-order events in the first place by changing the way perf record works
• Providing a means to reorder events in a trace file, perhaps by enhancing perf inject
• Modifying how perf script works to present the events to the event-handling functions in time order

But user-friendly is not the same as straightforward, nor easy. Also, none of the above are in the user's control.
I chose to implement a queue for incoming events that would be sufficiently deep to allow for proper reordering of all events. This required a significant redesign of the code, including implementation of classes for each event, and moving the event processing for each event type into a method in that event's class.
In the redesigned code, the actual event handlers' only job is to save the relevant data from the event into an instance of the event class, queue it, then process the top (oldest in time) event from the queue:

def raw_syscalls__sys_enter(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, args):

         event = Event_sys_enter(nsecs(common_secs,common_nsecs), common_cpu, common_pid, common_comm,id)

        process_event(event)

The simple reorderable queuing mechanism is in a common function:

events =[]

n_events =0

def process_event(event):

        global events,n_events,curr_timestamp

        i = n_events

        while i >0and events[i-1].timestamp> event.timestamp:

                i = i-1

        events.insert(i,event)

        if n_events < params.window:

                n_events = n_events+1

        else:

                event = events[0]

                # need to delete from events list now,

                # because event.process() could reenter here

                del events[0]

                if event.timestamp< curr_timestamp:

                        sys.stderr.write("Error: OUT OF ORDER events detected.\n Try increasing the size of the look-ahead window with --window=\n")

                event.process()

Note that the size of the queue is configurable, primarily for performance and to limit memory consumption. The function will report when that queue size is insufficient to eliminate out-of-order events. It is worth considering whether to consider this case a catastrophic failure and elect to terminate the program.
Implementing a class for each event type led to some consideration for refactoring, such that common code could coalesce into a base class:

class Event (object):

 

        def__init__(self):

                self.timestamp=0

                self.cpu=0

                self.tid=0

                self.command='unknown'

                self.mode='unknown'

                self.pid=0

 

        def process(self):

                global start_timestamp

 

                try:

                        task = tasks[self.tid]

                        if task.pid=='unknown':

                                tasks[self.tid].pid=self.pid

                except:

                        task = Task(start_timestamp,self.command,self.mode,self.pid)

                        tasks[self.tid]= task

 

                ifself.cpunotin task.cpus:

                        task.cpus[self.cpu]= CPU()

                        if task.cpu=='unknown':

                                task.cpu=self.cpu

 

                ifself.cpu!= task.cpu:

                        task.cpu=self.cpu

                        task.migrations +=1

 

                return task

Then a class for each event type would be similarly constructed:

class Event_sys_enter ( Event ):

 

        def__init__(self, timestamp, cpu, tid, comm,id, pid):

                self.timestamp= timestamp

                self.cpu= cpu

                self.tid= tid

                self.command= comm

                self.id=id

                self.pid= pid

                self.mode='busy-unknown'

                

        def process(self):

                global start_timestamp, curr_timestamp

                curr_timestamp =self.timestamp

                if(start_timestamp ==0):

                        start_timestamp = curr_timestamp

 

                task =super(Event_sys_enter,self).process()

 

                if task.mode=='busy-unknown':

                        task.mode='user'

                        for cpu in task.cpus:

                                task.cpus[cpu].user= task.cpus[cpu].busy_unknown

                                task.cpus[cpu].busy_unknown=0

 

                task.syscall=self.id

                ifself.idnotin task.syscalls:

                        task.syscalls[self.id]= Call()

 

                task.syscalls[self.id].timestamp= curr_timestamp

                task.change_mode(curr_timestamp,'sys')

Further refactoring is evident above, as well, moving the common code that updates relevant statistics based on a task's state change and the state change itself into a change_mode method of the Task class.

Start-of-trace timestamp

As mentioned above, for scripts that depend on elapsed time, there should be an easier way to get the first timestamp in the trace other than forcing every event-handling function to conditionally save its timestamp as the start-of-trace timestamp.

Awkward invocation

The syntax for invoking a perf Python script, including script parameters, is slightly awkward:

$ perf script –s ./curt.py -- --window=80

Also, it's awkward that perf Python scripts are not themselves executable.
The curt.py script was made directly executable and will invoke perf, which will in turn invoke the script. Implementation is a bit confusing but it's easy to use:

$ ./curt.py --window=80

This script must detect when it has been directly invoked. The Python environment established by perf is a virtual module from which the perf Python scripts import:

try:

        from perf_trace_context import *

If this import fails, the script was directly invoked. In this case, the script will exec perf, specifying itself as the script to run, and passing along any command line parameters:

except:

        iflen(params.file_or_command)==0:

                params.file_or_command=["perf.data"]

        sys.argv=['perf','script','-i'] + params.file_or_command + ['-s',sys.argv[0]]

        sys.argv.append('--')

        sys.argv +=['--window',str(params.window)]

        if params.debug:

                sys.argv.append('--debug')

        sys.argv +=['--api',str(params.api)]

        if params.debug:

                printsys.argv

        os.execvp("perf",sys.argv)

        sys.exit(1)

In this way, the script can not only be run directly, it can still be run by using the perf script command.

Simultaneous event registration required

An artifact of the way perf enables events can lead to unexpected trace data. For example, specifying:

$ perf record –a –e raw_syscalls:sys_enter –e raw_syscalls:sys_exit ./command

Will result in a trace file that begins with the following series of events for a single task (the perf command itself):

sys_enter

sys_enter

sys_enter

…

This happens because perf will register the sys_enter event for every CPU on the system (because of the -a argument), then it will register the sys_exit event for every CPU. In the latter case, since the sys_enter event has already been enabled for each CPU, that event shows up in the trace; but since the sys_exit has not been enabled on each CPU until after the call returns, the sys_exit call does not show up in the trace. The reverse issue happens at the end of the trace file, with a series of sys_exit events in the trace because the sys_enter event has already been disabled.
The solution to this issue is to group the events, which is not well documented:

$ perf record –e '{raw_syscalls:sys_enter,raw_syscalls:sys_exit}' ./command

With this syntax, the sys_enter and sys_exit events are enabled simultaneously.

Awkward recording step

There are a lot of different events required for computation of the full set of statistics for tasks. This leads to a very long, complicated command for recording:

$ 
perf record -e 
'{raw_syscalls:*,sched:sched_switch,sched:sched_migrate_task,sched:sched_process_exec,sched:sched_process_fork,sched:sched_process_exit,sched:sched_stat_runtime,sched:sched_stat_wait,sched:sched_stat_sleep,sched:sched_stat_blocked,sched:sched_stat_iowait,powerpc:hcall_entry,powerpc:hcall_exit}'
 -a *command --args*

The solution to this issue is to enable the script to perform the record step itself, by itself invoking perf. A further enhancement is to proceed after the recording is complete and report the statistics from that recording:

if params.record:

        # [ed. Omitting here the list of events for brevity]

        eventlist ='{' + eventlist + '}'# group the events

        command =['perf','record','--quiet','--all-cpus',

                '--event', eventlist ] + params.file_or_command

        if params.debug:

                print command

        subprocess.call(command)

The command syntax required to record and report becomes:

$ ./curt.py --record ./command

Process IDs and perf API change

Process IDs are treated a bit cavalierly by perf scripting. Note well above that one of the common parameters for the generated event-handling functions is named common_pid. This is not the process ID, but the task ID. In fact, on many current Linux-based distributions, there is no way to determine a task's process ID from within a perf Python script. This presents a serious problem for a script that wants to compute statistics for a process.
Fortunately, in Linux kernel v4.14, an additional parameter was provided to each of the event-handling functions—perf_sample_dict—a dictionary from which the process ID could be extracted: (perf_sample_dict['sample']['pid']).
Unfortunately, current Linux distributions may not have that version of the Linux kernel. If the script is written to expect that extra parameter, the script will fail and report an error:

TypeError: irq__irq_handler_exit_new() takes exactly 11 arguments (10 given)

Ideally, a means to automatically discover if the additional parameter is passed would be available to permit a script to easily run with both the old and new APIs and to take advantage of the new API if it is available. Unfortunately, such a means is not readily apparent.
Since there is clearly value in using the new API to determine process-wide statistics, curt provides a command line option to use the new API. curt then takes advantage of Python's lazy function binding to adjust, at run-time, which API to use:

if params.api==1:

        dummy_dict ={}

        dummy_dict['sample']={}

        dummy_dict['sample']['pid']='unknown'

        raw_syscalls__sys_enter = raw_syscalls__sys_enter_old

        […]

else:

        raw_syscalls__sys_enter = raw_syscalls__sys_enter_new

        […]

This requires two functions for each event:

def raw_syscalls__sys_enter_new(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, args, perf_sample_dict):

 

        event = Event_sys_enter(nsecs(common_secs,common_nsecs), common_cpu, common_pid, common_comm,id, perf_sample_dict['sample']['pid'])

        process_event(event)

 

def raw_syscalls__sys_enter_old(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, args):

        global dummy_dict

        raw_syscalls__sys_enter_new(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain,id, args, dummy_dict)

Note that the event-handling function for the older API will make use of the function for the newer API, passing a statically defined dictionary containing just enough data such that accessing it as perf_sample_dict['sample']['pid'] will work (resulting in 'unknown').

Events reported on other CPUs

Not all events that refer to a task are reported from a CPU on which the task is running. This could result in an artificially high migration count and other incorrect statistics. For these types of events (sched_stat), the event CPU is ignored.

Explicit migrations (no sched_migrate event)

While there is conveniently an event for when the kernel decides to migrate a task from one CPU to another, there is no event for when the task requests a migration on its own. These are effected by system calls (sched_setaffinity), so the sys_exit event handler must compare the event CPU to the task's CPU, and if different, presume a migration has occurred. (This is described above, but repeated here in the "issues" section for completeness.)

Mapping system call IDs to names is architecture-specific

System calls are identified in events only as unique numeric identifiers. These identifiers are not readily interpreted by humans in the report. These numeric identifiers are not readily mapped to their mnemonics because they are architecture-specific, and new system calls can be added in newer kernels. Fortunately, perf provides a means to map system call numeric identifiers to system call names. A simple example follows:

from Util import syscall_name

def raw_syscalls__sys_enter(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain,id, args, perf_sample_dict):

                print"%s id=%d" % (syscall_name(id),id)

Unfortunately, using syscall_name introduces a dependency on the audit python bindings. This dependency is being removed in upstream versions of perf.

Mapping hypervisor call IDs to names is non-existent

Similar to system calls, hypervisor calls are also identified only with numeric identifiers. For IBM's POWER hypervisor, they are statically defined. Unfortunately, perf does not provide a means to map hypervisor call identifiers to mnemonics. curt includes a (hardcoded) function to do just that:

hcall_to_name ={

        '0x4':'H_REMOVE',

        '0x8':'H_ENTER',     

        '0xc':'H_READ',     

        '0x10':'H_CLEAR_MOD',

[…]

}

 

def hcall_name(opcode):

        try:

                return hcall_to_name[hex(opcode)]

        except:

                returnstr(opcode)

Command strings as bytearrays

perf stores command names and string arguments in Python bytearrays. Unfortunately, printing bytearrays in Python prints every character in the bytearray—even if the string is null-terminated. For example:

$ perf record –a –e 'sched:sched_switch' sleep 3

$ perf script –g Python

generated Python script: perf-script.py

$ perf script -s ./perf-script.py

in trace_begin

sched__sched_switch      3 664597.912692243    21223 perf                
 prev_comm=perf^@-terminal-^@, prev_pid=21223, prev_prio=120, 
prev_state=, next_comm=migration/3^@^@^@^@^@, next_pid=23, next_prio=0

[…]

One solution is to truncate the length of these bytearrays based on null termination, as needed before printing:

def null(ba):

        null = ba.find('\x00')

        if null >=0:

                ba = ba[0:null]

        return ba



def sched__sched_switch(event_name, context, common_cpu,

        common_secs, common_nsecs, common_pid, common_comm,

        common_callchain, prev_comm, prev_pid, prev_prio, prev_state,

        next_comm, next_pid, next_prio, perf_sample_dict):



                print"prev_comm=%s, prev_pid=%d, prev_prio=%d, " \

                "prev_state=%s, next_comm=%s, next_pid=%d, " \

                "next_prio=%d" % \

                (null(prev_comm), prev_pid, prev_prio,

                flag_str("sched__sched_switch","prev_state", prev_state),

                null(next_comm), next_pid, next_prio)

Which nicely cleans up the output:

sched__sched_switch
     3 664597.912692243    21223 perf                  prev_comm=perf, 
prev_pid=21223, prev_prio=120, prev_state=, next_comm=migration/3, 
next_pid=23, next_prio=0

Dynamic mappings, like IRQ number to name

Dissimilar to system calls and hypervisor calls, interrupt numbers (IRQs) are dynamically assigned by the kernel on demand, so there can't be a static table mapping an IRQ number to a name. Fortunately, perf passes the name to the event's irq_handler_entry routine. This allows a script to create a dictionary that maps the IRQ number to a name:

irq_to_name ={}

def irq__irq_handler_entry_new(event_name, context, common_cpu, common_secs, common_nsecs, common_pid, common_comm, common_callchain, irq, name, perf_sample_dict):

        irq_to_name[irq]= name

        event = Event_irq_handler_entry(nsecs(common_secs,common_nsecs), common_cpu, common_pid, common_comm, irq, name, getpid(perf_sample_dict))

        process_event(event)

Somewhat oddly, perf does not pass the name to the irq_handler_exit routine. So, it is possible that a trace may only see an irq_handler_exit for an IRQ and must be able to tolerate that. Here, instead of mapping the IRQ to a name, the IRQ number is returned as a string instead:

def irq_name(irq):

        if irq in irq_to_name:

                return irq_to_name[irq]

        returnstr(irq)

Task 0

Task 0 shows up everywhere. It's not a real task. It's a substitute for the "idle" state. It's the task ID given to the sched_switch event handler when the CPU is going to (or coming from) the "idle" state. It's often the task that is "interrupted" by interrupts. Tracking the statistics for task 0 as if it were a real task would not make sense. Currently, curt ignores task 0. However, this loses some information, like some time spent in interrupt processing. curt should, but currently doesn't, track interesting (non-idle) time for task 0.

Spurious sched_migrate_task events (same CPU)

Rarely, a sched_migrate_task event occurs in which the source and target CPUs are the same. In other words, the task is not migrated. To avoid artificially inflated migration counts, this case must be explicitly ignored:

class Event_sched_migrate_task (Event):

        def process(self):

[…]

                ifself.cpu==self.dest_cpu:

                        return

exec

The semantics of the exec system call are that the image of the current process is replaced by a completely new process image without changing the process ID. This is awkward for tracking the statistics of a process (really, a task) based on the process (task) ID. The change is significant enough that the statistics for each task should be accumulated separately, so the current task's statistics need to be closed out and a new set of statistics should be initialized. The challenge is that both the old and new tasks have the same process (task) ID. curt addresses this by tagging the task's task ID with a numeric suffix:

class Event_sched_process_exec (Event):

  def process(self):

    global start_timestamp, curr_timestamp

    curr_timestamp =self.timestamp

    if(start_timestamp ==0):

      start_timestamp = curr_timestamp

 

    task =super(Event_sched_process_exec,self).process()

 

    new_task = Task(self.timestamp,self.command, task.mode,self.pid)

    new_task.sched_stat=True

    new_task.syscall= task.syscall

    new_task.syscalls[task.syscall]= Call()

    new_task.syscalls[task.syscall].timestamp=self.timestamp

 

    task.change_mode(curr_timestamp,'exit')

 

    suffix=0

    whileTrue:

      old_tid =str(self.tid)+"-"+str(suffix)

      if old_tid in tasks:

        suffix +=1

      else:

        break

  

    tasks[old_tid]= tasks[self.tid]

 

    del tasks[self.tid]

 

    tasks[self.tid]= new_task

This will clearly separate the statistics for the different process images. In the example below, the perf command (task "9614-0") exec'd exec.sh (task "9614-1"), which in turn exec'd itself (task "9614"):

-- [  task] command   cpu      user       sys       irq        hv      busy      idle |  util% moves

    [  9614] execs.sh    4  1.328238  0.485604  0.000000  0.000000  0.000000  2.273230 |  44.4%

    [  9614] execs.sh    7  0.000000  0.201266  0.000000  0.000000  0.000000  0.003466 |  98.3%

    [  9614] execs.sh  ALL  1.328238  0.686870  0.000000  0.000000  0.000000  2.276696 |  47.0%     1



-- [  task] command   cpu      user       sys       irq        hv      busy      idle |  util% moves

    [9614-0] perf        3  0.000000  0.408588  0.000000  0.000000  0.000000  2.298722 |  15.1%

    [9614-0] perf        4  0.059079  0.028269  0.000000  0.000000  0.000000  0.611355 |  12.5%

    [9614-0] perf        5  0.000000  0.067626  0.000000  0.000000  0.000000  0.004702 |  93.5%

    [9614-0] perf      ALL  0.059079  0.504483  0.000000  0.000000  0.000000  2.914779 |  16.2%     2

 

-- [  task] command   cpu      user       sys       irq        hv      busy      idle |  util% moves

    [9614-1] execs.sh    3  1.207972  0.987433  0.000000  0.000000  0.000000  2.435908 |  47.4%

    [9614-1] execs.sh    4  0.000000  0.341152  0.000000  0.000000  0.000000  0.004147 |  98.8%

    [9614-1] execs.sh  ALL  1.207972  1.328585  0.000000  0.000000  0.000000  2.440055 |  51.0%     1

Distribution support

Limit on number of traced events

As curt gets more sophisticated, it is likely that more and more events may be required to be included in the trace file. perf currently requires one file descriptor per event per CPU. This becomes a problem when the maximum number of open file descriptors is not a large multiple of the number of CPUs on the system. On systems with large numbers of CPUs, this quickly becomes a problem. For example, the default maximum number of open file descriptors is often 1,024. An IBM POWER8 system with four sockets may have 12 cores per socket and eight threads (CPUs) per core. Such a system has 4 * 12 * 8 = 392 CPUs. In that case, perf could trace only about two events! A workaround is to (significantly) increase the maximum number of open file descriptors (ulimit –n if the system administrator has configured the hard limits high enough; or the administrator can set the limits higher in /etc/security/limits.conf for nofile).

Summary

I hope this article shows the power of perf—and specifically the utility and flexibility of the Python scripting enabled with perf—to perform sophisticated processing of kernel trace data. Also, it shows some of the issues and edge cases that can be encountered when the boundaries of such technologies are tested.
Please feel free to download and make use of the curt tool described here, report problems, suggest improvements, or contribute code of your own on the curt GitHub page.