Peering Inside

Intro

Imagine that you’re responsible for “program X” working correctly, and one day it refuses to do its job. To troubleshoot that program – and the system on which it runs – you need to be able to look inside that system and see what’s happening.

What might we want to see?

– Is the program running at all?
– If not, why?
– Is it recording any errors?
– How about the operating system on which it’s running?
– Can I see low-level requests and replies?
– Can I see what it’s doing right now?
– Is there something non-standard about this hardware or operating system that’s causing trouble?

In a perfect world, your program would clearly log what’s wrong and offer ideas on how to fix it. Ah, we can dream. 🙂

What follows is a list of my favorite tools I use to get a picture of the underlying hardware, the operating system, the system configuration, and running applications.  To the best of my knowledge, these commands and the recommended parameters just return information; none of them should modify the system in the form I give below.  You should be able to use all of them safely even on a production system, but feel free to try them on a cloud instance or virtual machine first.

For more details about each one, try running

man programname

in a separate window. That will describe what the tool does and give you details about all the available command line options.

The following commands are very likely to be available on Linux systems. A few may need to be installed with the package manager; if you’re missing tool N, try running

sudo yum -y install N || sudo apt -y install N

For example, “lshw” is not installed by default, so try:

sudo yum -y install lshw || sudo apt -y install lshw

Many will also be available on MacOS, FreeBSD, OpenBSD, and NetBSD. Windows systems will have far fewer of them, but they may be available inside WSL (the Windows Subsystem for Linux) 2.0 or higher.

When a tool needs to be run as root to either 1) work at all or 2) give a full set of information, we include “sudo” in front of the command. As an example, the “dmesg” command will not work unless run as root, so we show “sudo dmesg” as the command to run.

If the output of any given tool scrolls up your screen too quickly, don’t forget you can feed the output into less, like:

sudo ps axf | less -S

which allows you to use the arrow keys to scroll up, down, left, and right. (Press “q” to exit less.)

 

Hardware

Devices

“lshw” stands for “list hardware”. It’s a fantastic tool for getting the details on the underlying physical or virtual hardware in use.

sudo lshw -short

Lists all devices. To see just one class of device:

sudo lshw -short -C memory
sudo lshw -short -C processor
sudo lshw -short -C storage
sudo lshw -short -C disk,volume
sudo lshw -short -C network
sudo lshw -short -C display

 

sudo lshw -C system

This gives details about the system. In particular, it can report on whether you are running on a cloud server of some kind.

lspci

List PCI devices

lsblk -fs

List block devices (hard drives, flash drives, SSDs)

 

System Details

The “Desktop Management Interface” describes a standard for retrieving custom fields on a piece of hardware, like the serial number assigned by the manufacturer. Linux can retrieve these fields – and much more! – with the dmidecode command:

sudo dmidecode | less

Request all DMI information

sudo dmidecode -s system-product-name
sudo dmidecode -s system-manufacturer
sudo dmidecode -s system-serial-number
sudo dmidecode -s chassis-asset-tag

Request specific fields from DMI.

cat /var/lib/dbus/machine-id

Request the unique machine identifier created at install time (not available on all distributions.)

 

Application Tools

Memory and CPU

top

Shows the processes using the most processing time (press “P” to sort by CPU time) or memory (press “M” to sort by memory usage). Good starting point when a system seems overloaded or low on memory)

lscpu

List CPUs, including number of cores, processor speed, and CPU features.

less /proc/cpuinfo

A raw version of lscpu in case lscpu is not available.

free

A quick summary of the amount of memory and swap, along with some details on how they’re being used. Most of this information is shown live in the “Mem” and “Swap” lines of “top”.

 

Mem/CPU/Disk/Network

atop

(If missing, install the “atop” package)

Atop looks like “top”, but it adds a lot of visibility that top doesn’t have. You’ll be able to see statistics on individual network devices, processors, and drives.

htop

Also similar to top. htop extends the display with graphs and some more interactive controls over running programs.

gkrellm

(If missing, install the “gkrellm” package)

gkrellm is a slim display that shows graphs of a system’s state. See http://www.stearns.org/doc/network-monitoring.current.html and https://www.activecountermeasures.com/why-is-my-program-running-slowly/ . For MacOS see Activity Monitor, and for Windows, see Resource Monitor, both below.

cat /proc/meminfo

Very detailed stats on where memory is being used.

cat /proc/swaps

Details on how much swap space is available.

Activity Monitor
(Mac OS) Applications/Utilities/Activity Monitor.app

Performance Monitor
(Windows) Start/Windows Administrative Tools

More focused on specific performance values than Task Manager.

Resource Monitor
(Windows) Start/Windows Administrative Tools

Graphs of overall system usage, with the ability to drill down into specific applications.

 

Disk

df -h

Free space on mounted drives, human readable.

df -i

Inode counts. If you ever get errors creating new files that initially look like the drive is full – but it’s not – try “df -i” to see if one of your filesystems has run out of inodes (which can happen if you have huge numbers of very small files.)

lsblk -fs

Show the block devices (hard drives, flash drives, and SSDs) and where they’re mounted.

findmnt

List all mounted filesystems in a tree (hierarchical) format. More details: https://linuxhandbook.com/findmnt-command-guide/ . This is a more readable display than the one you get from running mount .

du -sh *

Look at each of the directories underneath the current one and report on how much space is taken by all.

 

Running Processes

pstree

A quick summary of the major applications running on a system.

ps axf | less

Running processes in a “forest” view (a program that starts another program will be adjacent to it with the “child” process indented).
Processes surrounded by square brackets are kernel threads.
Process ID 1 is “Init”, parent of everything else.

See the “/proc/ filesystem” section below for more details about how the kernel publishes this information.

Task Manager
(Windows) Ctrl-Alt-Delete/Task Manager
CPU/Memory/Disk/network, performance graphs.

 

Running Daemons and Services

systemctl list-units

systemctl list-units -all

What are all the potential services/features that could be enabled on this system?

systemctl list-unit-files --type=service --state=enabled

What services will get loaded at boot time?

From the above lists of available services, find the one you’re interested in. I use “rsyslog.service” in the following examples; substitute your service of interest.

systemctl status rsyslog.service

Get a summary of a particular service, including recent logs.

systemctl start rsyslog.service

Start it now

systemctl stop rsyslog.service

Stop it now

systemctl enable rsyslog.service

Start on next boot (or device being plugged in)

systemctl disable rsyslog.service

Do not start on next boot (or device being plugged in)

 

Additional Places Programs Can Be Started From

Take a look at these files and directories to see common places from which tools can be started.

Files

/etc/bashrc
/etc/profile
/etc/rc.d/rc.local
/etc/rc.local
~/.bash_profile
~/.profile
~/.bashrc

Directories

/etc/cron.d/
/etc/cron.daily/
/etc/cron.hourly/
/etc/cron.monthly/
/etc/cron.weekly/
/etc/profile.d/
/etc/rc.d/rc0.d/
/etc/rc.d/rc1.d/
/etc/rc.d/rc2.d/
/etc/rc.d/rc3.d/
/etc/rc.d/rc4.d/
/etc/rc.d/rc5.d/
/etc/rc.d/rc6.d/ 
/var/spool/at/spool/
/var/spool/cron/crontabs/
/var/spool/cron/

 

Who Is Logged In?

w

This provides a list of all users logged in over ssh and what they’re running.

last

Show previous successful logins.

sudo lastb

Show previous failed login attempts. This depends on having a file called /var/log/btmp to record bad logins.

 

Error Messages

Every program has regular output (called “stdout” == “standard output”), but it also has a separate output for errors called “stderr”. Here’s an example of how to append the errors to a text file on disk:

program 2>>error_output.txt

sudo dmesg

These are the errors and messages from the kernel, starting from the most recent reboot. These include devices found, filesystems found, network interfaces found, and any kernel errors.

sudo dmesg -T

Like above, but show dates in human readable form.

sudo journalctl

Similar information to dmesg, but pulled from systemd.

sudo tail -F /var/log/syslog

Main system log file on Debian, Ubuntu, and derivatives.

sudo tail -F /var/log/auth.log

Security/authentication log file on Debian, Ubuntu, and derivatives. Handy if someone can’t log in. While running the above, have that person try again and see what error(s) show up.

sudo tail -F /var/log/messages

Main system log file on Fedora, RHEL, Centos, Rocky, and derivatives.

sudo tail -F /var/log/secure

Security/authentication log file on Fedora, RHEL, Centos, Rocky, and derivatives.

Note that ‘tail –f somelogfile’ shows you the most recent lines, but does not switch to the new file when that file is rotated (such as when a log file is replaced by a new one weekly or daily). To automatically switch to a new file when it’s rotated, use ‘tail –F somelogfile instead’.

sudo grep -i something /var/log/{messages,syslog}
sudo grep -i systemd /var/log/{messages,syslog}

Look for something in the 2 main log files (“-i” is case-insensitive.) The example in the second line looks specifically for any log lines that mention “systemd”.

 

System Calls

Applications do two things: 1) internal calculations and work and 2) making requests of the kernel for memory, network sockets and traffic, user IO, file IO, and more. Everything in that second category is made by a system call. The strace program can display what these system calls are and what result(s) came back from the kernel in response. While strace can’t show any of the internal work a program does (use “gdb” for that), it shows all external requests, which is commonly what we care about.

strace -p apppid
strace -p apppid -o my_trace.txt

The above gives you the system calls made by an already-running program with process ID apppid (use “ps axf” to find the process ID (in the far left column)). “-o my_trace.txt” stores the output in that file.

strace -p apppid -e trace=file

Instead of showing all system calls, just show the ones related to file operations. There are other filters as well – see ‘man strace’ .

strace app params

This starts a brand new app and immediately uses strace to show the system calls it makes

strace sleep 10

Here’s an example; run the “sleep” program with a command line parameter of 10 seconds, and show all the system calls made by that application as it runs.

Strace is for watching a single application. If you need system-wide visibility into classes of system calls, you may want to look into systemtap. You generate a text file saying what you’re interested in and how you’d like the results and this gets injected into the kernel. When one of these events happens, you get an output record in your format.

 

Shell Scripts Step-By-Step

set -x

Place at the top of a shell script to display each command before running it. Helpful if you’re having trouble finding how far into the script you’ve gone before a problem occurs.

 

Binaries Step-By-Step

gdb

The “gnu debugger”. This allows you to finely control how a program is run, along with stopping it so you can inspect its internal state. The process can be quite intricate, so unless you are writing a program and know you need this level of detail, you might consider using strace instead.

 

Network State

Network Interfaces, Hosts, and Routes

ip address

Show the IP addresses associated with your network interfaces.

ip link

See the mac addresses for each interface.

arp -an

(If missing, install the “net-tools” package)
This lists the IP addresses that are visible on your ethernet interfaces, along with the interface through which they’ve been seen and their Ethernet mac address. If this is listed as “incomplete”, that means that your system has not been able to locate this IP address.

ip neighbor

Like “arp -an”, show the hosts that are directly reachable (i.e., without having to go through a router) on your network segments.

ip route

Like “route -n”, display the routing table for both locally connected networks and networks that are reachable through a router.

ifconfig

Display the details of all live network interfaces.
(part of the “net-tools” package)

ifconfig -a

All interfaces, both live and down.
(part of the “net-tools” package)

vnstat

(If missing, install the “vnstat” package)
vnstat gives a detailed view of network statistics, but depends on internal kernel counters instead of sniffing packets so it uses fewer resources.

 

Open Files and Network Connections

sudo lsof -n | less

All open “files”, which include actual files, virtual files, network sockets, and more.

sudo lsof -nP +c 0 -iTCP -sTCP:LISTEN

Show listening TCP ports.

sudo lsof -nP +c 0 -iUDP

Show listening UDP ports.

sudo lsof -n | grep deleted

Find any deleted files that are still tying up disk space because some program is still holding them open. Try this on a system that doesn’t appear to have any huge files but has still run out of space.

sudo netstat -an | egrep '(^..p|Local Address)'

All in-use TCP and UDP ports.

sudo netstat -anp | egrep '(^..p|Local Address)'

Show the name of the program attached to the port too (Linux only, not MacOS)

The “State” column shows how this port is used:

LISTEN – Waiting for an incoming TCP connection
ESTABLISHED – A live connection between 2 systems
TIME_WAIT, FIN_WAIT1, etc – Closed TCP connection.
(blank) – A UDP endpoint that could be a client or server

sudo ss -p | less -S

Listening ports and sockets.

sudo ss -tuanp | less -S

Just tcp and udp ports.

 

Linux Firewall

iptables -L -nxv
ip6tables -L -nxv

List all the firewall rules in all chains, along with counts of how many packets (column 1) and how many bytes (column 2) have matched each rule. IPv4 and IPv6 firewalls are kept separately, so you need to run both commands to see all rules.

 

My Public IP Address

Even though we use reserved addresses (starting with 10. , 192.168. , 172.16. through 172.31. , or fe80:) internally, these get converted to a non-reserved address when they pass through the final firewall or router on the way to the Internet. A few organizations have set up free servers out on the Internet that will tell you what your non-reserved address is (they get huge amounts of connections per hour, so please limit your requests.)

http_proxy='' curl -s http://checkip.dyndns.org </dev/null | sed -e 's/.*Current IP Address: //' -e 's/<.*$//'

http_proxy='' wget -q -O - http://checkip.dyndns.org </dev/null | sed -e 's/.*Current IP Address: //' -e 's/<.*$//'

http_proxy='' wget -q http://icanhazip.com -O -

Get any public IP address used when this system communicates with the Internet.

http_proxy='' wget -4 -q http://icanhazip.com -O -

Get the public IPv4 address used when this system communicates with the Internet.

http_proxy='' wget -6 -q http://icanhazip.com -O -

Get the public IPv6 address used when this system communicates with the Internet.

 

Default Route

ip route | grep default
route -n | grep '^0\.0\.0\.0'

These both show the default gateway IP address – the router that carries your non-local traffic to the whole Internet – along with the ethernet interface that reaches it.

 

Do I Have an Internet Connection?

ping -n -c 4 default_gateway_ip_address

Can you ping the default gateway? If not, fix that first.

arping -I eth0 default_gateway_ip_address

arping sends arp requests instead of icmp echo requests (which might be blocked by a firewall.) If ping fails but arping succeeds, you likely have a firewall blocking some of your traffic. Note that arping only works for local systems, not ones beyond a router or your default gateway.

ping -n -c 4 8.8.8.8
ping -n -c 4 one.of.my.servers

If the default gateway is reachable, can you also ping a host on the Internet? 8.8.8.8 is one of Google DNS servers, which is all but guaranteed to respond if you can reach it.

ping6 -n -c 4 2001:4860:4860::8888

This tests to see if the system has a live IPv6 connection by pinging one of Google’s DNS servers over IPv6.

traceroute -n one.of.my.servers

(If missing, install the “traceroute” package. Note; if your version of traceroute doesn’t accept “-n”, remove it.)
If I’m not able to reach a host with ping, how close can I get? Traceroute reports back on the individual routers between this machine and one.of.my.servers , one step at a time.

sudo tcptraceroute -n -p 443 one.of.my.servers

If you’re trying to reach a specific TCP port on a system but can’t seem to talk to it, try tcptraceroute. Like traceroute, it reports on the routers between this system and one.of.my.servers, but it sends TCP packets (in this case, with a destination port of 443) to identify a potential firewall in the middle that may be dropping that specific TCP port.

 

Packet Capture

This section could totally be its own multi-part blog series. I’m bringing this up because every once in a while you want a tool to quickly display packets coming into or going out of the system to make sure that something else is working. tcpdump is almost always available (though on Windows you’ll need to use windump with almost exactly the same command line options.)

sudo tcpdump -i eth0 -qtnp

Capture all packets coming into or going out of interface eth0 and display them in a (usually) one-line-per-packet format.

sudo tcpdump -i eth0 -qtnp 'tcp port 80 or udp port 53'

Apply a filter (called a BPF) to ignore all traffic except packets with a source or destination port of 80/tcp (usually http) or 53/udp (usually dns). The filter must be the last thing on the command line and surrounded with single quotes (double quotes if you’re running windump on windows.) BPFs can be used to filter traffic to ports like above, addresses, and almost any header field. We have blogs and a webcast on BPF.

 

Special Cases

Kernel

uname -a

Get details about the running kernel, including version, date compiled, and architecture. See man uname if you want just one of those fields.

uptime
cat /proc/uptime

Both report on how long the system has been running – the first is more human-readable.

getenforce

(If missing, there’s no need to install it – selinux is not active on your system.)
Report on whether the selinux kernel module is not installed at all (“command not found”), Disabled, Enabled (blocking suspicious actions), or Permissive (just logging suspicious actions, not blocking them.) Actions are only blocked in “Enabled” mode, so if you’re in one of the other three states, selinux is not causing the problem.

sestatus

Give more detail on how selinux is configured.

 

/proc/ filesystem

Linux and Unix systems have a relatively standard way of reporting on their status, recognized devices, running processes, and configuration called the “proc filesystem”. If you look inside the directory “/proc/” with a command like:

ls -al /proc/

you’ll see:
– Directories whose names are digits. The directory names correspond to the process ID of all running programs. Inside these directories are virtual files that have more detail on each, such as the “cmdline” virtual file that contains the program name and parameters (all run together):

cat /proc/$$/cmdline

will show the command line for your current shell.

– Some other directory trees for major kernel components, like “acpi”, “bus”, “driver”, “fs” (filesystem), “irq”, “scsi”, and “sys”. Under these trees are more details on the current state of each.

– Other virtual files line “uptime”, “swaps”, “modules”, and more. System tools that need to report on the system like the ‘uptime’ command will commonly go down to one or more files located inside proc to get their raw data (in uptime’s case, it reads /proc/uptime .) SImilarly, lsmod (that lists loaded kernel modules) gets its raw data from /proc/modules; compare the output of the following two commands:

cat /proc/modules | sort | head
lsmod | sort | head

While the command line tools have a much more readable output format, there may be times when you need the raw data (such as when a system has been hacked and you don’t trust the tools on it.) It’s generally safe to read the files located under /proc/ to get it; reading those files should not alter the system. Writing to any of those objects could alter how the kernel runs, so please consult the /proc/ filesystem documentation first.

 

Docker

sudo docker ps

Running containers. In docker, a “container” is a currently running instance.

sudo docker ps -a

Also show dead ones

sudo docker inspect container_name

“docker inspect” returns a json-formatted block of detailed information about the running instance. To find the container name, run “sudo docker ps” and look in the NAME column.

sudo docker logs achunter_db

Retrieve the log entries from the container named “achunter_db”.

sudo docker inspect -f '{{ .NetworkSettings.Networks.achunter_default.IPAddress }}' achunter_db

If you’re only interested in one specific field from that json block, the -f option allows you to go down in the hierarchy to the specific field. Alternatively, you can feed the output into “jq” (json query) and parse it there.

sudo docker images

List the available images (the static objects that can be started into containers.)

 

Flatpak

Flatpak is a second approach to packaging applications with their dependencies. Flatpak is available across the majority of Linux distributions.

flatpak list

List installed flatpack applications and runtimes. If you get “command not found”, that means flatpak is not in use on your system.

 

Snaps

Snaps are an alternate packaging tool used only on Ubuntu distributions. Like Docker and Flatpack, they give a way to distribute a program and its dependencies.

snap list

List all running snaps. If you get “command not found”, that means snaps are not in use on your system.

 

OS

What Distribution?

cat /etc/*release*

One of the first things I’ll do on an unfamiliar Linux system is get my bearings by seeing what Linux distribution I’m on. This gives me the Linux flavor and version information.

 

List Packages Installed By rpm/apt

rpm -qa | less

List package names and versions on rpm-based distributions.

rpm -qai | less

Names and package details for all packages on an rpm-based distribution.

apt list --installed | less

List package names and versions on apt-based systems.

 

What Files Have Changed Since They Were Installed?

rpm -Va

(rpm-based Linux systems only) This walks through the list of all files that were installed by rpm, yum, or dnf, and reports on just the files that have changed. For example, this line of output:

S.5....T. c /etc/sudoers

means that the /etc/sudoers file has been modified. Specifically, its size, md5sum, and time are different than when it was originally installed. “c” means that the original packager flagged it as a configuration file. See the “VERIFY OPTIONS” section of “man rpm” for details on these flags.

This does not verify all files – just the ones installed by rpm-based tools. To do a full system comparison look into a File Integrity Monitoring package for your distribution.

 

Add a New Package

sudo yum -y install packagename
sudo dnf -y install packagename

Install a package on an rpm-based distribution (Fedora, RHEL, Centos, Rocky, Alma, etc.) Yum will work on more systems, though dnf is newer. Use whichever works as they have the same effect.

sudo apt -y install packagename

Install a package on a deb-based distribution (Debian, Ubuntu, others)

 

What Package Installed This File?

rpm -qf filename
rpm -qf /bin/bash

On an rpm-based distribution this will tell you what package provided this file.

dpkg -S filename
dpkg -S /bin/bash

Equivalent for debian-based distributions.

 

Python Libraries

pip list

(If missing, install the “python3-pip” package)
Gives a list of library names and versions currently installed by pip

pip show python-dateutil

Give more specific detail about a particular library (in this case, python-dateutil.)

 

Perl Modules

cpan -l

List all modules installed by CPAN

cpan -D 'modulename'
cpan -D 'URI::https'

Show details about a particular module

 

References

https://www.youtube.com/watch?v=35HCzGxGK94
Linux and Security Basics (Hands on)

http://www.stearns.org/gratr/gratr
Tool that will pull out system details from a Mac or Linux system.

http://www.stearns.org/doc/pcap-apps.html
Additional ideas for packet capture tools, though outdated.

 

Where to Go Next?

• While you’ve seen short examples of valuable tools, remember to check the man page for them to see other ways they can be used.
• To get a more detailed explanation of an individual command with options – or even a series of commands in a pipeline – see https://www.explainshell.com/ .
• If you’d like to see more about how the command line – bash – works and how to use it to manage systems, please take a look at the “Bash scripting for Server Administration” class at https://www.antisyphontraining.com/on-demand-courses/bash-scripting-for-server-administration-w-bill-stearns/ . This course goes beyond the individual tools to cover scripting that you can use to manage your own systems. It also goes into more detail about a number of the tools above.

 

Credits

Many thanks to Ethan, Logan, and Keith for their contributions to this article! Many thanks, too, to Keith and Shelby for getting this published!