Phase 4 · Free

4.2 Logs: journalctl, /var/log, dmesg

When something fails quietly, the explanation is in a log. Linux writes down nearly everything: services starting and crashing, logins, disk errors, hardware hiccups. You just need to know where to look and how to filter the noise.

25 minPhase 4: Break It, Fix ItFree

Phase 4. Lesson 2 of 5. About 25 minutes.

Plain entry

When something fails quietly, the explanation is in a log. Linux writes down nearly everything: services starting and crashing, logins, disk errors, hardware hiccups. You just need to know where to look and how to filter the noise.

Terms

  • Log: a time-stamped record of events.
  • journaldThe systemd logging service that collects messages from the kernel, services, and programs into one indexed journal, read with the journalctl command. Read more → / the journal: systemdThe init system and service manager for most modern Linux distributions, responsible for booting the system and managing running services. Read more →’s log store, read with journalctl. See systemd.
  • Unit: anything systemd manages, such as a service (ssh.service).
  • KernelThe core of an operating system that runs with full hardware control and decides how every program gets CPU time, memory, and device access. Read more → ring buffer: the kernel’s own messages, read with dmesg.
  • Priority: severity level, from emerg (0) to debug (7). err is 3.

Depth

journalctl -b                 # everything since this boot
journalctl -b -p err          # only errors and worse
journalctl -u ssh             # one service (install openssh-server first if you want data)
journalctl -f                 # follow live
journalctl --since "10 min ago"
journalctl --list-boots       # every boot the journal remembers
journalctl -b -1 -p err       # errors from the previous boot

If journalctl prints Hint: You are currently not seeing messages from other users and the system., your account is not in the adm group. The first user created at install normally is; anyone else adds sudo in front. Example journalctl -b -p err line:

Nov 01 09:41:07 bnlc-ubuntu systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.

When you start the service by hand, systemctl tells you where to look:

Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

Then dig into that unit:

systemctl status nginx
journalctl -u nginx -n 50 --no-pager

Kernel messages:

sudo dmesg -T | tail -20
sudo dmesg -T --level=err,warn

Classic text logs in /var/log:

  • /var/log/auth.log: logins and sudoA Linux command that runs a single command with administrator (root) privileges, after checking that your account is allowed to do so. Read more → use
  • /var/log/syslog: general system messages (rsyslog is installed by default on Ubuntu 26.04 desktop and server)
  • /var/log/apt/history.log: every package install and removal
  • /var/log/dpkg.log: low-level package actions
sudo grep sudo /var/log/auth.log | tail -5

Going further

The journal survives reboots only if /var/log/journal exists (Ubuntu creates it). Check size with journalctl --disk-usage (it prints a line like Archived and active journals take up 8M in the file system.) and trim it with sudo journalctl --vacuum-size=200M. journalctl -o json-pretty -n 1 shows the fields behind each line, which is how a SIEMA platform that collects, correlates, and analyzes log data from across your infrastructure to detect security threats and support incident investigation. Read more → ingests them later in the Security track.

Why it works this way

Programs fail in ways they cannot always show on screen, so they write down what happened. The journal collects all of it with timestamps, which is why a log usually answers “what happened first?” The first error is usually the cause. Later errors are often side effects. Read from the first failure forward, not from the last line back.

Self-check

  1. Show only errors from the current boot.
  2. Show errors from the boot before this one.
  3. Where would you look to see who used sudo?

Answers: (1) journalctl -b -p err. (2) journalctl -b -1 -p err. (3) /var/log/auth.log, lines tagged sudo:.

Sources