Phase 4 · Free

4.4 Filesystems, fstab, fsck, and Full Disks

Two of the most common "server is down" calls are a disk that filled up and a bad line in the file that tells Linux which disks to mount at boot. Both look scary. Both are quick to fix once you know where to look.

35 minPhase 4: Break It, Fix ItFree

Phase 4. Lesson 4 of 5. About 35 minutes.

Plain entry

Two of the most common “server is down” calls are a disk that filled up and a bad line in the file that tells Linux which disks to mount at boot. Both look scary. Both are quick to fix once you know where to look.

Terms

  • /etc/fstabThe /etc/fstab file that tells Linux which filesystems to mount at boot, where to mount them, and with what options. Read more →: the file listing which filesystems to mount at boot, and where.
  • UUIDUniversally Unique Identifier: a 128-bit value used to label things, like disks and partitions, so each one can be identified reliably no matter the order it is detected. Read more →: a unique ID for a partitionA defined section of a physical disk that the operating system treats as a separate unit, each usually holding its own filesystem. Read more →. fstab uses it so disk renaming does not break boot.
  • fsckFile System Consistency Check: the Linux tool that checks a filesystem for errors and repairs them, usually run on an unmounted disk. Read more →: the filesystem check-and-repair tool. Only run on unmounted (or read-only) filesystems.
  • InodeA record inside a Linux filesystem that stores everything about a file except its name: owner, permissions, size, timestamps, and where its data lives on disk. Read more →: the record a filesystem keeps for each file. You can run out of inodes before you run out of space.

Depth

Read your fstab

cat /etc/fstab
lsblk -f

Example fstab line:

UUID=3f6c...a91e  /  ext4  defaults  0  1

Fields: what, where, type, options, dump (ignore), fsck order (1 for root, 2 for others, 0 for none).

Break and fix: bad fstab line

Take a snapshotA saved, point-in-time state of a virtual machine or storage volume that you can roll back to if a change goes wrong. Read more → first. Then:

sudo cp /etc/fstab /etc/fstab.bak
echo "UUID=deadbeef-0000 /data ext4 defaults 0 2" | sudo tee -a /etc/fstab
sudo findmnt --verify

findmnt --verify catches it before you reboot (on a clean fstab it prints Success, no errors or warnings detected):

0 parse errors, 2 errors, 1 warning
/data
   [E] unreachable on boot required target: No such file or directory
   [E] unreachable on boot required source: UUID=deadbeef-0000
   [W] your fstab has been modified, but systemd still uses the old version;
       use 'systemctl daemon-reload' to reload

The exit code is 1 when errors are found, so scripts can test it. The target error appears because /data does not exist either.

Reboot anyway to see what failure looks like. The boot waits 1 min 30 s for the missing disk, prints Timed out waiting for device, then Dependency failed for data.mount - /data. and Dependency failed for local-fs.target - Local File Systems., and you land at:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit"
to continue bootup.
Press Enter for system maintenance
(or press Control-D to continue):

Press Enter. On Ubuntu 26.04 this gives a root shell even though the root account has no password (tested on the 26.04 server image; the desktop is expected to match). If your machine asks for a root password instead and refuses you, reboot into GRUB recovery mode (lesson 4.3) and pick root. Then:

mount -o remount,rw /               # only needed if / is read-only; harmless otherwise
cp /etc/fstab.bak /etc/fstab        # or edit out the bad line with nano
systemctl daemon-reload
findmnt --verify                    # expect: Success, no errors or warnings detected
reboot

Prevention: add nofail to the options of any non-essential disk so a missing disk does not stop boot.

fsck

Never on a mounted, writable filesystem. From recovery mode, choose fsck in the menu, or on a second disk:

sudo umount /dev/sdb1
sudo fsck -f /dev/sdb1

Expected output on a healthy ext4 disk (counts are environment-specific):

fsck from util-linux 2.41.3
e2fsck 1.47.2 (1-Jan-2025)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: 12/131072 files (0.0% non-contiguous), 18091/523776 blocks

Forget the umount and fsck refuses: /dev/sdb1 is mounted. then e2fsck: Cannot continue, aborting. To force a check of root on next boot, add fsck.mode=force to the kernel line in GRUB (press e, lesson 4.3).

Break and fix: full disk

df -h /
fallocate -l 5G ~/bigfile    # fills 5 GB instantly; adjust to your free space
df -h /

When Use% hits 100%, programs fail with No space left on device. Find what is big:

sudo du -xh / --max-depth=1 2>/dev/null | sort -h | tail
du -sh ~/* | sort -h | tail

Common real-world culprits and safe fixes:

sudo journalctl --vacuum-size=200M    # shrink the journal
sudo apt clean                        # clear downloaded packages
sudo apt autoremove --purge           # old kernels and unused packages

Then rm ~/bigfile.

Out of inodes looks the same but df -h shows free space. Check with df -i.

Myth vs fact

Myth: “Deleting a big file always frees its space.” Fact: Linux frees the space only when no program still has the file open. A log file that a running service is writing to keeps its space after rm, and df still shows the disk full. Tested on the course VM with a 12 MB file held open by a small Python program:

rm log.bin
df -h /mnt/small
lsof +L1
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       15M   13M  1.2M  92% /mnt/small
python3 2302 student 3r   REG    7,0 12582912     0   13 /mnt/small/log.bin (deleted)

lsof +L1 lists open files with zero links, meaning deleted but still held. The space came back only when that process stopped. The fix in real life is to restart the service that holds the file, or to empty the file in place with truncate -s 0 instead of deleting it. Use sudo lsof +L1 to see other users’ processes. The PID will differ on your machine.

Self-check

  1. Which command checks fstab without rebooting?
  2. Why add nofail to a data disk’s options?
  3. The disk is full. What two commands find the biggest folders?
  4. Why never fsck a mounted filesystem?

Answers: (1) sudo findmnt --verify. (2) So a missing disk does not drop the machine into emergency mode. (3) du ... | sort -h (and df -h to confirm). (4) The filesystem is changing under fsck and it can cause corruption.

Sources