The Janitor and the CEO: Finding High-Leverage Points on a Full Disk

Junior/Mid Engineer Asked at: FAANG, Startups, Cloud Providers

Q: Show the top 5 largest directories under /home.

Why this matters: This isn't a trivia question; it's a proxy for determining if you can handle a real crisis. A full disk is one of the most common and disruptive "grey failures" in any system. It doesn't crash the server, it just slowly strangles it to death. Your ability to find the problem's epicenter in seconds is a critical operational skill.

Interview frequency: High. This is a fundamental system diagnostic task.

❌ The Death Trap

The candidate attempts a manual, brute-force approach. They suggest `ls -lh /home` and then say they would `cd` into each directory and run `du -sh` one by one. This is the equivalent of using a teaspoon to empty a swimming pool. It's slow, error-prone, and demonstrates a complete lack of fluency with the power of the command line.

"The fumbling answer: 'I'd run `du -sh /home/*` and then just look through the list for the biggest ones.' This works for 3 directories. For 300, it's a recipe for mistakes and wasted time, especially when a system is lagging and every command is painful."

🔄 The Reframe

What they're really asking: "Can you think in pipelines? Can you compose simple, universal tools to transform a chaotic list of data into a prioritized, actionable insight?"

This reveals if you understand the core philosophy of Unix/Linux: small, sharp tools that do one thing well and can be chained together. They want to see you build a data-processing machine on the fly, not just execute a single memorized command.

🧠 The Mental Model

I call this "The Data Funnel." It's a universal pattern for investigation on the command line: Generate, Order, Select.

1. Generate: Get the raw data. In this case, get the disk usage for all relevant directories. This is the wide mouth of the funnel.
2. Order: Sort the data by the metric that matters. Here, it's disk size, from largest to smallest. This narrows the funnel.
3. Select: Isolate the most critical items. We only care about the top 5. This is the narrow spout of the funnel, giving you a precise answer.

📖 The War Story

Situation: "Our shared CI/CD build server was becoming unusable. Builds that normally took 10 minutes were timing out after an hour. The entire engineering team was effectively blocked from deploying."

Challenge: "The server wasn't down, it was just agonizingly slow. `df -h` showed the `/home` partition, where all the build workspaces lived, was at 98% capacity. The system was choking on I/O operations."

Stakes: "This was a productivity outage. With 50 engineers blocked, the company was burning thousands of dollars an hour. We couldn't push critical bug fixes or new features. The business was at a standstill because of one misbehaving server."

✅ The Answer

My Thinking Process:

"My first hypothesis was that a build process had gone rogue and filled its workspace with artifacts. I needed to find the needle in the haystack without bringing the server to its knees with a heavy command. I needed a fast, low-impact way to get a prioritized list. This is the perfect use case for the `du | sort | head` pipeline."

What I Did:

"I built the command piece by piece using the Data Funnel model:"

du -sh /home/* | sort -hr | head -5

"And this is how I'd explain it to the interviewer:

  • Generate: `du -sh /home/*`
    • du: The 'disk usage' command.
    • -s: Summarize. Gives me one total for each directory in the path.
    • -h: Human-readable. Shows `2.1G` instead of a meaningless block number like `4321876`.
    • /home/*: The wildcard expands to every directory directly under `/home`.
  • Order: `| sort -hr`
    • |: The pipe. It connects the output of `du` to the input of `sort`.
    • sort: The sorting command.
    • -h: Human-numeric sort. This is the magic flag. It understands that `1G` is greater than `500M`. Without this, a standard sort would incorrectly place `500M` before `1G`. This flag shows you know the details.
    • -r: Reverse the sort, so the largest is at the top.
  • Select: `| head -5`
    • |: Another pipe, sending the sorted list to the final command.
    • head -5: Shows only the first 5 lines of the input it receives.

The Outcome:

"The command executed in under 10 seconds and immediately showed that a single workspace, `/home/legacy-android-build`, was consuming 3.5TB of the 4TB disk. It turned out a new developer had accidentally checked in a flag that disabled artifact cleanup. We cleared the workspace, fixed the build script, and within minutes the server was healthy and the entire engineering team was back to being productive."

What I Learned:

"A CEO's job is to find the highest-leverage action to move the company forward. A janitor's job is to clean up messes. On a dying server, an engineer has to be both. This command is the ultimate high-leverage janitorial tool. It lets you ignore the 99% of things that don't matter and focus on the one thing that does."

🎯 The Memorable Hook

You can't hold the state of a complex system in your head. The goal isn't omniscience; it's effective inquiry. The `du | sort | head` pipeline is a form of inquiry. It's a question you ask the system, and it gives you a perfectly prioritized answer.

💭 Inevitable Follow-ups

Q: "What if you needed to find the top 5 largest *files*, not directories, on the whole system?"

Be ready: "For that, `find` is a better starting point. `find / -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 5`. This searches for files, prints their size in bytes and their path, discards permission errors, sorts numerically in reverse, and shows the top 5."

Q: "Your `du` command is running very slowly on a system with millions of files. How could you speed it up?"

Be ready: "The `du` command can be slow because it has to traverse every subdirectory. A common strategy on a large filesystem is to run it with a limited depth: `du -h --max-depth=1 /home | sort -hr | head -n 5`. This gives you the summary for each home directory without going all the way down, which is much faster."

Written by Benito J D