Skip to content

Why Cargo Ate 45GB Overnight (And How I Fixed It)

· Tom Tang · 6 min read
rust cargo build-optimization incremental-compilation ai-agents developer-tools

Midnight. My M-series Mac is thermal throttling. top shows a test binary eating 880% CPU. Three stale cargo test processes are orphaned from dead agent sessions. target/ is 45GB across three git worktrees.

This is the story of how I diagnosed it, what actually worked, and — more importantly — what I rejected after empirically disproving my own assumptions.

TL;DR: Rust’s target/ directory ballooned to 45GB because Cargo’s incremental compilation cache never garbage collects, and each unique (crate, features, profile) combination creates a separate ~700MB cache directory. With multiple AI agent sessions running cargo test across git worktrees, this meant 31 cache directories for a single crate. The fix: cargo-hakari to unify feature flags (793 → 81 .rlib files), CARGO_INCREMENTAL=0 for one-shot commands, and a flock-based build queue (cq) with process group isolation to prevent orphaned builds.

The Setup

I’m building claude-view, a Rust + React monitoring tool for Claude Code sessions. The workspace has 5 crates, 7 feature flags on the server crate, and multiple git worktrees so parallel AI agent sessions can work on different features simultaneously.

Each agent session runs cargo test or cargo check independently. Sounds reasonable. Here’s what actually happens:

Root Cause #1: Cargo’s Incremental Cache Never GCs

Every unique (crate, features, profile) combination creates a separate incremental compilation cache directory — around 500MB to 1GB each. Cargo never deletes stale ones.

target/debug/incremental/
  claude_view_server-29hup4e92f3dv/   1.0 GB
  claude_view_server-1zu8e7cdrpz54/   831 MB
  claude_view_server-16ko9ing0uwy2/   831 MB
  claude_view_server-0jatfr97mrful/   830 MB
  ... 27 more server hashes ...

31 directories for one crate. Why? Because:

  • cargo test → hash A
  • cargo test --features codegen → hash B
  • cargo build → hash C
  • cargo build --features codegen → hash D
  • … × multiple agent sessions = 31 hashes × ~700MB = 15GB for one crate alone

Across all 5 crates: 379 incremental directories, 45 unique crate variants.

Root Cause #2: Every Dependency Compiled 7 Times

Without feature unification, each crate pulls in slightly different feature sets for shared dependencies:

hashbrown:  7 copies
getrandom:  6 copies
sha2:       5 copies
url:        5 copies
idna:       5 copies

793 .rlib files in target/debug/deps/. Most of them are duplicates with slightly different feature flags enabled.

Root Cause #3: No Process Isolation

When an agent session ends, its cargo test process might still be running. Cargo spawns rustc, which spawns the linker, which spawns test binaries. Nothing ties these processes to the session that created them. They become orphans, eating CPU forever.

The 880% CPU test binary? An orphaned cargo test process from a session that ended 20 minutes ago.

What I Built

Fix 1: Build Queue (cq)

A wrapper script that serializes all cargo commands across worktrees using flock. One build gets all CPU cores (burst mode), others wait in line.

macOS doesn’t have the flock CLI, so the script uses perl’s Fcntl::flock:

# Instead of:
cargo test -p my-crate

# Use:
cq test -p my-crate
# → acquires flock → runs with all cores → releases on exit

Three protections in one script:

  1. flock queue — burst mode, no CPU oversubscription
  2. CARGO_INCREMENTAL=0 for one-shot commands (test/check/clippy) — prevents write-only cache bloat
  3. Process group isolation — setpgid(0,0) puts cargo + all children in a dedicated group. Kill cq → kill(-TERM, -$pgid) → zero orphans

Fix 2: cargo-hakari

cargo-hakari creates a “workspace-hack” crate that forces all workspace members to use the union of all features. Every dependency compiles once.

cargo install cargo-hakari
cargo hakari init workspace-hack --yes
cargo hakari generate
cargo hakari manage-deps --yes

Originally built by the Diem team at Meta, now maintained by the nextest team. ~1M downloads.

Fix 3: Incremental = 0 for Agents

The cq script sets CARGO_INCREMENTAL=0 for test, check, clippy, bench, and doc commands. These are one-shot — the agent runs them once and exits. The incremental cache is write-only overhead.

For build and run (your interactive dev loop where you edit → compile → test), incremental stays ON.

The Results

Clean build to a fresh target directory, before and after:

MetricBeforeAfterChange
.rlib files79381-90%
hashbrown copies71-86%
Incremental cache dirs3790-100%
Max CPU (concurrent builds)880%100%queue serialized
Orphan processes after kill3+0process groups

What I Rejected (And Why)

Every fix was adversarially audited before implementation. I challenged each claim with empirical benchmarks. Four proposals got rejected:

Cranelift backend — Replaces LLVM with a faster compiler for debug builds. Sounds great. Reality: nightly-only, requires a full toolchain dance (RUSTC=$(rustup which --toolchain nightly rustc) + nightly cargo binary + unstable config flags). And the kicker: cargo check doesn’t do codegen at all — cranelift only helps build and test, not the most common dev command. Too fragile for a dev tool.

setsid for process groups — The standard Unix way to create a new process group. Except setsid doesn’t exist as a CLI on macOS. which setsid → not found. Had to use perl -MPOSIX -e 'setpgid(0,0)' instead. And discovered a classic footgun: if the parent calls setpgid on itself, kill -$pgid kills the parent too.

codegen feature always-on — The codegen feature gates #[derive(TS)] for TypeScript type generation. I hypothesized that always enabling it would prevent duplicate builds. Benchmarked it:

WITHOUT codegen: 57.3s
WITH    codegen: 76.1s (+33%)

+19 seconds from ts-rs proc macro overhead. Not worth it.

sccache for cross-worktree sharing — sccache was already installed as rustc-wrapper. I assumed it was caching Rust compilations across worktrees. Benchmarked it:

sccache Rust cache hit rate: 0.00%

Zero. sccache doesn’t cache cargo check output (metadata-only, not cacheable) and shows 0% Rust hits even for cargo test across different target directories. It only caches C/C++ dependencies (ring, aws-lc-sys). The “55% lifetime hit rate” in sccache --show-stats was entirely from C/C++ deps.

The Lesson

The build wasn’t slow because Rust is slow. It was doing 31× the work.

Before reaching for compiler flags, alternative backends, or caching layers — check whether you’re actually compiling the same code multiple times. Feature flag combinatorics and incremental cache accumulation are the silent killers. cargo-hakari and CARGO_INCREMENTAL=0 for CI/agents are the highest-ROI fixes, and they’re both well-proven at scale.

Measure first. Reject what doesn’t survive empirical testing. The fanciest optimization is worthless if it solves the wrong problem.