The QD~1 KV-load bottleneck
found with eBPF, fixed with parallel loads

01The symptom

LMCache offloads LLM KV cache to raw NVMe (the raw_block engine, io_uring_cmd passthrough on /dev/ng). Each KV object is a large blob — 8–20 MiB per rank — split into 128 KiB device commands. Loading it back should be near sequential-read speed. It wasn't.

The device is not the bottleneck — the software path is. The only way to know that for sure is to look at the actual NVMe commands on the wire.

02The wire evidence

The eBPF tracer nvme_uring_cmd_monitor records every NVMe command on the io_uring_cmd path — opcode, LBA, byte count, the issuing thread, the io_uring user_data, and a nanosecond timestamp. Replaying a real 70B/TP4 load and reading the trace back:

The trace also carries the LMCache trace_id in the high 32 bits of user_data, so every command is attributed back to the KV object (and TP rank) that caused it — that join is what makes this a precise measurement rather than an aggregate guess.

03Why this matters — the crossover turns on it

Whether it is cheaper to reload a prefix's KV from storage or to recompute it on the GPU is a live question in LLM serving. With TP held fixed at 4, and load time set purely by the (crippled) ~1.2 GB/s path, the balance decomposes cleanly:

Load time ≈ aggregate_bytes ÷ 1.2 GB/s, so it grows with KV bytes (×2.27), while recompute grows with compute (×5.0). R improves ×2.2 at fixed TP4 — a real model-scale effect, isolated from TP. But that load number is a fixable software floor, not storage physics. Speed the loader up and R falls with it — the crossover moves toward smaller models.

04Reproduce it

All three steps run on any box with an NVMe char device (/dev/ngXnY) and the LMCache raw_block engine built. No GPU or model needed — storage geometry is content-independent.

1 · Build the eBPF tracer

ebpf-syscall
# in the ebpf-syscall tree; needs clang + libbpf-dev + bpftool
make LIBBPF_SYSTEM=yes nvme_uring_cmd_monitor

2 · Measure the load path under varying concurrency

kvio_parallel_probe.py
# stores 128 × 20 MiB objects, then loads them back at W = 1,2,4,8,16,32
KVIO_SRC=~/LMCache python kvio_parallel_probe.py \
    --device /dev/ng1n1 --obj-mib 20 --nobj 128 \
    --workers 1,2,4,8,16,32 --qd 8

3 · Confirm the wire behavior

analyze_load_path.py
# run the tracer around a load, then attribute commands to objects/ranks
sudo nvme_uring_cmd_monitor --jsonl trace.jsonl --lba-size 4096 &
# ... issue the load ...
python analyze_load_path.py trace.jsonl sem.jsonl   # prints TIDs, in-flight objects, per-cmd gap

05The fix

The per-object read already released the engine lock before doing I/O, and the Rust io_uring binding drops the GIL during the wait — so the objects of one batched load can simply be issued concurrently. The change factors the per-object body into _load_one_into and dispatches it over an optional pool sized by a new RawBlockCoreConfig.load_parallelism (default 1 = unchanged). The L2 adapter passes its existing num_load_workers (default 4) through, so a knob that previously only sized dispatch threads now actually parallelizes the read loop.

load_many_into — before → after
# before: one object at a time, each read awaited before the next
for i, (key, entry) in enumerate(items):
    self._read_buffers([entry.offset + header], [buf], [len], [total])   # QD~1

# after: issue several objects' reads at once over the load pool
if self._load_pool and len(pending) > 1:
    futures = [self._load_pool.submit(self._load_one_into, i, key, entry, objs, results, roe)
               for (i, key, entry) in pending]
    for f in futures: f.result()

06End to end, in a live serving stack

The numbers above are the storage leg. The open question was whether it moves user-facing TTFT once the DRAM→GPU copy, the connector, and scheduling join the path. Measured on a real vllm serve Llama-3.1-8B + LMCache MP stack, raw_block L2 on NVMe passthrough, KV resident on the device — it does.

Default connector config (mq_timeout=10s, recompute fallback on) — what a user sees:

With recompute rescue disabled (mq_timeout=90s) the isolated storage penalty is unmasked:

Batched TTFT is stable across both (779 → 763 ms) — it loads fast enough to never hit the timeout. Baseline swings 3154 → 11885 ms with the timeout, proving its slowness is the QD~1 load.

Why a naive serving A/B lies — four confounds

Each of these silently collapses the two arms to equal; each had to be instrumented out. Skip the guards and you "measure" no difference and wrongly conclude the fix does nothing.

07See it — the crossover on one timeline

Everything above is also a picture. The crossover measurements were captured with the full kvio tracing stack (eBPF device completions + schema-2 semantic trace + serving spans) and converted to Perfetto timelines (how-to). Below: the 8B cell — the crossover point — with the stock (QD~1) and fixed (batched) arms stacked. Same 18-prompt warm phase, same bytes, same drive; the only variable is the load path.

08What’s left — honest status

• Single-ring ceiling (~5 GB/s, ~44% of device). eBPF shows the thread-pool still funnels through one submitter; the per-command gap only drops 83 → 54 µs. Reaching device bandwidth needs multiple io_uring rings (true multi-submitter), or intra-object pipelining in the engine — a follow-up prototype.
• Skip the DRAM bounce entirely. The real endgame is NVMe→GPU DMA (GPUDirect Storage / cuFile, or io_uring into a GPU dma-buf), which removes both the software round-trips and the host-memory copy the probe didn't even count.
• Storage must scale with TP. Per-rank objects are 1/TP the size, but the aggregate crossing one shared drive is constant — so the "storage scales with GPUs" case (per-rank drives / striping) is the decisive test for a production crossover claim.
• Frontier caveat. MLA models (DeepSeek-class) replicate KV per rank, so aggregate offload bytes grow ×TP — offload may never cross on a shared tier there.