Running a Small Language Model on Laptops: Speculative Decoding and KV-Cache Prefix Reuse in a Rust Inference Runtime

The Context: "The Laptop That Could Not Keep Up"

It was August 2026. Khepri — my autonomous small-language-model agent — had become genuinely useful. It could read my repository, decide which files mattered, and propose changes. What it could not do was keep up.

The model was a 3B-parameter quantised build running entirely on my workstation, because I refuse to send my source code to somebody else's inference endpoint. On my laptop's CPU it generated a meager 7 tokens per second. Every tool call it decided on cost two full seconds of staring at a frozen terminal. The agent loop, which needed to plan, act, and observe five or six times to finish a task, took the better part of a minute.

The naive fix was to use a larger, faster model on a rented GPU. That fix meant shipping my entire private codebase to a third party every time the agent wanted to read a file. So instead of paying the privacy tax, I treated this as what it actually was: an inference-runtime engineering problem. Two techniques closed most of that gap on hardware that already existed.

The Problem Statement

Serve a multi-turn agent loop on a CPU-only workstation without the agent feeling like it is wading through treacle:

• Throughput: Sustain at least 25 tokens/sec sustained decode on a single consumer machine with no accelerator.
• Latency: Keep per-token decode P99 under 40ms so the terminal never appears to hang mid-sentence.
• Context Reuse: An agent loop re-sends the same system prompt and repository preamble on every turn. Naively, each turn re-processes thousands of tokens that did not change.
• Privacy: Zero network egress. No prompt, no code, no telemetry leaves the machine.

Visual Architecture

+-RESETFULLSCREEN

The pipeline splits the work: a small draft model proposes several tokens cheaply, and the large target model verifies them in a single batched forward pass. It is the same principle as a proofreader who reads fast and an editor who only checks what the proofreader flagged.

The Core Logic / Process

1. Prefix-Cached Prefill

The agent loop's prompt is highly redundant. Turn 1 sends a 4,000-token system prompt plus tool definitions. Turn 6 sends that same 4,000 tokens, then another 1,200 tokens of tool output.

The prefill engine hashes each prompt chunk and maintains a radix tree of previously computed keys and values, keyed by content rather than by position:

pub struct PrefixCache {
    nodes: HashMap<u64, CacheNode>,   // content hash -> node
}

impl PrefixCache {
    /// Returns the longest cached prefix (in tokens) shared with `prompt`.
    pub fn longest_match(&self, prompt: &[u32]) -> usize {
        let mut matched = 0;
        for (i, &token) in prompt.iter().enumerate() {
            // Walk the radix tree; a miss ends the match.
            if !self.node_at_depth(i).has_token(token) {
                break;
            }
            matched = i + 1;
        }
        matched
    }
}

On a typical agent turn only 15–25% of the prompt is new, so prefill drops from a full 4,000-token forward pass to roughly 900 tokens. That single change took first-token latency from 1.8s to 380ms.

The critical subtlety: the cache key must include the token itself and its full prefix, never just the token. A naive implementation keyed on tokens alone will happily serve a cached key from an unrelated context and silently corrupt the model's attention — producing fluent, plausible, confidently wrong output. This is the single easiest way to build a content-addressed KV cache wrong.

2. Speculative Decoding

Autoregressive decoding is latency-bound, not compute-bound, on CPU. The weights are already in cache; the bottleneck is the serial dependency where token N cannot begin until token N-1 exists.

Speculative decoding breaks the serial chain without changing the output distribution. A 1B draft model emits 5 candidate tokens autoregressively. The 3B target model then evaluates all 5 positions in one batched pass. Because attention is causal, that single pass yields valid predictive distributions for each position independently:

• If the target model agrees with the draft at every position, accept all 5 tokens.
• On the first disagreement, accept the agreed prefix plus one target-sampled correction token, discard the rest, and re-enter the draft model.

This is exact, not an approximation. The output distribution is provably identical to running the target model alone, because every accepted token is a genuine sample from the target's own distribution at that position.

Accept rate falls off a cliff past γ=8. A longer draft wastes target compute re-validating a suffix that gets thrown away, and the KV cache for 12 speculative positions costs more than the arithmetic it saves. γ=5–8 is the sweet spot on CPU.

3. Keeping the Loop Responsive

Even at 25 tokens/sec, a blocking decode starves the terminal's event loop. All model work runs on a dedicated thread with a bounded channel; the UI polls a shared atomic for the latest partial token. The agent never blocks the interface it is reporting through.

The "Baigan" Take (Expert Insights)

The industry spent years treating inference throughput as a GPU procurement problem. Then a wave of engineers discovered that on constrained hardware, what you reuse matters more than how fast you compute.

baigan ka inference is the belief that "just use a bigger model" solves a latency problem. A bigger model does solve it — by moving your latency to someone else's bill, and your source code to someone else's disk. Neither is an engineering decision; both are procurement decisions wearing an engineering costume.

Three things I got wrong and would warn you about:

1. Aggressive quantisation of the draft model. I quantised the 1B drafter to 4-bit to match the target. Its accept rate collapsed from 72% to 41%, because a low-precision drafter proposes tokens the target almost never confirms. Speculative decoding needs the drafter to be faithful, not small. I moved it to 8-bit and lost nothing.

2. Content hashing without position binding. Covered above, and it is worth a second mention because it fails silently. Your agent starts returning subtly wrong tool arguments and you will blame the model.

3. Assuming the KV cache is free. It is not. An unbounded prefix cache on a long agent loop will exhaust RAM within a dozen conversations. I cap it with an LRU keyed on prefix length, and evict the shortest prefixes first — they save the least time and cost the same memory.

The real lesson: a private, local agent is not a compromise you accept for privacy. With prefix caching and speculative decoding, it is genuinely faster than the API call it replaces, because you skip the network round-trip entirely.

Performance Checklist

• Key the KV cache on content plus full prefix, never on token position alone.
• Sweep speculative draft length (γ) empirically; do not trust a default.
• Keep the draft model at higher precision than feels justified — accept rate is the metric that matters.
• Bound the prefix cache with LRU eviction, shortest-prefix-first.
• Run decode on a worker thread; never block the UI event loop on token generation.
• Validate speculative decoding output against the target model on a fixed corpus before trusting the equivalence proof in practice.