Slower by default

Summary

Speculative decoding in llama.cpp can be tuned with a draft-confidence gate, the --spec-draft-p-min CLI flag. Since 2026-05-19 that flag has defaulted to 0.00: no draft is ever cut short for low confidence. Before that it defaulted to 0.75 for about fifteen months, and to 0.9 for the three months before that. Enabling speculation without setting the flag selects the build's shipped default: 0.00 on a current build, 0.75 on a build from February 2025 through May 2026.

On 2026-08-10 we measured what the gate does on one specific deployment: Qwen3.5-397B-A17B in Unsloth's UD-IQ3_XXS quantization, served by llama.cpp with 57 of its MoE layers' experts held in CPU RAM, on one RTX 5090. Five treatments (no speculation, the 0.00 default, and gates of 0.50, 0.75 and 0.90) on llama.cpp's draft-mtp self-speculation path (section 02 explains it), in two counterbalanced cycles over sixteen fixed prompts, half prose and half structured output. 160 measurements, every one paired against the no-speculation baseline within the same prompt and cycle.

Five results:

1. The shipped default is a prose regression on this deployment. Ungated speculation made prose 19.8% slower than no speculation at all [95% CI −24.7 to −17.9] while making structured output 22.5% faster [+7.6 to +41.9]. Same run, same baseline, opposite signs.
2. Every tested gate at or above 0.50 reverses it. Prose at +18.6% / +21.9% / +22.8% for gates 0.50 / 0.75 / 0.90; structured at +38.4% / +47.0% / +47.7%. Against the shipped default, setting any of those three gates was worth roughly +50% on prose, with every interval clear of zero. On structured output that same contrast is smaller, and at 0.50 it is not resolved from zero.
3. A single pooled number represents neither workload. Pooled across both output shapes, the default reads −4.45% with a 95% interval of [−19.6, +19.6], while its two components sit at −19.8% and +22.5%, both nominal intervals excluding zero. Reporting a single pooled number for a workload that mixes prose and structured output does not represent either one.
4. Speculation is not output-preserving at temperature 0, gated or not. Prose came back byte-identical to the baseline 0 times out of 16 under every speculative setting. The baseline itself is byte-reproducible across restarts (16/16), so the divergence belongs to speculation. Most of the divergent pairs compare two answers that both ran to completion; section 08 separates truncation from wording.
5. Below 0.50, the reversal is absent. Gates of 0.10 and 0.25, measured the following night against their own same-night baseline, were prose regressions of −32.0% [95% CI −38.6 to −27.4] and −23.7% [−27.9 to −20.1]; neither low gate's structured contrast was resolved from zero in that run. The crossover from prose regression to gain is therefore bracketed between 0.25 and 0.50, a bracket that spans the two runs' two nights, with the between-night axis uncontrolled (section 07).

All intervals on this page are per-contrast cluster-bootstrap intervals with no multiplicity correction, and calibration drift is visible across the evening; sections 03 and 11 state those limits.

None of this is a bug report. The maintainer changed the default deliberately, documented a real tradeoff when he did (section 06), and the code behaves as designed. What this page adds is a measured curve with stated uncertainty (five arms in the main run, two more in a next-night follow-up) on a regime absent from the public comparisons we could retrieve, a CPU-offloaded ~400B MoE, together with the provenance of the number everyone recommends.

Environment

Everything below is a property of one model file, one runtime build, one placement and one request shape, on 2026-08-10, on one workstation. Nothing here measures any model's intelligence or benchmark standing, and no such claim appears on this page.

What the flag does, in the code we ran. The draft-mtp speculation type drafts from the model's own multi-token-prediction (MTP) head; there is no separate draft model. The gate is a draft-stop threshold, not a per-token filter: while a draft is being extended, if the draft's top-token probability falls below p_min, drafting stops for that sequence (common/speculative.cpp:316-321 at our build, whose comment reads “only collect very high-confidence draft tokens”; the flag's help text at common/arg.cpp:3588-3589 reads “minimum speculative decoding probability (greedy)”). At 0.00 drafting never stops early, so every draft extends to n_max and is submitted for verification, which is what the lede means by every draft being submitted in full. We measured only the draft-mtp path: the stop-check lives in llama.cpp's common speculative module, but we did not measure the classic separate-draft-model path (--model-draft), and none of this page's numbers should be assumed to transfer to it.

The full command line for every one of the ten server loads is preserved in the data package, and the running process's real argv was read back from /proc/<pid>/cmdline on every row, so the treatment each session measured is the treatment its label claims. The VRAM step between baseline and speculative loads is a third, independent witness that speculation was actually loaded.

The experiment

Ten server loads: five treatments × two cycles, cycle 2 in exact reverse order, one fresh process per session. Sixteen measurement prompts per session (plus a warmup and calibration probes, excluded from analysis), 160 measurements in all, zero excluded, every session returning 16/16 usable with exit 0.

A follow-up run the next night (2026-08-11) extended the design below 0.50 with its own census (six server loads: a fresh no-speculation baseline plus gates 0.10 and 0.25, two counterbalanced cycles, 96 measurements, zero excluded) on the same instrument, the same sixteen prompts, the same build and placement; section 07 reports it, and its 96 measurements are never pooled with this run's 160.

The analysis was pre-registered in the run log before row 1 existed, written in the same session, immediately before the first measurement; a run-log entry, not a third-party registry: the primary endpoint is decode rate in tokens per second, paired within (prompt, cycle); prose and structured are reported separately and never pooled first; the primary contrast is each treatment against no speculation; adjacent-gate contrasts are secondary and exploratory. Intervals are 95% cluster-bootstrap intervals (2,000 resamples, percentile method, no BCa correction), and no multiplicity correction is applied: each interval describes one contrast's uncertainty, and the collection does not have 95% simultaneous coverage.

Decode rate is client-observed from the stream: visible token events divided by the time from the first to the last visible token, a window that excludes prompt processing and time-to-first-token. That estimator is biased high by N/(N−1), so an unbiased interval-basis variant was recorded alongside it with a pre-registered tiebreak; the two never diverged by more than 0.02 percentage points on any headline cell.

A decision rule for “the knee” was also written down in advance, before any data: the lowest gate whose prose contrast against no speculation is positive with a 95% interval excluding zero, and whose contrast against the next higher gate has an interval containing zero. Section 07 reports what that rule did and, just as carefully, what it cannot show.

Observed: the default is a prose regression, and any gate reverses it

Median decode rates and paired median % change against the no-speculation baseline (n = 16 pairs per cell; brackets are 95% cluster-bootstrap intervals).

Prose: baseline median 10.20 tokens/s

Structured: baseline median 10.25 tokens/s

All figures from the recorded analysis output (data/analysis/analysis.md); reproduced to the last digit by a recompute from the raw records on 2026-08-11 (same project, fresh analysis pass).

The same speculation that speeds structured output up by a fifth slows prose down by a fifth, in the same run, against the same baseline. That is the finding this page exists to publish, and it is why no single number can summarise this flag.

The contrast a deployer actually faces is not against no speculation. It is against what the flag does out of the box:

On prose, each of the main run's three gates was worth roughly +50% against the shipped default, with all three intervals clear of zero. Do not carry that sentence over to structured output: the structured 0.50-vs-default interval contains zero, and the structured gains against the default are several times smaller.

The two tables in this section share the same underlying measurements and differ only in baseline. The ~+50% figures against the shipped default and the +19–23% figures against no speculation are two views of one effect; they do not stack.

Observed: one pooled number represents neither workload

If the two subsets are pooled, the default treatment reads:

That interval does not resolve the direction of the pooled effect, and the point estimate is small. Split, the components are −19.8% on prose and +22.5% on structured, with both nominal intervals excluding zero. The pooled figure is a different estimand: a wide interval around it is uninformative about either subgroup, and a narrow one would still describe only the mix it was computed on. Reporting a single pooled number for a workload that mixes prose and structured output does not represent either one.

The previous evening's exploratory run on the same machine (a different prompt set, with 0.75 as its only gated arm) recorded a pooled −6.5% for the ungated default, close to this run's −4.45%, a mild reproducibility note, though the different prompt sets license no strict comparison in either direction.

Observed: the mechanism, in the server's own counters

Every record carries the server's draft counters (the draft_n / draft_n_accepted timing fields on the streamed response), so acceptance (the fraction of drafted tokens the model kept) is available per observation (n = 32 per treatment, 16 per subset). Ranges observed:

Computed from the per-observation counters in every record (server_diagnostic.timings.draft_n / draft_n_accepted), not from log-line samples; this is the corrected computation from the 2026-08-11 verification pass.

On prose the ranges do not even overlap between adjacent treatments: ungated speculation got only a fifth to a third of its drafted prose tokens accepted, and acceptance climbs monotonically with the gate. Our reading of the split (an interpretation consistent with the placement, not an instrumented result) is that every rejected draft wastes verification work this CPU-bound deployment can least afford, and prose is where the drafts fail. Structured ranges are wide and overlap heavily (several structured prompts reach 1.00 at every gated setting, and the structured minimum is ~0.72 at both 0.75 and 0.90), which is consistent with structured output speeding up even ungated: its drafts are good.

The follow-up run (section 07) carries the same per-observation counters, exposed on all 32 observations in each gated arm, and its rows are tabulated separately because its baseline and process cohort are its own:

The split is stark at the low end too: prose acceptance sits in a ~0.21–0.40 band at these gates while structured reaches 0.89, consistent with the acceptance-monotone story continuing downward, and with low-probability drafts being generated and mostly rejected on prose: wasted draft work. Consistent-with is the ceiling here as above: these counters establish correlation between gate, acceptance and output shape, not a causal decomposition of the per-token cost.

The maintainer's stated tradeoff, and what this study can and cannot say about it. In the pull request that dropped the default to 0.00 (llama.cpp #23269, 2026-05-19), ggerganov states that a positive gate produces differently-sized drafts, which prevents CUDA-graph reuse, and advises using the gate “with caution”. That cost should weigh most where graph launch overhead is a large share of each step (small models running fully on GPU) and least in a regime like ours, where decode waits on CPU memory. But this study did not instrument that mechanism: we measured no graph-reuse statistics, no memory-bandwidth counters, and no draft-versus-verify time split, any of which is what confirming it would take. What we can say is that the two published signs are consistent with the two sides of the tradeoff the maintainer described: consistency, not confirmation.

When might the sign flip? An interpretation, not a measurement. The record anchors exactly two points on the same GPU model: our CPU-offloaded 397B MoE, where every gate at or above 0.50 helped prose, and cmp-nct's fully-VRAM-resident 27B (section 10), where the gate cost 14%. Read through the maintainer's stated tradeoff, the dividing line would run somewhere between those regimes: as more of the model leaves VRAM and decode slows toward CPU memory speed, wasted draft verification costs more and graph reuse matters less. Where that line sits, and whether a single line describes it, is unmeasured; if your deployment sits between those poles, the only reliable answer is your own A/B (section 12).

The gate ladder, read carefully

A decision rule written before any data existed (the lowest gate whose prose contrast against no speculation is positive with an interval excluding zero, and whose contrast against the next higher gate has an interval containing zero) was first satisfied at 0.50, the lowest gate in the main run:

Here is what that does and does not mean, stated as carefully as we can:

• At 0.50, prose was 18.6% faster than no speculation. The 0.75 and 0.90 settings measured +21.9% and +22.8%, but neither adjacent increase above 0.50 had a 95% interval excluding zero, although those intervals permit additional gains of roughly 5.6% and 3.0%. This experiment does not resolve whether raising the gate above 0.50 helps. An interval containing zero is a statement about our resolution, not evidence of equivalence.
• 0.50 is the lowest gate in the main run, so that run's data alone cannot distinguish a threshold at 0.50 from one below it. Whether 0.10 or 0.25 would do as well was the open edge the follow-up run below was run to measure.
• The structured adjacent contrasts are even less separable (0.75 vs 0.50: +4.64% [−3.42, +12.25]; 0.90 vs 0.75: +6.19% [−2.27, +8.65]): wider intervals, and the +4.64% point estimate is a reminder that “contains zero” describes our design's power, not the model.
• What is solid and does not depend on any of this: on prose, each of the main run's gates (0.50, 0.75 and 0.90) beats the ungated default by +49.8% to +53.1% with all corresponding intervals excluding zero, and the ungated default is a genuine prose regression against no speculation at all. That is the finding. The ladder's shape above 0.50 is the secondary, weaker result.

Below 0.50: measured the following night

A second run on the same machine the following night (2026-08-11, 00:09–01:05 EDT) measured that open edge directly: same build, same placement, same sixteen prompts, same instrument, with the same per-row treatment witnesses (a per-treatment server alias, the argv read back from /proc/<pid>/cmdline, the VRAM step). Six server loads: a fresh no-speculation baseline plus gates 0.10 and 0.25, two counterbalanced cycles, 96 measurements, zero excluded. Each low gate is paired against that run's own baseline, never against the first night's.

Prose: follow-up baseline median 11.04 tokens/s

Structured: follow-up baseline median 11.40 tokens/s

Both low gates are prose regressions against no speculation at all, with both intervals well clear of zero: the reversal measured at 0.50 and above is absent at 0.25 and below, and the measured prose gain had not yet appeared at 0.25. Within the regression region the curve does rise: 0.25 against 0.10 on prose is +13.67% [+7.74, +21.40], less bad, still below baseline. Neither low gate's structured contrast was resolved from zero in this run; that is a statement about resolution, not a null result: the 0.10 structured interval permits anything from −9.8% to +24.7%.

The six gate values now tested sit on two sides of a divide: 0.10 and 0.25 on the regression side (measured in the follow-up, internally controlled), 0.50 and above on the gain side (measured in the main run, internally controlled). The crossover from regression to gain is therefore bracketed between 0.25 and 0.50, and that bracket is a cross-run statement. Each half is internally controlled against its own same-night baseline, but the interpolation between the halves spans two runs on two nights, and the follow-up's baseline decoded roughly 5–10% faster than the main run's on identical configuration, a between-night shift of unknown origin that nothing in either design controls, and that is itself larger than several of the contrasts in play. Where the crossover falls within (0.25, 0.50), and the curve's shape inside that interval, is unmeasured; locating it would take a same-night run holding arms on both sides of it.

Taken at face value the low end reads non-monotone: the ungated 0.00 arm (−19.8%, main run) against gate 0.10 (−32.0%, follow-up). A recompute from the raw records resolves that pair: the 0.10 arm's outputs are byte-identical to the main run's ungated outputs on 32 of 32 (prompt, cycle) pairs, across the two nights and different server processes, and its draft counters match on 22 of 32 pairs, the rest differing by at most a dozen drafted tokens. The 0.10 stop threshold fired rarely and changed nothing visible: on this model and prompt set, gate 0.10 was operationally near-identical to running ungated, so the face-value gap is measurement variance on an effectively unchanged treatment, not a treatment effect. Between the nights the baseline route sped up (prose median 10.20 → 11.04 t/s, +8.2%) while the effectively unchanged speculative route slowed (8.12 → 7.56, −6.9%), opposite drifts that compound in any relative contrast. Two consequences. The effectively distinct arms tested below 0.50 are two, not three (the ungated-equivalent 0.10 and the genuinely gating 0.25, whose outputs differ from the main run's ungated on 18 of 32 pairs), and the (0.25, 0.50) bracket above is unchanged. And the pair is a direct caliper on between-night variance, larger than the baseline-only shift, which strengthens the between-night caveat on that bracket.

The follow-up settles nothing at or above 0.50: the adjacent contrasts there remain exactly as unresolved as the bullets above state. Its acceptance counters are in section 06 and its output-equivalence results in section 08, labelled by run.

Observed: speculation is not output-preserving at temperature 0

The textbook account of speculative decoding is that verification makes it exact: the target model checks every drafted token, so you get the target's own output, just faster. At temperature 0 that account predicts byte-identical text with and without speculation. On this build and route, that is not what happened. Prose diverged under every setting we tested:

The attribution holds because we ran the control this claim requires: the two independent no-speculation sessions (separate processes, about 85 minutes apart, same seed, temperature 0) reproduced each other 16/16 byte-identical. The serving route is byte-reproducible across restarts; the divergence appears when speculation is enabled, and gating does not restore it. The gate is a throughput control, not a determinism control.

Truncation does not carry this result. Counting from the truncation census below, the prose pairs in which either member hit the 256-token cap number 6 (default 0.00), 5 (gate 0.50), 4 (0.75) and 6 (0.90) out of 16; the remaining 10, 11, 12 and 10 pairs compare two answers that both ran to completion, and every one of them still differs. All structured mismatches are between complete answers; no structured output reached the cap.

We did not identify the mechanism, and this page does not claim one. Exact verification at temperature 0 leaves a short list of candidates: batched verification changes floating-point evaluation order, and an implementation path can depart from the textbook scheme, but nothing in this run distinguishes among them. The control pins down only where the divergence enters: with speculation, on this build (c8e03ce), on this route.

Three fence-posts around that result:

• All 160 structured outputs passed the strict JSON validator in every treatment, including baseline, with zero fenced outputs. That establishes validator-validity, not semantic correctness: we did not assess whether the differing texts say the same thing.
• The run used a 256-token cap, reached on a minority of prose prompts: 12 of 160 observations (baseline: prose_05 and prose_07 in both cycles; default 0.00: prose_01 and prose_07 in both cycles; gate 0.50: prose_03 and prose_07, cycle 2; gate 0.90: prose_03 in both cycles; gate 0.75: none). The equivalence check therefore compares complete answers on most prompts and truncated answers on that minority, and byte differences on prose_01, prose_03, prose_05 and prose_07 can arise from truncation as well as wording in the treatments where they truncate.
• None of the truncated rows shows the empty-answer signature (near-zero visible characters against a spent budget): characters-per-token sits mid-band, and the run logged zero hidden-reasoning events in 160 measurements.

The follow-up run (section 07) extends this result downward. At gates 0.10 and 0.25, prose again reproduced the baseline 0 times out of 16, while that run's two baseline sessions reproduced each other 16/16 byte-identical, so no tested gate value, 0.10 through 0.90, preserves prose output at temperature 0 on this route. The recompute in section 07 also gives the speculative route a stronger reproducibility datum than the within-night restart control (the 0.10 arm reproduced the main run's ungated outputs byte-identically on 32 of 32 pairs, across two nights and different server processes), reinforcing that control, not replacing it. The follow-up's truncation census (9 of 96 measurements, all prose) shows the truncation set itself differing by treatment: the gated arms truncate prose_01, which the baseline completes, while the baseline truncates prose_05 and prose_07 exactly as it did in the main run. Truncating different prompts under different treatments is direct evidence that speculation changes what is generated, not just how fast; the follow-up's content-equivalence statements exclude the truncated prompts by name, as above.

One observation from that run, labelled as an observation because no mechanism is established: gate 0.25 reproduced its own prose output across its two fresh processes on only 1 of 8 prompts, while gate 0.10, the baseline, and every structured cell reproduced themselves 8/8. The main run's gated arms were never checked this way, so we cannot say whether 0.25 is special. As stated, it means this much and no more: under speculation, the generated prose was not a stable function of the configuration across process restarts in that run, and any equivalence claim conditioned on “same configuration, same seed” now carries that asterisk.

Where the famous 0.75 actually came from

If you search for this flag today you will find mid-2026 posts recommending --spec-draft-p-min 0.75 as the setting that “fixes” speculative decoding. We found no published sweep establishing 0.75 as a tuned setting; its documented provenance is a changelog. (The nearest thing to a public sweep, section 10's pastebin artifact, swept below 0.75 on a different regime and landed on 0.5.) Three commits in llama.cpp's own history, verified directly against a local clone of the repository:

So 0.75 was llama.cpp's own shipped default for roughly fifteen months. The pull request that set it (#11954) carries no benchmark and no sweep: its entire body reads “Better defaults for speculative decoding.”, and it was opened and merged the same day with no recorded discussion, unremarkable for a small default-only change by the project's maintainer, and noted here for one reason only: no measurement is attached to the value. The same commit also moved the minimum draft length from 5 to 0.

This reframes the whole story:

1. On this deployment's prose, the current default selects the slower treatment we measured. On 2026-05-19 the gate's default went from 0.75 to 0.00. The gate did not stop working; it stopped being on, and section 04's ungated row is what 0.00 measured here.
2. The mid-2026 “magic flag” posts recommend the value that was the default. A build between February 2025 and May 2026 shipped 0.75 without the user asking; a build after May 2026 ships 0.00, with the change recorded in the project's history rather than surfaced at runtime. Whatever each author knew, a post recommending 0.75 after the flip functionally re-supplies the prior shipped default by hand. One dated exception cuts against any tidy story: the earliest task-split measurement we cite (2026-05-12, section 10) predates the flip by a week. Its author reached a gated setting while 0.75 was still the default, not by losing it. The earliest gated-reversal write-up we cite is dated three days after the flip.
3. 0.75 should be described as an inherited project default, not a tuned value. Residue inside llama.cpp's own tree: the server README still ships example payloads with "speculative.p_min": 0.75, and the speculative-simple example README still shows the original 0.9. Outside the tree (a separate project, not llama.cpp residue), the ik_llama.cpp fork still defaults to 0.75.

Two dated negatives belong here. First, current Unsloth artifacts contain no p_min recommendation at all: we checked the raw READMEs of three MTP GGUF repositories and the Unsloth MTP documentation page on 2026-08-11 and found zero occurrences, so the “Unsloth recommends 0.75” framing that circulates is not supported by Unsloth's current docs. Second, nothing in this section is a criticism of the maintainer's change: the tradeoff he documented (section 06) is real, the TODO was in the code, and a project is entitled to change its defaults. The gap is that a silent default flip inverted the sign of prose-shaped decode on at least one class of deployment, and no measurement we could retrieve separated the classes before the folklore formed.

The public record: prior art, and three results that disagree

We claim no firstness on the gated reversal. The record we could retrieve:

• neoteric.no (2026-05-22) published the gated reversal on a Qwen3.6-27B Q4 on a 3090 Ti: autoregressive 29.0 t/s, gate 0.75 → 48.9 t/s (+68%). Their statement that ungated speculation is slower is qualitative (the blog's text says it collapses throughput, but their table has no ungated column), so we cite it as a qualitative claim, not a number.
• The earliest split-by-task result we found is a Chinese-language post on 掘金 (juejin), 2026-05-12: “MTP 加速不是玄学”: “MTP speedup is not black magic” (玄学: literally “metaphysics”, idiomatically “voodoo”). The title itself is on this page's side of the argument: the post's thesis is that MTP gains are tunable and measurable, not superstition. On a dense 27B it reported creative writing at −9%/−4% against +171% on coding, with acceptance 39–48% vs 79–89%, ten days before the earliest English-language post we found, and a week before the default flip. The scrutiny we apply to the pastebin artifact applies here too: it is a single post reporting point numbers, with no stated intervals or repeat counts, but unlike the pastebin sweep it did split by task, the split this page argues is the load-bearing one. URL: juejin.cn/post/7638806086187057178 (resolving as of 2026-08-11, when we confirmed the title and headline figures on the live page; whether it also appeared on other platforms is not something we established).
• A sub-0.75 sweep exists and concluded 0.5. A launch-config artifact from an r/LocalLLM post (~2026-07-01, preserved at pastebin.com/raw/P57Uk6rz) carries the inline comment that a gate of 0.5 was best across 2k/15k/64k context depths at 6 drafts (119/122/118 t/s against a 106–126 range for the others). The artifact itself names Qwen3.6-27B-MTP-Q6_K and a draft maximum of 10; the claims that the hardware was an RTX 5090 and the build 86b9470 come only from a secondary newsletter (smol.ai, 2026-07-02), not from the artifact, and we attribute them accordingly. Their sweep grid is unpublished: only the winner's three numbers and an aggregate range survive, whether 0.75 was even in the grid is unstated, and there is no interval and no prose/structured split. But it is a sub-0.75 result that landed on 0.5 before we did, and pretending otherwise would be the exact failure our prior-art gate exists to prevent.

And the reason a curve is worth publishing at all: the public two-point comparisons disagree about the sign.

Three public comparisons, three configurations, no agreement about which direction the flag moves throughput, and one of the three arrives without a stated setup, so it weighs less than the other two. Every one of them is two points. That is folklore's natural habitat, and it is the strongest argument for publishing a measured curve with intervals: not because our number is the right one, but because a curve with stated uncertainty on a stated regime is the format the argument has been missing. The cmp-nct row also matters for a second reason: it shows the opposite sign is real, on the same GPU we used, when the model is small and fully resident. Nothing on this page generalizes beyond the regime in section 02, and that row is the proof.

What this does not show

• One regime, one quant, one placement, one speculation path. 397B-A17B, UD-IQ3_XXS, --n-cpu-moe 57, ctx 65536, --no-mmap, flash-attn on, one request at a time, the draft-mtp path only. Nothing here generalizes to a different placement, and section 10's cmp-nct row measures the opposite sign on a small fully-GPU model. Measure your own lane.
• The ladder's shape above 0.50 is unresolved. The adjacent contrasts contain zero, which is a statement about this design's power (two cycles, sixteen prompts), not a demonstration that the settings match. The intervals permit real gains of up to ~5.6% (0.75 over 0.50) and ~3.0% (0.90 over 0.75).
• Below 0.50, the crossover is bracketed, not located. The follow-up run measured gates 0.10 and 0.25 against its own fresh baseline and found both to be prose regressions, so the crossover from regression to gain lies between 0.25 and 0.50. That bracket is a cross-run statement: its two halves come from different nights' runs, the follow-up's baseline decoded roughly 5–10% faster than the main run's on identical configuration (on prose the baseline sped up between nights, median 10.20 → 11.04 t/s, +8.2%, while the operationally unchanged speculative route slowed, 8.12 → 7.56, −6.9%: opposite drifts that compound in the relative contrasts; section 07), and the between-night axis is uncontrolled in both designs. The crossover's location within (0.25, 0.50), and the curve's shape inside that interval, is unmeasured. The follow-up's intervals are per-contrast with no multiplicity correction, like every other interval on this page. The pastebin sweep in section 10 landed on 0.5 on a different model and hardware.
• Counterbalancing buys monotone drift only. Two cycles in reverse order balance first-order position effects; restart-correlated effects remain, and two cycle clusters carry little information about restart variability.
• Calibration drift is visible and asymmetric. The largest drift is the default treatment's cycle 2 at +9.98%; the no-speculation baseline itself drifted +6.35% in cycle 1, the night's first cold process. That the baseline drifts materially is an argument for the counterbalanced design and against any single-cycle reading.
• No multiplicity correction. Five treatments × two subsets × two contrast families; each interval describes one contrast, and the collection does not have 95% simultaneous coverage.
• Output equivalence is byte-level only, on a 256-token cap reached by a minority of prose prompts (12/160), and establishes nothing about meaning.
• Treatment labels are client-side unverifiable by the instrument, mitigated three ways (a per-treatment server alias, the argv read from /proc/<pid>/cmdline, and the VRAM step); and the build identity rests on --version plus the streamed fingerprint, not on a binary hash, because the server binary is a thin launcher.
• Other inference stacks were not surveyed. This page did not examine whether vLLM, text-generation-inference or TensorRT-LLM expose an analogous draft-confidence control, nor how their speculative paths behave; nothing here speaks to any stack but llama.cpp at the stated build.
• This is not a model evaluation. Nothing here ranks Qwen against anything, and the speed effects belong to a runtime flag, not to the model's quality.

If you run llama.cpp speculation

Five narrow things, each scoped to a stated regime; none is a universal number:

1. Know which default you are inheriting. Since 2026-05-19, the --spec-draft-p-min CLI flag defaults to 0.00. If you enabled speculative decoding without setting it, you are running ungated. If your config predates May 2026 and never set it, your effective setting changed when you rebuilt.
2. Measure your own workload, split by output shape. On our deployment the same default was −19.8% on prose and +22.5% on structured output. A pooled number from a mixed test set would have left the direction of the pooled effect unresolved and would have represented neither half of what we actually run (section 05).
3. Do not assume speculation is output-preserving, even at temperature 0, even gated. If byte-stable output matters to your pipeline, test it on your own route: ours diverged on every prose prompt.
4. Expect the sign to depend on your regime. A CPU-offloaded MoE (ours) gained from gating at 0.50 and above; a small fully-GPU model on the same GPU (cmp-nct, section 10) lost. The maintainer's stated CUDA-graph tradeoff is consistent with that split, but this study did not instrument the mechanism (section 06). There is no universal setting to copy, including ours.
5. If your regime resembles ours, set the gate explicitly, and not low. Here is the minimal honest A/B. On this large CPU-offloaded MoE, every tested gate at or above 0.50 reversed the prose regression: 0.50 reversed it at +18.6%; 0.75 and 0.90 measured +21.9% and +22.8%, adjacent differences our design could not resolve from zero (section 07). A low gate is not a safe fallback on this regime: gates of 0.10 and 0.25, measured the following night against their own baseline, were worse on prose than no speculation at all: −32.0% and −23.7%, and 0.10 proved operationally near-identical to running ungated, byte-identical in output (section 07). On a small fully-GPU model the public record shows the opposite sign (section 10), so no value, including ours, travels without a measurement. The measurement is cheap: your real prompts, split prose versus structured; the same prompts on both arms; a fresh server process per arm; no prompt repeated within a session (llama.cpp #26100: reused prompts can bypass the gate); temperature and seed pinned; compare paired per-prompt decode rates.

Artifacts

The complete evidence for this page ships as a data package: the files the two runs actually produced, not a summary. If a number on this page disagrees with a file in the package, the file is right and the page is wrong; tell us and we will fix the page. The follow-up run's records (its own run log with pre-registration, flags, load witnesses, analysis outputs and per-observation records) are in the package under data/lowerarm/, kept as their own census under the same rules.

• data/README.md: what is in each folder, the record schema, and the provenance of every file.
• data/NUMBERS.md: every figure printed on this page, with the file and the field it came from.
• The pre-registration and run log (data/RUN_LOG.md): the knee decision rule and the analysis plan, written before row 1 existed, with the preflight record.
• The analysis outputs every table above is read from: primary and contrast analyses, the paired table, and the per-measurement CSV, under data/analysis/.
• The exact server command line for all ten loads (data/flags/) and the per-load witness JSONs (data/loads/): PID, load time, VRAM after load, resident set, and the speculation flags.
• The per-observation measurement records, one JSONL per session, carrying every request, every response, the server's timing and draft counters, and the streamed build fingerprint.
• The follow-up run's package (data/lowerarm/RUN_LOG.md, data/lowerarm/analysis/): the same layout for the 2026-08-11 run, kept as its own census.
• The preserved third-party artifact behind section 10's pastebin citation (data/third-party/), fetched 2026-08-10, because pastebin links rot.

Every claim on this page is dated. If today is much later than the rows above, treat version-specific findings as unverified since that date: the flag's default has already changed twice in the project's history, and may change again.

Related pages on this site

The wrong flag is this page's thesis on another llama.cpp default, the thread-count heuristic, on the same workstation: a misattribution that lived six days in our own records, corrected by a pre-registered three-arm A/B. Empty Answer is the same editorial thesis (an aggregate number that represents no real workload) in the thinking-budget domain, on the same workstation. The Qwen3.5 pair field card is the local models' measured record on this machine. The Qwen hub page carries downloadable serving profiles for these models.