The wrong flag

Summary

On 2026-08-05 this machine's records began carrying a claim about Mistral Large 3 675B (Unsloth's UD-IQ1_S quantization, experts CPU-offloaded behind one RTX 5090): warm decode is about 6 tokens per second and llama.cpp's warmup pass “is what unlocks” it; with --no-warmup the same model reads 3.6. Four separate records repeated the claim. On 2026-08-11 a paper audit graded the attribution confounded: the two measurements behind it differed not only in warmup but in thread count and server slot layout. The same night, a supervised three-arm A/B on the canonical serving script settled it.

Three results:

1. Warmup is innocent of the speed claim. Adding --no-warmup to the canonical configuration and changing nothing else left warm decode unchanged: 5.82 / 6.96 / 7.09 tokens per second across the three warm probes, against 6.83 / 7.07 / 6.96 with warmup on.
2. The thread default reproduces the slow number. Removing the --threads 24 --threads-batch 24 line and letting llama.cpp pick its own thread count read 3.56 / 3.57 tokens per second warm, against 3.58–3.61 in the original 2026-08-05 record. The build's heuristic resolves to 4 compute threads on this 24-core CPU (a process-thread cross-check: 33 threads versus 53 with the pin present, a difference of 20, which is 24 minus 4). That makes the thread default the supported explanation for the recorded 3.6-versus-6 gap in this tested configuration; it does not quantify every contributor to the historical comparison, which also differed in slot count (section 05).
3. What warmup did in this run is move pain, not remove it. With warmup on, load took 6 m 51 s and the first answer still took 897 s on a cold page cache; with warmup off, load took 2 m 06 s and the fault cost reappeared in the first generation (970 s). The 177.7 GiB file plus process memory exceeds 188 GiB of RAM, and the observations are consistent with warmup partially evicting itself; the hot expert set settles only through use. Keeping the server resident between chats remains the real fix.

Also corrected while we were in there: the true 1-slot service VRAM is 24,420 MiB, which is 23.85 GiB used, with 8,187 MiB, which is 8.00 GiB, free, and every earlier figure recorded as “GiB” was a raw MiB reading divided by 1000. Section 08 owns both mistakes in detail.

Environment

Everything below is a property of one model file, one runtime build, one placement, and one probe shape, measured 2026-08-11 (with the prior record's runs dated 2026-08-04/05), on one workstation. Nothing here measures the model's intelligence or benchmark standing, and no such claim appears on this page.

The record before the test

The claim under test was born as a two-point comparison. On 2026-08-04/05 a tuning ladder established the serving profile; every ladder rung ran with --no-warmup baked into its base arguments and no --threads flag, and the winning rung decoded at 3.58–3.61 tokens per second. The same night, a smoke test of the canonical serving script, which has warmup on and pins --threads 24 --threads-batch 24, read 5.98–6.28. Two real measurements, three variables apart:

The records kept the numbers and picked one variable: “warmup pre-faults every expert and is load-bearing for speed.” That sentence propagated into four documents. The 2026-08-11 audit graded both speeds measured and the attribution confounded, and specified the experiment below, verdict logic first.

The experiment

Three arms, all scratch copies of the canonical serving script differing only in declared flags (diffs and SHA-256 hashes preserved in the data package), run one at a time the same night, each stopped and its VRAM release verified before the next: (a) the canonical configuration as-is, warmup on, threads pinned to 24; (b) as-is plus --no-warmup, nothing else changed; (c) as-is plus --no-warmup with the threads line removed, so the build's own default decides. There is no fourth arm (warmup on, threads unset): the thread reading below rests on arm (b) clearing warmup plus arm (c) reproducing the slow number, not on a single-variable contrast against arm (a).

The verdict logic was written down before the first arm loaded. If (b) collapses to about 3.6 while (a) reads about 6, warmup is the cause and the record stands. If (b) stays about 6 and (c) collapses to about 3.6, the finding is real but it is a threads finding wearing warmup's name. Anything else: report what is seen, no forced story. Pre-declared bands: “about 3.6” means 3.2–4.2, “about 6” means 5.5–6.7.

Honesty about size: this is a three-arm experiment with four probes per arm, of which three are warm. The observed stable readings show an approximately twofold separation under this run's conditions; with three warm probes per arm and no intervals, the experiment is not designed to estimate uncertainty or smaller effects. There are no confidence intervals here; there are individual probe readings, all published. Identical repeat prompts hit slot prefix reuse, so repeat-probe prefill numbers are not meaningful and decode is the metric; that too was pre-declared.

Observed: removing warmup did not move warm decode; removing the thread pin did

Two of the twelve probes are transient dips, one each in arms (b) and (c): arm (b)'s probe 2 read 5.82 with a mid-run sag, and arm (c)'s probe 2 read 1.70; in both cases the flanking probes agree with each other and with the wall clocks (200 tokens in ~28.5 s is 7.0 t/s; in ~56.3 s is 3.56 t/s). We publish the dips rather than smooth them. First-generation numbers are cache-history-dependent and are not an arm contrast (sections 06 and 11).

Against the pre-registered logic this is branch 2, read directly: removing warmup alone did not move warm decode (5.82–7.09 against 6.83–7.07), and removing the thread pin reproduced the ladder's slow band to within 0.05 tokens per second (3.56–3.57 tonight against 3.58–3.61 on 2026-08-05). On slots: the thread change alone was sufficient to reproduce the historical 3.6 range at one slot, but no arm varied --parallel, so the 4-versus-1-slot contribution to the historical comparison remains unmeasured (section 11). One pre-registration footnote for the record: two warm probes (7.07, 7.09) landed slightly above the declared “about 6” band's top edge of 6.7; the branch verdict does not depend on that edge, because the branch question is whether (b) collapsed into the 3.2–4.2 band, and it did not.

The thread-count witness: the arm (c) server process ran 33 threads where arms (a) and (b) ran 53. The difference is 20, which is 24 minus 4: the pinned 24-thread compute pool replaced by a 4-thread default, with the server's non-compute threads unchanged.

What warmup did in this run

Warmup is not fake. It is a real llama.cpp behavior doing a real thing: it touches the weights at load time so the first generation does not have to page them in. On this machine's memory budget, this run suggests that repositioning is the most it can deliver, and sometimes not even that:

• With warmup on (arm a, cold page cache): load took 6 m 50.9 s, and the first answer still took 897 s of wall clock. During that first generation the server sat in uninterruptible sleep major-faulting expert pages at roughly 67 MB/s, and cumulative reads passed 237 GB against a 190.80 GB file: cache thrash, not a single pass. The file plus the process's anonymous memory plus the desktop exceeds the machine's RAM, a budget consistent with warmup's own tail evicting its head.
• With warmup off (arm b, warm-ish cache): load took 2 m 06.1 s and the fault cost reappeared where warmup would have absorbed it, in the first generation (969.8 s).
• By the third load of the night (arm c) the first answer was fine: 130.5 s wall, prefill at 8.35 t/s, consistent with the hot expert set having settled in the page cache.

In this run, warmup-on cold loading took longer, and the no-warmup arm's first generation was slow instead. The process-state observations are consistent with warmup shifting some of the faulting from the first answer into load time; cache history was not controlled across arms, so this experiment does not isolate that effect. What the run does show is that warmup did not set warm decode speed on this machine. The durable operational advice survives the correction: keep the server resident between chats, because every fresh start risks a multi-minute first answer while roughly 154 GiB of experts re-settle, warmup or not.

The heuristic, in the code we ran

llama.cpp resolves --threads -1 (the default) through common_cpu_get_num_math(). On x86-64 Linux with a hybrid CPU, that walks every logical CPU, pins itself to each in turn, and asks the CPU what it is running on. Two rules, quoted from the build we ran (common/common.cpp, cpu_count_math_cpus): efficiency cores are skipped with the comment “efficiency cores harm lockstep threading”, and the counter additionally advances two logical CPUs at a time with the comment “hyperthreading isn't useful for linear algebra”.

On this CPU the two rules compound: the 16 E-cores are excluded, and the skip-by-two halves the remaining 8 P-cores even though the 285K has no hyperthreading. (Logical CPUs are the operating system's enumeration; on a chip without hyperthreading every logical CPU is a physical core, which is exactly why the skip is costly here.) Result: 4 compute threads on a 24-core machine, confirmed at runtime by the process-thread delta in section 05. And the resolved value is invisible where you would first look: llama-server --help prints (default: -1), not the number the heuristic will pick on your machine (verified on this build).

Read as a rule, from this build's code: on hybrid Intel under x86-64 Linux, expect the default to land near your P-core count if the chip has hyperthreading, and near half your P-core count if it does not. On a non-hybrid x86-64 CPU, AMD included, the hybrid path never runs at all: common_cpu_get_num_math() falls back to the physical core count, so SMT siblings are excluded but nothing is halved. Both statements are read from this build's source, not measured on other machines; treat them as what to expect, and verify with the checks in section 10.

Provenance, for fairness: this heuristic is not an accident. It arrived in llama.cpp in April 2024 as part of Justine Tunney's llamafile performance work (PR #6414, “ggml : add llamafile sgemm”, merged 2024-04-16), and it encodes a finding that was real and public in 2023: on hybrid Intel CPUs running dense models fully on CPU, threads scheduled onto E-cores dragged the whole lockstep pool down, and pinning to P-cores was measured up to 3x faster (section 09). The heuristic is a reasonable answer to that problem. This page's regime is a different workload: decode on a CPU-offloaded 675B MoE, which on this machine wants more threads than the heuristic provides. Whether the optimum here is 8, 16, or 24 threads is unmeasured; what is measured is that 24 roughly doubles the heuristic's 4.

What we got wrong

This site publishes corrections of our own records as a genre, and this page is one. The instrument was right and the prose was wrong, in two ways.

The misattribution

From 2026-08-05 to 2026-08-11, four records on this machine (the model's spec card, its serving-knobs file, the model reference, and the machine's state ledger) said warmup “is what unlocks ~6 t/s”. The numbers those records carried were all real, and all reproduced this week. The causal sentence connecting them was a two-point comparison across three uncontrolled variables, and it survived six days because it was plausible: warmup genuinely does pre-fault experts, the smoke test's first generation genuinely was fast, and the mechanism story wrote itself. Nobody ran the third arm until the audit forced it. The corrected records now carry the thread attribution, and the old entries are annotated as superseded rather than rewritten.

The unit slip

Every VRAM figure in those records written as “GiB” was a raw nvidia-smi MiB reading divided by 1000: the recorded “24.6 GiB” default profile is 24,627 MiB, which is 24.05 GiB; “27.6” is 27,624 MiB, which is 26.98 GiB; “15.1” is 15,082 MiB, which is 14.73 GiB. The same slip family had already been caught on three other models' records on this machine, which is exactly why the audit checked. The corrected service figure, measured this week on the 1-slot canonical configuration: 24,420 MiB, which is 23.85 GiB used, and 8,187 MiB, which is 8.00 GiB free (a desktop baseline of roughly 830–900 MiB is inside the reading). The operational ceiling this machine runs under (at least 6 GiB of VRAM kept free) holds with more margin than the old records claimed, so the slip was conservative in effect here; it does not always have to be.

Prior art: the E-core effect is old news, and we claim none of it

The interaction between llama.cpp threading and hybrid Intel cores has been public since early 2023, and the record we could retrieve is consistent:

• llama.cpp discussion #572 (2023-03-28, user delock, i7-12700H): restricting to performance cores only ran 2.4x–3x faster than using all cores on 7B and 65B dense models; an early public observation, the earliest we located.
• llama.cpp issue #842 (2023-04-08, 13900K): only ~50% CPU utilization at high thread counts, E-cores saturated while P-cores idle.
• Justine Tunney, “LLaMA Now Goes Faster on CPUs” (justine.lol/matmul, 2024-03-31) and llama.cpp PR #6414 (merged 2024-04-16): the E-core-excluding default heuristic itself, upstreamed from llamafile, with its rationale in the code comments quoted in section 07.
• Downstream surprise reports exist too, for example ramalama issue #934 (2025-03-10), a user asking why CPU inference uses “only 4 cores”; the low invisible default surprises people on non-Intel hardware as well.
• Current CPU-MoE offload guides we checked (for example the widely shared January 2026 MoE offload guide on Hugging Face) pass an explicit -t in their example commands but do not discuss the default heuristic or hybrid-CPU thread selection at all.

What this page adds is scoped and small: a controlled measurement of the default-versus-pinned contrast on a regime not represented in the sources we reviewed above (a 675-billion-parameter CPU-offloaded MoE, where decode is CPU-side and memory-bound), the close reproduction (3.56–3.57 against 3.58–3.61) tying a previously misattributed two-point comparison to the thread default, and the misattribution case study itself. We claim no firstness on the E-core effect, the heuristic's behavior, or thread tuning in general. Note the direction, too: the 2023 findings were “all cores can be slower than P-cores only” on dense CPU-only models; our finding is “4 threads is roughly half the speed of 24” on a CPU-offloaded MoE. Both can be true; they are different workloads, and we did not measure a P-cores-only arm (section 11).

If you run a big CPU-offloaded MoE

Four narrow things, each scoped to a stated regime; none is a universal number:

1. See your effective thread count; do not assume you know it. llama-server --help prints (default: -1) and will not tell you what the heuristic resolves to, and on this build's default logging the resolved count did not appear in the server log either (grep yours for n_threads; if your build prints it, that is the cheapest check). Two routes that need no log: watch per-thread CPU during a decode (top -H -p <pid>, or htop with threads shown); the threads that go busy while tokens stream are the compute pool. Or read Threads: from /proc/<pid>/status (or ps -o nlwp <pid>) and compare against a run with --threads pinned: on this machine the pinned-24 server ran 53 threads and the default server ran 33, a difference of 20, exactly 24 minus 4. The raw count alone proves nothing, because it includes the server's non-compute threads; the comparison, or the busy-thread view, is the witness.
2. Know the heuristic's shape. On hybrid Intel, the default picker excludes E-cores outright and skips alternate logical CPUs even on chips without hyperthreading; expect about your P-core count with hyperthreading, about half of it without (on a 24-core 285K: 4). On non-hybrid x86, AMD included, this build falls back to your physical core count instead: SMT excluded, nothing halved. The code is cpu_count_math_cpus in common/common.cpp, there since April 2024; read it on your build rather than trusting our summary.
3. Pin --threads explicitly and measure; do not copy our 24, which is this machine's number, not advice. On this machine, this model, this placement, --threads 24 --threads-batch 24 roughly doubles warm decode against the default. (--threads governs generation; --threads-batch governs batch and prompt processing. This page measured decode only; we pin both to the same value, and the batch half is unmeasured here.) Whether your optimum includes E-cores, or stops at your P-core count, depends on your memory bandwidth and your model; the 2023 record shows regimes where fewer threads win. A cheap screen first: watch the per-thread view from item 1 during a decode; if the compute threads are not saturated, thread count is unlikely to be your bottleneck, and if they are all pegged while the rest of the machine idles, the sweep is worth the evening. The sweep itself is cheap: same prompt, same server, two thread settings, three warm probes each, compare decode tokens per second.
4. Keep the server resident if your weights approach your RAM. Warmup cannot pin a hot set your RAM cannot hold; on this machine the first answer after a cold start took 897 s with warmup on. Load-then-idle costs you nothing but memory you were already spending.

What this does not show

• One regime, one machine. One model (Mistral Large 3 675B), one quantization (UD-IQ1_S), one placement (--n-cpu-moe 59, ctx 32768), one build (5f55650), one CPU (285K), one prompt shape. Nothing here generalizes to a different placement, and the 2023 record already shows dense CPU-only regimes where the thread answer has the opposite shape. Measure your own lane.
• Four probes per arm, three warm, no intervals. This experiment resolves a factor of two; it would not resolve ten percent. Two of twelve probes were transient dips, published as-is, with wall clocks corroborating the stable readings.
• The thread axis has two measured points, 4 and 24. No sweep. A P-cores-only arm (8 threads) was not run, so this page cannot say whether E-cores help or hurt at the margin here, only that the default's 4 is about half of 24's speed on this regime. There is also no warmup-on, default-threads arm (section 04).
• The slot axis was not varied. All three arms ran --parallel 1; the historical ladder ran 4 slots. The thread change alone reproduced the slow number at one slot, which shows it is sufficient, not that slots contributed nothing to the historical 4-slot comparison. That contribution remains unmeasured.
• The mechanism beyond thread count is uninstrumented. No memory-bandwidth counters, no per-core utilization capture, no accounting of why 24 lands where it lands. The page-fault story in section 06 is read from process state, I/O counters, and wall clocks, which is evidence of faulting, not a full accounting.
• Warm-band compatibility, stated plainly. This week's warm band on the canonical configuration is 5.8–7.1 t/s across arms (a) and (b); the 2026-08-05 smoke recorded 5.98–6.28. The two are compatible, and that is the whole claim: with three warm probes per arm and no intervals, the difference between the bands is not a finding, and no sentence on this page treats it as one. Two probes exceeded the pre-registered “about 6” band's 6.7 top edge (section 05). We treat 5.8–7.1 as the current measured band.
• First-generation numbers are not comparable across arms. Arm (a) ran on a cold page cache, arm (c) on the night's third load; the 897 s / 970 s / 130 s first answers measure cache history as much as configuration, and are reported for the warmup story, not as an arm contrast. Repeat-probe prefill is contaminated by prefix reuse and was pre-declared non-reportable.
• VRAM readings include the desktop. Roughly 830–900 MiB of desktop baseline is inside the 24,420 MiB figure, as it was in every earlier reading.

Artifacts

The complete evidence for this page ships as a data package: the files the runs actually produced, not a summary. If a number on this page disagrees with a file in the package, the file is right and the page is wrong; tell us and we will fix the page. data/README.md maps every file, states its provenance, and states the package's redactions plainly; data/NUMBERS.md maps every figure on this page to its file.

• The pre-registered run log (data/ab_run/RUN_LOG.md): arms, probe protocol, verdict logic and bands, all written before the first arm loaded, with per-arm records appended live.
• The three arm configurations: the llama-server invocation each arm ran (data/ab_run/arm_exec_blocks.txt), the diffs proving each scratch script differs from the canonical serving script only in its declared flags (arm_scripts.diff), and SHA-256 hashes proving the canonical script was never edited (arm_scripts.sha256). The full launcher scripts are project records and are not in the package; the README says so and says why.
• All twelve probe JSONs, the probe harness, and the per-arm summaries (data/ab_run/arm_*_probe*.json, run_probes.sh, arm_*_probes_summary.txt): every decode figure on this page, with llama.cpp's own timings fields.
• Per-arm server logs and nvidia-smi captures (data/ab_run/arm_*_server.log, arm_*_nvidia_smi.txt): load times, slot counts, and the raw-MiB VRAM readings.
• The 2026-08-05 ladder record the old claim was built from (data/ladder/l3_ladder_results.tsv and the ladder_R*.log rung logs; the TSV's VRAM column is raw MiB), plus the canonical smoke log (l3_canonical_smoke.log).
• The claims audit that graded the old record and specified this experiment (data/REPORT_l3_claims_dossier.md), and the findings memo (data/FINDINGS_l3_warmup_ab.md).

Every claim on this page is dated. If today is much later than the rows above, treat build-specific findings as unverified since that date: a default, like the one this page is about, can change.

Related pages on this site

Slower by default is the same editorial thesis on the same workstation: a shipped llama.cpp default that costs a specific regime real speed, measured, with its provenance traced. The Qwen3.5 pair field card is the local models' measured record on this machine, and the Qwen hub page carries downloadable serving profiles.