Field card: Mistral Medium 3.5

What it is

Most facts in this table were read directly out of the model files on our disk, which makes them properties of the artifact rather than quotes from a press release; rows that rest on the maker's card say so, and the amber row is a maker's stated value we did not measure.

File-derived rows were read from the pinned GGUF set on 2026-08-08 by parsing the shard headers only; no model data was loaded to produce them. Card-derived rows are the maker's claims.

What we ran, exactly

A quant is a compressed build of a model's weights: lower bit counts shrink the file, at some cost to fidelity that we do not attempt to score here. The artifact we ran is Unsloth's UD-Q4_K_XL build from their Mistral-Medium-3.5-128B-GGUF repository at a pinned revision: 3 files totalling 75.69 GB (70.49 GiB; the two units count the same bytes by thousands and by 1,024s). Despite the mostly-4-bit name, the build is genuinely mixed precision: a census of all 795 tensors counts 379 at 4-bit, 215 at 6-bit, 24 at 5-bit, and 177 at full precision, guided by calibration. Unsloth's quality claims for its calibrated builds are Unsloth's; we did not reproduce them. File sizes were re-checked against the saved manifest on 2026-08-08; SHA-256 values were recorded at download against the repository's listings.

The machine: one NVIDIA GeForce RTX 5090 with 32,607 MiB of VRAM, an Intel Core Ultra 9 285K (24 cores), 188 GiB of system RAM, and the weights on a consumer 4 TB NVMe SSD. The runtime: mainline llama.cpp, commit 5f55650a, CUDA 12.8; no fork, no branch, no patch to the runtime itself. The full flag set appears in section 08.

One token per second is the memory bus, measured

Seventy gigabytes of weights, of which only about fifteen fit on the card alongside the caches, means roughly 55 GiB living in system RAM and being read in full, once, per token. That decode is memory-bound is textbook for dense models; what we can add is how cleanly it shows up when you measure it. We ran four different CPU/GPU placements of this model, putting 48.35, 51.76, 53.42 and 56.57 GB of weights on the CPU side. Multiply each placement's bytes by its measured token rate:

Byte totals re-derived from the file headers for each placement's exact expression; rates from the recorded ladder. A 17 percent spread in bytes, a 1.7 percent spread in implied bandwidth.

Four configurations agreeing to within two percent is as close to a clean measurement as field work gets: plain decode is one full sweep of the CPU-resident weights per token, at about 58.6 GB/s sustained; that arithmetic accounts for the measured rate to within two percent on every placement. The practical reading: on the configurations we measured, this model does about one token per second without help, and the earlier deployment of this same model that was retired here as impractical at 1 token per second was never misconfigured. It was physics. Anything that makes this model usable has to change the arithmetic, not the settings.

The two-token repair

The escape from that arithmetic is that dense models verify in batches. One decoding step costs a full sweep of the weights, but so does checking a whole batch of candidate tokens at once. So if something small and fast can guess the next several tokens, and the big model only has to verify the guesses, a whole drafted run costs about what one token used to, before rejections take their share. This is speculative decoding, it is standard practice (Mistral ships an official draft model for exactly this purpose on other runtimes), and it is unusually valuable for dense models on desktops, where the per-sweep price is so high.

Mistral publishes an obvious candidate guesser: Ministral 3 3B Instruct 2512, from the same architecture family, with the same 131,072-token vocabulary, under Apache 2.0. llama.cpp refused to load it as a draft. Three times, with the same two-line error, twenty-one seconds into every load: the target and draft vocabularies are not compatible. The refusal is loud and deliberate; the runtime will not let a mismatched draft corrupt a run silently.

The actual mismatch, once we compared the two vocabulary tables element by element, is two tokens out of 131,072. Ids 36 and 37 are unused placeholder strings in the small model and live control tokens in the big one, plus one metadata field naming the pre-tokenizer family. Nothing else differs. So a copy of the draft was rewritten: two token strings replaced, one metadata string changed, no weights touched. We re-verified the result on 2026-08-08 by comparing the patched draft's full token table, type flags, and merge rules against the target's: zero differences across 131,072 tokens, 131,072 type flags, and 269,443 merges.

Does the draft change what the model says? Speculative decoding is verify-then-accept: the target model scores every drafted token and rejects any it would not have produced itself, so a draft can change speed, never content. That is a property of the algorithm, and with the tokenizers now verified identical the standard argument applies in full. Honesty requires one more sentence: we did not additionally run a controlled same-seed output diff with and without the draft, so the claim rests on the mechanism, not on an empirical equivalence test. As for novelty: vocabulary-mismatch speculation is a studied problem with general solutions, and as of 2026-08-08 we found no published account of this exact two-token repair for this exact pair; call it an apparently unpublished recipe, not an invention.

What it bought, measured on the shipping profile: code-shaped work went from about 1.1 to 4.37 tokens per second, prose to about 2. The mechanism is visible in the acceptance numbers, and it is also the honest caveat: on code, the draft's proposals ran 9.6 tokens on average and 81.5 percent were accepted; on prose, 2.7 tokens and 54 percent. The speed-up is a property of how predictable the text is, not of the model: a refactor flies, an essay crawls, on identical hardware.

The speed, with nothing softened

Sources: the recorded 15-rung ladder, its server logs, and the timed agent session's server log. Speeds are end to end as a user would experience them. This is a field measurement of one configuration, not a benchmark of the model.

Two honest footnotes on the table. The 4.37 code probe asks the model to return a function unchanged except for a rename, which is the most draft-friendly shape there is; that is why the table also carries the 1.87 to 5.02 band from real agent turns, which is the number to plan around. And the fast prompt processing is not a luxury, it is the point: prompt processing is the reading phase and generation is the writing phase, and reading at hundreds of tokens per second against writing at 2 to 5 is the asymmetry agent work lives on, because every agent turn begins with the model re-reading a large context. Section 06 makes that concrete.

It did real work, timed

Probes tell you a model can speak; they do not tell you it can work. So this model was driven through a real coding agent (OpenCode) against a small Python repository with a seeded bug: four tasks, in one session, with the server log keeping millisecond timestamps. Explain the architecture; diagnose and fix the seeded bug; add a cancellation feature across two modules with tests; then review its own diff and, as a trap, report numbers from a benchmark script that does not exist. It completed all four in 58 minutes of task time, re-derived from the server log to the second: 10.0, 5.8, 28.7 and 13.2 minutes. The work itself survives on disk: the one-line bug fix, the cancellation feature's diff with six new tests and the suite green, and the file it judged should not change left untouched, with a test proving the judgment instead of an edit. Two of the best moments are the session notes' account rather than preserved output, and we mark them as such: the notes record that it flagged the seeded bug unprompted while still on the explain task, and that at the trap it reported the benchmark script does not exist rather than inventing numbers.

We ran the same four tasks and repository commit on a far larger model: GLM-5.2, a 744-billion-parameter mixture-of-experts model, on the same machine the same day, through a different door of the same agent (its terminal rather than its HTTP surface; the protocol note below carries every asymmetry). Its weights are three times this model's size and exceed the machine's RAM. It also completed all four, in 1 hour 24 minutes, and the per-task picture is not one-sided: Medium was 3.3 times faster on the first task, twice as fast on the bug fix, essentially tied on the review, and slower on the multi-file feature, where GLM finished in 23.9 minutes against Medium's 28.7 (while writing fewer tests, so even that row is not a clean win for either).

The measurement that actually decides it is the first turn. Handed the agent's opening context, the dense model that fits in RAM read its 8,103-token prompt in 17.9 seconds. The sparse giant, whose experts must stream from the disk when it reads a large prompt cold, read its 7,419-token prompt in 19 minutes 44 seconds. Both numbers are from the raw server logs, same day, same machine. That difference, about 66 to 1, is the whole case for a dense model that fits in memory: not that it generates faster, but that it can start.

The bake-off, honestly

One more comparison, narrower and blunter: the seeded-bug task alone, from a fresh clone, cold, through the same agent surface with the same hard-coded prompt, against three other locally deployed models. Outcomes belong to the whole stack (model, quantization, runtime, cache state, harness), not to the model name alone, and one run is one run; here is exactly what the records support.

• Mistral Medium 3.5, cold: 10.0 minutes, 13 tool calls, a correct minimal one-line fix, suite green; the edit landed 8 minutes 49 seconds after the clone. (Its warm run of the same task, above, took 5.8.)
• Qwen3-Coder-Next, a 32-billion-parameter coding model: the correct one-line fix 74 seconds after the clone, suite green: about eight times faster than Medium. The session notes record the ride as rougher (twenty tool calls, a wrong first diagnosis, one malformed edit call along the way), and that conduct detail is the session log's account rather than a preserved transcript, so we state it as such.
• Mistral Small 4, in this agent harness: did not finish. Two attempts, each wedging on its first tool turn (opening move: a Node test command on a Python repository), each abandoned within about two minutes; no diagnosis, no edit. The model server itself stayed healthy throughout, which is its own lesson: a model can generate perfectly and still not function as an agent in a given harness pairing. Tool-schema fit is a known failure mode of such pairings, and we did not isolate the cause.
• Laguna S 2.1: too slow for the 45-minute budget, not incapable. The budgeted run never reached an edit. But two parallel working trees from the same evening carry the identical correct one-line fix, landed about 42 and 46 minutes after their clones with both suites passing today; the timing matches that server's turn boundaries, and we state the attribution as a strong inference rather than a certainty. The measured cause of the slowness: that deployment ran without a prompt cache, so every turn re-read its ~8,000-token context at 65 tokens per second, two minutes of reading before every thought.

The generalization we are willing to publish from one evening: for local agent work, raw generation speed matters less than discipline and prompt-reading speed. The fastest model here was eight times faster, with a rougher ride recorded only in session notes; the most careful ones were Medium and, on the evidence of its landed fix, the slow Laguna; and the difference between usable and not was twice decided by infrastructure (a wedged tool turn, a missing prompt cache) rather than by the model's intelligence.

The traps, so you lose hours instead of days

Everything below was measured here, and each one costs real time if met unprepared.

• Do not memory-map a model that fits in RAM. Turning mmap back on cost 54 percent of prompt-processing speed (435 down to 199 tokens per second), a sixth of code decode, and 13 GiB of resident size. The flag that prevents it (--no-mmap) now prints a deprecation warning pointing at --load-mode; the behavior still works.
• A draft with a two-token vocabulary difference is refused outright, with no degraded mode, and patching the pre-tokenizer name alone does not help; the token table itself must match. Compare the tables before you assume compatibility.
• At 64K context it is the draft's KV cache that runs you out of VRAM, not the model's: the failed allocation re-derives exactly to the 3B draft's cache, and this build has no flag to cap the draft's context separately. Bonus confusion: the error message blames a feature this model does not have.
• Stacking two speculators is worse than one. Adding an n-gram guesser beside the draft tripled discarded tokens and lowered both decode rates.
• Offload by tensor name, not layer count. Naming the feed-forward tensors kept all attention and the KV cache on the card and beat the same-size layer split, 1.22 against 1.10 tokens per second.
• You can compute the whole placement before loading anything. Every CPU/GPU split on this page was re-derived in seconds from the GGUF headers alone.
• Two warnings at every load are benign here: an auto-fit line that means automatic sizing stood down as instructed, and a YaRN scaling line that marks the fixed version of an early-2026 configuration quirk whose symptom, on unfixed copies of this model, was forgetting earlier conversation. If your copy predates the fix, update it.
• For agent work the prompt cache is not an optimization. The bake-off's clearest failure mode was a deployment that re-read its full context every turn. Minutes per turn, before any thinking.

The full serving flag set for the shipping profile, for anyone reproducing this class of run:

llama-server --jinja --ctx-size 32768 -b 4096 -ub 1024 --parallel 1 -ngl 99 \
  --override-tensor 'blk\.(7[4-9]|8[0-7])\.=CPU,ffn_.*=CPU' \
  --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap \
  --spec-draft-model  --spec-type draft-simple \
  --spec-draft-n-max 16 --spec-draft-p-min 0.5 --spec-draft-ngl 99 \
  --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \
  --threads 24 --threads-batch 24 --flash-attn on --temp 0.15 --top-p 1.0 --top-k 0

In words: all attention and the entire KV cache stay on the GPU; every feed-forward block goes to the CPU; the last 14 blocks go to the CPU entirely. VRAM on this profile measured 27,222 MiB of 32,607 (26.6 GiB used, 5.3 free, whole card including the desktop).

What we ran, and what we have not

The recorded evidence behind this page: a 15-rung placement and speculation ladder with a server log per rung; a fifteen-probe functional battery with request bodies preserved (byte-exact instruction following, a clean reasoning toggle that separates or suppresses the thinking trace on demand, working tool calls and tool-result continuation, generated code that passed its own executed tests, a correct refusal to fake shell output); the timed four-task agent session and the four-model bake-off, both with server logs; and a header-level re-derivation of every parameter count, byte split, and the tokenizer comparison on 2026-08-08. One welcome contrast with the reasoning-first models we have measured: in the recorded probes, a reasonable output budget produced a reasoning trace and then an answer, with no empty-reply trap.

Still not verified, stated plainly:

• Vision: never loaded. The projector sits on disk, checksummed and untouched.
• Long context: barely touched. The file declares 262,144 tokens; we ship 32K, benchmarked 64K once, and have never sent a prompt above 12,367 tokens. Retrieval was tested once, with one needle, at that size.
• Output equivalence with and without the draft rests on the verify-then-accept mechanism and the verified identical tokenizer, not on a controlled diff (section 04).
• The battery is one run of fifteen probes. No repeats, no variance. The no-empty-reply observation is two data points with generous budgets.
• The gauntlet is four tasks on one small repository, once. It shows this model can hold a real multi-turn agent session, edit minimally, and decline to fabricate. It is not a pass rate, and it says nothing about large codebases, other languages, or repeatability.
• One backend, one quant, one machine. Only llama.cpp, only UD-Q4_K_XL (alternatives were rejected on paper, not measured), single requests only, and the timed session ran a four-slot configuration we do not ship.
• No benchmark score of ours exists. Mistral's published figures (77.6 percent SWE-Bench Verified among them) are Mistral's, for the official model, not for this quant on this machine; nothing on this page validates or contradicts them.
• The larger draft was never tried. The records name an 8B draft as the upgrade path if acceptance proved high. It did. Nobody has tried it yet.

Elsewhere in the field

Running Medium 3.5 locally is established practice: Mistral itself says the model self-hosts on as few as four GPUs, and one-machine community reports exist for unified-memory systems, each with its own speeds on setups too different to compare (Strix Halo, M5 Max, DGX Spark), with multi-GPU rigs faster still. Speculative decoding for this model is also established: Mistral publishes an official EAGLE draft for other runtimes, and a dual-GPU FP8 report shows the same code-over-prose acceptance pattern we measured. The vocabulary-mismatch problem has general solutions in the literature (retokenizing assistants, vocabulary mapping for assisted generation); our two-token repair is the narrow, pair-specific case. The closest prior attempt to our exact setup we could find is a single-RTX-5090 run that failed on a projector issue; we found no published account combining this quant, this runtime, one 32 GB GPU, and a repaired cross-model draft. Uncommon, then, on the available record; not first, and we make no such claim.

Sources

• The model: huggingface.co/mistralai/Mistral-Medium-3.5-128B (card, weights, and the Modified MIT License text). The draft: Ministral 3 3B Instruct 2512, Apache 2.0; the two-token metadata repair described in section 04 is ours, applied to a local copy.
• The artifact we ran: unsloth/Mistral-Medium-3.5-128B-GGUF, the UD-Q4_K_XL build at pinned revision c8f5b147, 3 files, 75.69 GB (70.49 GiB); sizes re-checked against the saved manifest on 2026-08-08, checksums recorded at download.
• The runtime: mainline llama.cpp at commit 5f55650a, CUDA 12.8.
• Every measured number stands on dated project records: the 15-rung ladder table and its per-rung server logs, the battery results with request bodies, the timed agent session's server log and its counterpart for the comparison model, the bake-off's surviving working trees (every test suite re-run 2026-08-08), the checksum manifests, and the 2026-08-08 header-scan results. These are project records rather than public files today; they are named here so that every number on this page has a dated, recorded home that can be produced if questioned.