{"id":635,"date":"2026-08-31T13:01:29","date_gmt":"2026-08-31T13:01:29","guid":{"rendered":"https:\/\/abrarqasim.com\/blog\/llm-quantization-4-bit-beat-its-own-16-bit-source\/"},"modified":"2026-08-31T13:01:29","modified_gmt":"2026-08-31T13:01:29","slug":"llm-quantization-4-bit-beat-its-own-16-bit-source","status":"publish","type":"post","link":"https:\/\/abrarqasim.com\/blog\/llm-quantization-4-bit-beat-its-own-16-bit-source\/","title":{"rendered":"LLM Quantization Flipped: A 4-Bit Model Beat Its 16-Bit Source"},"content":{"rendered":"

I keep a mental asterisk next to every quantized model I run: this is the dumber version<\/em>. Download the GGUF, load the Q4 file, accept that you traded some benchmark points for the privilege of running the thing on hardware you actually own. I’ve repeated that trade-off to clients. I’ve repeated it on this blog. And I believed it right up until last week, when Multiverse Computing published a recipe<\/a> that produced a 4-bit model that beats the 16-bit checkpoint it was quantized from. Not matches. Beats. On 7 of 9 benchmarks.<\/p>\n

Short version for the impatient: the trick is called quantization-aware healing (QAH), and the core idea is that you distill the quantized model against the original, uncompressed<\/em> teacher instead of the shrunken checkpoint it was cut from. If you want to know why that one change flips the usual result, read on.<\/p>\n

The assumption baked into every GGUF download<\/h2>\n

The standard pipeline for making a big model deployable has two stages. First you compress the architecture itself: drop layers, prune heads, remove neurons, so a 120B model becomes a 60B model. Then you quantize whatever weights survive down to 4 bits so the memory footprint shrinks again. Each step saves real money. Each step also chips away at the capabilities people actually pay for, and the damage lands hardest on reasoning, math, and code generation.<\/p>\n

Because of that, serious pipelines add a recovery step before shipping, usually called healing. OpenAI’s gpt-oss models<\/a>, NVIDIA’s Nemotron family, and Multiverse’s own Hypernova line all go through some version of compress-then-heal. What nobody had pinned down was how well the healing step works when a model has been through both<\/em> structural compression and quantization, and what the right way to do it even is.<\/p>\n

My working model of LLM quantization came from watching my own downloads get worse as the bit count dropped. Q8 felt close to the original. Q4 was usable but noticeably sloppier on hard prompts. Anything below that was a party trick. I wrote about this when I was running Qwen locally and fighting its reasoning settings<\/a>, and my advice at the time boiled down to: quantization is a tax, budget for it.<\/p>\n

Turns out the tax was partly a billing error.<\/p>\n

What quantization-aware healing actually does<\/h2>\n

To see why QAH works, you need the two recipes it replaces.<\/p>\n

Quantization-aware training (QAT) is the incumbent. You insert fake-quantization ops into the forward pass and keep fine-tuning on a task loss, so the weights learn to live with low precision. The catch is that you’re re-running an expensive post-training process (SFT, RLHF, agentic tuning) through a noisier forward pass. It costs a lot, and as we’ll get to, it can fall apart if you train past its sweet spot.<\/p>\n

Quantization-aware distillation (QAD) skips the retraining and instead distills a frozen full-precision teacher into the quantized student via KL divergence on the output logits. That works fine when quantization is the only<\/em> change, because a true full-precision twin of the same architecture exists to teach from. But once the model has been structurally compressed, there is no honest teacher left. The only full-precision candidate is the recovered bfloat16 checkpoint, which is itself a degraded approximation of the original. Distill from that and you’ve anchored your student to a ceiling below where it could go.<\/p>\n

QAH makes one change: the teacher is the original, pre-compression, full-size model. Teacher and student don’t even share an architecture. One is 120B in full precision; the other is 60B running in MXFP4. That mismatch doesn’t matter, because a teacher’s output distribution is architecture-agnostic. The student never sees hard labels at all, just the teacher’s logits through a KL loss.<\/p>\n

The reframing here is what got me. Under QAH, quantization stops being a lossy post-processing step and becomes a second full round of distillation against the original teacher. The 4-bit student picks up information the earlier recovery stage never had time to transfer, supervision the bfloat16 checkpoint simply never received.<\/p>\n

The numbers, including where it lost<\/h2>\n

The headline experiment took GPT-OSS 120B, compressed it to 60B, recovered it in bfloat16, then re-quantized to MXFP4 under QAH. The fair comparison is against that same 60B model’s bfloat16 checkpoint, the best full-precision version of the architecture that exists. The 4-bit model wins 7 of 9 benchmarks.<\/p>\n

The gains cluster exactly where compression hurts most: +7.4 points on AA-LCR (long-context reasoning) and +5.6 on AIME 2025 math. Agentic coding, tool use, science QA, and instruction following all improved by one to three points. The two losses, MMLU-Pro and SciCode, were each under a point and a half.<\/p>\n

The comparison against the original 120B teacher surprised me more. At half the parameter count and roughly a quarter of the weight memory, the QAH model beat the full-size teacher<\/em> on LiveCodeBench, 66.5 to 66.0, and came within 1.6 points on GPQA Diamond. The one place the teacher stays comfortably ahead is extreme long context, where capacity lost to compression is hardest to claw back. That feels honest to me. You can’t distill your way around having half the parameters when the task is holding 32k tokens of thread in your head.<\/p>\n

All the tables are in the paper<\/a> if you want to check my summary against the source, which you should.<\/p>\n

Why QAT keeps burning people<\/h2>\n

The part of the paper with the most practical bite is a head-to-head between QAH and QAT on a smaller 9B model, tracking average benchmark performance as training progresses. Peak accuracy is basically a tie: 54.9 for QAH, 54.6 for QAT. If you stopped reading at the peaks you’d call it a wash.<\/p>\n

The trajectories are the story. QAH hits its peak in about 100 steps, roughly 7x faster than QAT’s 700, then sits within about two points of that peak for the rest of the run. QAT climbs slowly, peaks, and then collapses, shedding nearly 19 points by step 1,200.<\/p>\n

The mechanism is intuitive once stated. KL distillation ties the student to a fixed teacher distribution, so once the student catches up, there’s no pressure left to drift. A cross-entropy task loss never stops pushing toward hard labels, and eventually it grinds away capabilities the model inherited. In deployment terms: a QAT checkpoint needs careful early stopping against a held-out signal or you risk shipping a model that already started rotting, while a sufficiently trained QAH checkpoint just parks.<\/p>\n

I’ve been on the wrong side of this pattern with ordinary fine-tuning, watching eval numbers sag while the loss curve kept politely descending. I didn’t have a name for the fix. Now I do.<\/p>\n

What this changes if you run models locally<\/h2>\n

Some honesty about scope first: you can’t QAH the GGUF files on your disk. This is a training-time recipe that needs the original teacher’s logits and real GPU budget. Nothing changes about your local setup today.<\/p>\n

What changes is what vendors can ship, and what you should demand. The efficiency math is blunt: 4-bit weights take roughly a quarter of the memory of bfloat16, and half the parameters roughly halves compute per token. Multiverse used this exact recipe for Hypernova-60B<\/a>, released open-weight under Apache 2.0. If healed 4-bit checkpoints become the normal release artifact, the standing advice to grab the biggest quant your VRAM tolerates because bigger is closer to the real model<\/em> quietly dies. The healed 4-bit release might be the best version of that architecture, full stop.<\/p>\n

It also shifts the inference cost conversation I keep having with clients. I argued in my post on LLM cost optimization<\/a> that most teams overpay for capability they never use. QAH sharpens that: if a healed 4-bit model at half the size matches a flagship on your actual workload, the price gap between them is pure margin for someone who isn’t you. This is the kind of sizing question I work through in client engagements<\/a> all the time, and my default answer just moved a notch smaller.<\/p>\n

One thing to do this week<\/h2>\n

Stop trusting the folk hierarchy of quant levels and measure your own case. Take the task you actually run, collect 30 or 40 real prompts, and score a healed or well-quantized 4-bit model against the full-precision version you assume is better. Use your own rubric, not a leaderboard. If Hypernova-60B fits your hardware, it’s a free Apache 2.0 test subject. The worst outcome is you confirm the tax is real for your workload and you keep paying it with better conscience. The best outcome is you find out you’ve been paying for bits that weren’t helping, and that one’s worth an afternoon.<\/p>\n","protected":false},"excerpt":{"rendered":"

A 4-bit model just beat the 16-bit checkpoint it was quantized from. I break down quantization-aware healing, the numbers, and what it changes for local LLMs.<\/p>\n","protected":false},"author":2,"featured_media":634,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"","rank_math_description":"A 4-bit model just beat the 16-bit checkpoint it was quantized from. I break down quantization-aware healing, the numbers, and what it changes for local LLMs.","rank_math_focus_keyword":"llm quantization","rank_math_canonical_url":"","rank_math_robots":"","footnotes":""},"categories":[4,217],"tags":[692,693,5,313,691],"class_list":["post-635","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-llm","tag-distillation","tag-gpt-oss","tag-llm","tag-local-llm-2","tag-quantization"],"_links":{"self":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/635","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/comments?post=635"}],"version-history":[{"count":0,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/635\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media\/634"}],"wp:attachment":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media?parent=635"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/categories?post=635"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/tags?post=635"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}