I Quantized Qwen3.8-27B. “4-Bit” Doesn’t Mean What You Think

FROM CHRIS / ORIGINAL VIDEO DESCRIPTION

I quantized Qwen3.8-27B — and used the process to look at what “4-bit” actually means in practice. Quantization is usually described as if a model simply goes from 16-bit to 8-bit to 4-bit. But real model representations are more complicated than that: different tensors can use different formats, scales, group sizes, packing schemes and precision choices — and two things described as “4-bit” may behave very differently. In this video I use VINDEX3 to inspect, quantize and compare representations of Qwen3.8-27B, looking at what is actually stored, what changes during quantization, and why representation matters more than a single headline bit-width. The broader idea behind VINDEX3 is that model representation should be explicit, inspectable and measurable — rather than hidden behind a format name or quantization preset. We look at: • how Qwen3.8-27B is represented before and after quantization • why “4-bit” is an incomplete description • mixed and per-tensor precision • the difference between storage format and execution representation • measuring quality rather than assuming it from a quantization label • exporting and inspecting VINDEX3 model representations VINDEX3: https://vindex3.org LARQL: https://github.com/chrishayuk/larql This is part of my ongoing work on model representation, inference and making very large language models practical on local hardware.