Quantization, explained

01

What is quantization?

Quantization is a way of compressing a neural network by storing its numbers, chiefly its weights, in fewer bits: 8 or 4 bits each instead of 16 or 32. At 4 bits, a language model’s weights take about a quarter of the memory they need at 16 bits, which is how models with billions of weights fit on a phone.

A language model is mostly a very large set of learned numbers, called weights or parameters. Qwen3 1.7B, the open model from Alibaba that Brello Core is built on, has 1.7 billion of them, and its published weights use bfloat16, a 16-bit number format 9. Writing each new token involves nearly all of those weights, so their size decides how much memory a model needs and, on a phone, much of how fast it runs. Brello is not affiliated with or endorsed by Alibaba or Google.

Quantization replaces each weight with a small integer and keeps a scale factor, so that an approximation of the original value can be rebuilt when the model runs 1. The approximation is the cost: every weight is rounded, and rounding error is what can reduce accuracy. The word, also spelt quantisation, comes from signal processing, where it means mapping a continuous range of values onto a limited set of levels.

This page works through the arithmetic, the common formats and what research says about accuracy, then looks at the three files Brello 1.0 runs. For why phones run small models in the first place, see Small language models, explained.

02

How do bits translate into model size?

Weight memory is the number of weights multiplied by the bits stored for each, divided by eight to give bytes. Halving the bits halves the memory.

Table 1 applies that to a model with 1.7 billion weights, the size of Qwen3 1.7B. Sizes on this site are decimal, so 1 GB is 1,000,000,000 bytes.

The table counts weights only. A running model also needs memory for its key-value cache, which holds the text it is working on and grows with the length of the conversation, for intermediate results and for the runtime itself. Context windows, explained works through the cache.

These numbers decide what fits. Brello recommends Brello Core for phones with 4 GB of RAM; at 16 bits, the weights of its base model alone would take 3.4 GB of that.

Fewer bits also mean less data to move. A model writes one token at a time, and each token means reading the weights from memory again, so generating text a request at a time is usually limited by memory bandwidth rather than by arithmetic 7. litert-community, which publishes the LiteRT-LM files Brello downloads, measured two versions of Qwen3 1.7B on a Samsung Galaxy S26. On its GPU, the 4-bit file wrote 40.5–40.7 tokens per second against 14.9–18.1 for the 8-bit file; on its CPU the two were close, at 8.0–8.3 and 7.7–7.8 10. These are the publisher’s measurements on one phone, not Brello’s, and they show that the gain depends on the hardware as well as the format.

03

What are INT8, INT4 and mixed precision?

INT8 and INT4 store each weight as an 8-bit or 4-bit integer, which can take 256 or 16 distinct values. Mixed precision uses different bit widths for different parts of the same model.

The integers are mapped back to real numbers with a scale and, in some schemes, an offset. In the scheme Jacob and colleagues used for integer-only inference, a real value r is represented by an integer q through a real-valued scale S and an integer zero-point Z, the integer that stands for exactly zero 2:

Trained weights are spread fairly evenly around zero, so they are often quantized symmetrically, with Z fixed at 0. A 4-bit integer can hold the 16 values from −8 to 7; a symmetric scheme can use −7 to 7, so that the levels sit evenly either side of zero, and set S to the largest weight’s magnitude divided by 7. Figure 1 below uses that scheme. At 8 bits the divisor is 127, so the steps between levels are about 18 times finer (127 ÷ 7 ≈ 18.1).

Not every part of a model is equally sensitive to rounding, so many formats mix precisions. Dettmers and colleagues found that large language models develop a few outlier features with very large values; their LLM.int8() method handles those dimensions in 16-bit while “more than 99.9% of values are multiplied in 8-bit” 3. Lin and colleagues found that protecting about 1% of the most important weights, identified from the activations they meet rather than from the weights themselves, greatly reduces quantization error 6. The same idea appears in files made for phones: litert-community says its build of Gemma 4 E2B, the file Brello Vision runs, “uses a mixture of 2bit, 4bit and 8 bit weights” 11.

04

How does block-wise quantization work?

Block-wise quantization splits the weights into small groups, such as 32 consecutive weights, and gives each group its own scale. An unusually large weight then coarsens the levels of its own block and no other.

With one scale for a whole matrix, the step size has to stretch to cover the largest weight anywhere in it, and with only 16 levels, ordinary weights end up crowded into the few levels nearest zero. Smaller blocks keep the step matched to the values around it. In more than 35,000 experiments on inference at 3 to 8 bits, Dettmers and Zettlemoyer found small block sizes to be one of only two changes that improved the trade-off between bits and accuracy; the other was the choice of data type 4.

The scales are not free. One 16-bit scale per 32 weights adds 16 ÷ 32 = 0.5 bits to every weight, so a 4-bit block format averages 4.5 bits per weight. Smaller blocks follow the weights more closely but spend more memory on scales.

Figure 1 follows one block of 32 weights through symmetric 4-bit quantization, using the block size of the file Brello Core runs, and then applies the same arithmetic to a whole model.

Two things in the figure hold in general. Rounding error is bounded: no weight moves by more than half a step, here 0.0041 against a half-step of 0.00415. And the largest value in a block decides the step for everything else in it. One weight of 0.0581 sets a step of 0.0083, and the other 31 weights use only the seven codes from −3 to +3. That is the outlier problem in miniature, and it is why the block, not the whole matrix, is the unit that gets a scale.

05

Are weights and activations quantized the same way?

Usually not. Weights are fixed once a model is trained, so they can be quantized ahead of time; activations, the intermediate values computed from each input, change with every token and are often kept at higher precision.

Weight-only quantization shrinks the file and the memory the weights occupy; as the model runs, the stored integers are combined with activations held at higher precision. Brello Core’s file is described by its publisher as a “dynamic INT4 variant (block-32 weights, FP32 activations)”: weights stored as 4-bit integers in blocks of 32, and activations kept as 32-bit floating-point numbers 10.

“Dynamic” refers to how activations are handled. In LiteRT’s documentation, dynamic-range quantization “statically quantizes only the weights from floating point to integer at conversion time”; operators that support it then quantize activations on the fly, from the range of values they actually see, and the outputs “are still stored using floating point” 8.

Full integer quantization goes further and makes, in LiteRT’s words, “all model math” integer 8. That is the approach Jacob and colleagues developed for hardware with fast integer arithmetic. It has to fix a range for every activation in advance, and they trained their models with quantization simulated during training so that accuracy held up 2. Activations are harder to quantize in large language models because of the outlier features described above, a few channels with values far larger than the rest 3. Keeping activations in floating point, as Brello Core’s file does, sidesteps that problem.

06

What does quantization cost in accuracy?

Some accuracy, by an amount that depends on the bit width, the method and the model. Published results show little or no loss at 8 bits and small losses at 4 bits with careful methods, and below 4 bits the trade-off usually gets worse.

Three results frame the range. LLM.int8() ran inference at 8 bits in models of up to 175 billion parameters “without any performance degradation” 3. GPTQ, which adjusts the weights not yet rounded to make up for each one it rounds, reduced GPT models with 175 billion parameters to 3 or 4 bits per weight “with negligible accuracy degradation relative to the uncompressed baseline” 5. And across their 35,000 experiments, Dettmers and Zettlemoyer found 4-bit precision “almost universally optimal for total model bits and zero-shot accuracy”: for a fixed memory budget, a larger model at 4 bits tended to beat a smaller one at higher precision 4.

Those results come from particular models, methods and benchmarks, mostly far larger than a phone model, and they do not transfer automatically to a model with 1.7 billion parameters or to a particular file. Rounding to the nearest level, as in Figure 1, is the simplest method; GPTQ and AWQ choose their roundings with the help of a small sample of data, and both report lower error than plain rounding 5 6.

We have not measured how much accuracy the files Brello 1.0 runs lose against their 16-bit originals, so this page gives no figure for them. Like any on-device model, they can be wrong for many reasons; AI hallucinations, explained covers the most visible one.

07

Which quantized files does Brello 1.0 run?

Brello 1.0 runs three LiteRT-LM files that their publisher, litert-community on Hugging Face, quantized when converting the models: 977 MB for Brello Core, 2.59 GB for Brello Vision and 3.66 GB for Brello Pro. Brello downloads them as published, without an account.

Brello Core’s file shows the arithmetic at work. Its 1.7 billion weights at 4.5 bits come to about 0.96 GB, close to the 977,184,032-byte file. litert-community also publishes an INT8 build of the same model, recipe dynamic_wi8_afp32, of 2,056,729,520 bytes, 2.1 times the size 10. Moving from 8-bit to 4-bit weights roughly halved the download.

The Gemma 4 files are organised differently. Google’s E2B and E4B models use per-layer embeddings, which give “each decoder layer its own small embedding for every token”, so the model card counts 2.3 billion effective parameters for E2B (5.1 billion with embeddings) and 4.5 billion for E4B (8 billion with embeddings) 13. litert-community says the E2B file’s “weight footprint in memory can be as low as 0.8 GB while the runtime uses memory mapping to support the 1.12GB of embedding parameters” 11. Memory mapping lets the operating system read parts of a file from storage as they are needed, so the embedding tables do not all have to sit in RAM at once.

For how these files reach a phone and run there, see how a local language model runs on Android, and for how Brello matches a model to a phone’s memory, see our research note on fitting a model to the phone.