|
|
--- |
|
|
license: mit |
|
|
base_model: zai-org/GLM-4.7 |
|
|
base_model_relation: quantized |
|
|
quantization: exl3 |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- exl3 |
|
|
library_name: exllamav3 |
|
|
--- |
|
|
|
|
|
# GLM 4.7 (EXL3 Quants) |
|
|
|
|
|
- Original Model: |
|
|
- [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) |
|
|
|
|
|
> [!WARNING] |
|
|
> ⚠️ Proper support for GLM-4.7 reasoning requires [PR #295 in TabbyAPI](https://github.com/theroyallab/tabbyAPI/pull/295#issuecomment-3689013998), see my modified Dockerfile. |
|
|
|
|
|
This repo contains: |
|
|
- base quants (2, 3, 4, 5, 6, 8 bits) for Exllamav3 (using SOTA random Hadamard transforms and Trellis quantization for high-quality reconstruction) |
|
|
- layer and tensor level KL-divergence measurements for bit-allocation optimization given a target size |
|
|
- theoretical research related to quantization, in particular MoE quantization |
|
|
|
|
|
## Motivation |
|
|
|
|
|
The goals are: |
|
|
- to provide the best possible quants for what is arguably the top general model of 2025 |
|
|
- to serve as a reference for quantization strategies (as of 2025 knowledge) |
|
|
|
|
|
The base model is 355B parameters, which when 4-bit quantized should take about 177GiB, leaving almost 20GB for context, a perfect situation when you have 196GiB of VRAM (i.e. 8x 3090/4090, 6x 5090, $x RTX A6000, 4x RTX 6000 Ada or 2x RTX Pro 6000 Blackwell). Too bad all the 4-bit quants for my usual framework of choice, vllm, start at 191~200GiB of VRAM (AWQ / GPTQ / NVFP4 are actually ~4.5bpw). |
|
|
|
|
|
So while looking for a new backend that could leverage tensor parallelism, I landed on Exllamav3. And even better it had already in place the proper tools to fully quantized Mixture-of-Experts (MoE) models, unlike vllm/llmcompressor that requires you extra code to ensure all experts are activated (or their activation might be quantized away as unimportant if you have a non-comprehensive calibration dataset). |
|
|
|
|
|
<details> |
|
|
<summary>Visual showcase of why ensuring quantization of all MoE experts is important</summary> |
|
|
|
|
|
- Source: https://avtc.github.io/aquarium-side-by-side/ |
|
|
- Context: https://github.com/ModelCloud/GPTQModel/pull/2235 |
|
|
|
|
|
 |
|
|
|
|
|
</details> |
|
|
|
|
|
## Artifacts |
|
|
|
|
|
### Base Quants |
|
|
|
|
|
The base quants use the new "MCG" multiplier from https://github.com/turboderp-org/exllamav3/pull/26#issuecomment-3395345415 |
|
|
|
|
|
- Size measured through: https://github.com/turboderp-org/exllamav3/pull/103 |
|
|
- Kullback-Leibler divergence (KL-div) and Top-K agreement measured through: https://github.com/turboderp-org/exllamav3/blob/v0.0.14/eval/model_diff.py |
|
|
- Perplexity measured through: https://github.com/turboderp-org/exllamav3/blob/v0.0.14/eval/model_diff.py |
|
|
- Caveat both quantization calibration and perplexity use the same dataset in EXL3, hence we have overfitting.\ |
|
|
The most appropriate measure for quality is KL-divergence (i.e. how well the quant reproduces the original probability distribution of token output, before samplers)\ |
|
|
For example the 3-bit quant have lower perplexity than the original FP16.\ |
|
|
|
|
|
| Quant | Size | KL-div (quant, FP16) | KL-div (FP16, quant) | Perplexity | Top-1 | Top-2 | Top-3 | Top-4 | Top-5 | |
|
|
| ---------------------------------------------------------------- | ------- | -------------------- | -------------------- | ---------- | ------ | ------ | ------ | ------ | ------ | |
|
|
| [2bpw-H6](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/2bpw_H6) | 83 GiB | 0.65096196 | 0.75914080 | 9.36106675 | 0.7315 | 0.3852 | 0.1653 | 0.0628 | 0.0221 | |
|
|
| [3bpw-H6](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/3bpw_H6) | 124 GiB | 0.27578034 | 0.28499938 | 6.95262863 | 0.8388 | 0.5717 | 0.3306 | 0.1713 | 0.0805 | |
|
|
| [4bpw-H6](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/4bpw_H6) | 165 GiB | 0.13722391 | 0.13577676 | 6.60474035 | 0.8947 | 0.6948 | 0.4810 | 0.3007 | 0.1754 | |
|
|
| [5bpw-H8](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/5bpw_H8) | 206 GiB | 0.10889671 | 0.10216227 | 6.41035355 | 0.9168 | 0.7520 | 0.5609 | 0.3905 | 0.2481 | |
|
|
| [6bpw-H8](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/6bpw_H8) | 247 GiB | 0.08202591 | 0.0784423 | 6.32611481 | 0.9334 | 0.7951 | 0.6274 | 0.4597 | 0.3190 | |
|
|
| [8bpw-H8](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/8bpw_H8) | 328 GiB | 0.07552261 | 0.07230427 | 6.38240525 | 0.9396 | 0.8172 | 0.6598 | 0.5048 | 0.3666 | |
|
|
| FP16 | 656 GiB | | | 6.49784813 | | | | | | |
|
|
|
|
|
### Optimized Quants |
|
|
|
|
|
> [!TIP] |
|
|
> 🛈 Despite the KL-divergence, even the 2.10bpw quant looks quite smart for creative writing.\ |
|
|
> Succinct test on a scenario with 1 narrator and 6 leads. |
|
|
|
|
|
> [!TIP] |
|
|
> 🛈 HuggingFace reports file sizes in GB while VRAM is in GiB, there is a factor (1024/1000)³ = 1.0734 between both. |
|
|
|
|
|
> [!WARNING] |
|
|
> ⚠️ For the 2.57bpw weights, the max number of tokens in context represents a very tight fit, there is only ~250MiB of spare space, and if you have a graphical environment ... |
|
|
|
|
|
| Quant | Size | Context / VRAM | KL-div (quant, FP16) | KL-div (FP16, quant) | Perplexity | Top-1 | Top-2 | Top-3 | Top-4 | Top-5 | |
|
|
| -------------------------------------------------------------------------------- | ---------- | ----------------------------------------- | -------------------- | -------------------- | ---------- | ------ | ------ | ------ | ------ | ------ | |
|
|
| [2.10bpw-tuned🂱](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/2.10bpw-tuned)| 86 GiB | 131072 tokens, k5v4 for 96 GiB VRAM | 0.54398251 | 0.61162654 | 7.15544606 | 0.7584 | 0.4237 | 0.1948 | 0.0801 | 0.0306 | |
|
|
| [2.57bpw-tuned🂱](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/2.57bpw-tuned)| 105 GiB | 160000 tokens, k5v4 for 128 GiB VRAM</br>102400 tokens, k5v4 for 120 GiB VRAM| 0.41910998 | 0.44874423 | 6.63182633 | 0.7903 | 0.4787 | 0.2463 | 0.1132 | 0.0482 | |
|
|
| [3.15bpw-tuned🂱](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/3.15bpw-tuned)| 129 GiB | 102400 tokens, k5v4 for 144 GiB VRAM | 0.21854555 | 0.21465828 | 6.35729832 | 0.8573 | 0.6119 | 0.3776 | 0.2107 | 0.1071 | |
|
|
| [3.84bpw-tuned🂱](https://huggingface.co/mratsim/GLM-4.7-EXL3/tree/3.84bpw-tuned)| 158 GiB | 202752 tokens (max), k6v5 for 192GiB VRAM | 0.15823333 | 0.15401253 | 6.41935951 | 0.8854 | 0.6743 | 0.4587 | 0.2832 | 0.1638 | |
|
|
|
|
|
- "opt🂡" for automatically optimized quants |
|
|
- "tuned🂱" for hand-tuned quants |
|
|
|
|
|
They can be downloaded with huggingface-cli with the following command: |
|
|
``` |
|
|
hf download mratsim/GLM-4.7-EXL3 --revision 3.84bpw-tuned --local-dir /path/to/your/models/directory |
|
|
``` |
|
|
|
|
|
Unfortunately, as of December 2025 automatically optimized quants are not able to beat hand-tuned heuristics and research-based mixed-precision quantization for I suspect one of 2 reasons (or both): |
|
|
- An optimization algorithm with no backtracking, i.e. single-pass but not comparing current layer importance with past layer importance. |
|
|
- Not taking synergies into account. Just like LLMs have emergent properties with size, it might be that up-quantizing certain projections significantly improve KL-divergence even if it appears as noise if we only measure improvement of a single up-quant. |
|
|
|
|
|
### Modified Dockerfile |
|
|
|
|
|
You can build a custom tabbyAPI image that integrates PR #295: |
|
|
|
|
|
<details> |
|
|
<summary>Dockerfile</summary> |
|
|
|
|
|
```Dockerfile |
|
|
# Use an official CUDA runtime with Ubuntu as a parent image |
|
|
FROM nvidia/cuda:12.8.1-runtime-ubuntu24.04 |
|
|
|
|
|
# Install system dependencies |
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \ |
|
|
build-essential \ |
|
|
curl \ |
|
|
ca-certificates \ |
|
|
python3.12 \ |
|
|
python3-pip \ |
|
|
python3.12-venv \ |
|
|
git \ |
|
|
&& rm -rf /var/lib/apt/lists/* |
|
|
|
|
|
# Create a virtual environment |
|
|
RUN python3 -m venv /opt/venv |
|
|
|
|
|
# Activate the venv and set the PATH |
|
|
ENV PATH="/opt/venv/bin:$PATH" |
|
|
|
|
|
# Upgrade pip and install uv |
|
|
RUN pip install --no-cache-dir --upgrade pip |
|
|
|
|
|
# Set the working directory in the container |
|
|
WORKDIR /app |
|
|
|
|
|
# Clone tabbyAPI repository |
|
|
RUN git clone https://github.com/theroyallab/tabbyAPI.git /app |
|
|
|
|
|
# Configure git user (required for merge) |
|
|
RUN git config --global user.email "docker@tabbyapi.local" && \ |
|
|
git config --global user.name "Docker Build" |
|
|
|
|
|
# Fetch and merge PR #295 |
|
|
RUN git fetch origin pull/295/head:pr-295 && \ |
|
|
git merge --strategy-option theirs pr-295 |
|
|
|
|
|
# Install packages specified in pyproject.toml cu12, extras |
|
|
RUN pip install --no-cache-dir .[cu12,extras] |
|
|
|
|
|
# Make port 5000 available to the world outside this container |
|
|
EXPOSE 5000 |
|
|
|
|
|
# Set the entry point |
|
|
ENTRYPOINT ["python3"] |
|
|
|
|
|
# Run main.py when the container launches |
|
|
CMD ["main.py"] |
|
|
``` |
|
|
|
|
|
</details> |
|
|
|
|
|
### Detailed measurements of KL-div improvements |
|
|
|
|
|
<details> |
|
|
<summary>Measurements for autotuning</summary> |
|
|
Exllamav3 offers tools to measure per layer (with `-l2`) or even per-tensor (with `-l3`) contributions to KL-div improvements. |
|
|
They might take 2 hours to 5 hours, if comparing 2 quants -- to 12 hours if comparing 3 quants -- to 24h of compute if comparing all quants. |
|
|
|
|
|
The json file can be fed to https://github.com/turboderp-org/exllamav3/blob/v0.0.14/util/optimize.py with a target `bpw` to output an optimized quant. |
|
|
|
|
|
Please note that from experimentations, manual tuning using the heuristics below can achieve better KL-divergence than optimizing by only mixing 3 quants and is less likely to overfit the calibration set. Having `shared experts` or `self_attn` layers use 6 or even 8-bit provide a very large improvement to KL-divergence. Even a measurement with all available quants currently doesn't achieve manual tuning results. Hence for now, I don't plan to add measurements. |
|
|
</details> |
|
|
|
|
|
## Quantization theory and heuristics for manual tuning |
|
|
|
|
|
<details> |
|
|
<summary>In-depth overview of quantization theory and heuristics for manual tuning</summary> |
|
|
|
|
|
### Layers to quantize |
|
|
|
|
|
Quantization should be focused on Linear layers (also called Dense or Fully-Connected layers i.e. MatMul+Bias) |
|
|
In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1] |
|
|
> LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the |
|
|
> LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression. |
|
|
> Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized. |
|
|
|
|
|
This is also reported in Intel and Nvidia repo: |
|
|
- https://github.com/intel/neural-compressor/issues/1963#issuecomment-2274873441 |
|
|
- https://github.com/NVIDIA/TensorRT/issues/4084#issuecomment-2294513950 |
|
|
|
|
|
EXL3 can only quantize linear layers. |
|
|
|
|
|
### Tensors to up-quantize |
|
|
|
|
|
If there is enough bits, down projections should be prioritized. |
|
|
|
|
|
According to [4] |
|
|
> Fig. 3: Maximum absolute value over layers for a LLaMA3-8B. |
|
|
> Each color represent a different projection and we clearly see that down_proj has the biggest |
|
|
> spikes in input and output. We also observe that RMSNorm propagate spikes through the entire model |
|
|
|
|
|
According to [5] |
|
|
> Figure 5(a) illustrates the extremal ratio across layers and modules in LLaMA2-7B, highlighting |
|
|
> that weight outliers are concentrated in the down-projection matrices Wdown |
|
|
> ℓ of the second layer and |
|
|
> the last two layers. Figures 5(b) and 5(c) provide detailed visualizations of these outliers in the last |
|
|
> two layers. |
|
|
|
|
|
### Mixture-of-Experts quantization (MoE) |
|
|
|
|
|
Mixture-of-Experts require specific quantization techniques. |
|
|
|
|
|
#### Mixed-precision quantization |
|
|
|
|
|
Some layers have a higher impact on LLM performance. |
|
|
According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers. |
|
|
According to [3] on 2-bit quantization: |
|
|
- quantizing expert FFN layers do not seriously impact model quality |
|
|
- quantizing cross-attention has some impact |
|
|
- quantizing self-attention has a large impact |
|
|
- quantizing dense FFN has a very significant impact |
|
|
|
|
|
Hence to preserve model quality we should choose not to quantize dense FFN layers and self-attention layers. |
|
|
|
|
|
We notice that: |
|
|
- official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16: |
|
|
- https://huggingface.co/openai/gpt-oss-120b/blob/main/model.safetensors.index.json |
|
|
- NVFP4 weights of DeepSeek-R1 quantized by Nvidia also keep self-attention in BF16: |
|
|
- https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4/blob/main/model.safetensors.index.json |
|
|
|
|
|
#### Layers with high-impact |
|
|
|
|
|
According to [2], giving more bits to the first `k` blocks have a significantly higher impact on model quality than for the same last `k` blocks. |
|
|
|
|
|
#### Expert quantization |
|
|
|
|
|
When quantizing MoE, quantizing activations is tricky as only a subset of experts are activated per request. |
|
|
|
|
|
EXL3 has the tooling in-place to ensure all experts are activated during quantization, though it is unsure if the dataset should be expanded to be diverse enough so that all experts have a high likelyhood of taking the full range of values they can exhibit to avoid clipping. |
|
|
|
|
|
## References |
|
|
|
|
|
1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)\ |
|
|
Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia\ |
|
|
https://arxiv.org/pdf/2506.12044 |
|
|
|
|
|
2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)\ |
|
|
Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen\ |
|
|
https://arxiv.org/pdf/2406.08155v1 |
|
|
|
|
|
3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)\ |
|
|
Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla\ |
|
|
https://arxiv.org/pdf/2310.02410 |
|
|
|
|
|
4. Precision Where It Matters: A Novel Spike\ |
|
|
Aware Mixed-Precision Quantization Strategy for\ |
|
|
LLaMA-based Language Models (2025)\ |
|
|
Lucas Maisonnave, Cyril Moineau, Olivier Bichler, and Fabrice Rastello\ |
|
|
https://arxiv.org/pdf/2504.21553 |
|
|
|
|
|
5. Systematic Outliers in Large Language Models (2025)\ |
|
|
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang\ |
|
|
https://arxiv.org/pdf/2502.06415v2 |
|
|
|
|
|
</details> |
