ThousandML / Journal / Release
GCMP: LoRA quality with 55× fewer parameters
Training Factory now ships GCMP, a parameter-efficient adapter that replaces LoRA. On Qwen3.8-27B it matched LoRA r=32 on every benchmark we ran, with 2.9M trainable parameters instead of ~159M. It works on a full-precision base too. Paired with a CQ 4-bit base, it lets you fine-tune a 27B model on one 48 GB GPU and ship adapters measured in megabytes.
55×
fewer trainable parameters than LoRA r=32
~11 MB
adapter size, vs. ~320 MB for LoRA
10 / 10
checkpoints with lower validation loss than LoRA
Qwen3.8-27B, Alpaca, 500 steps. Same evaluation code, prompts, and questions for every row. Greedy decoding. Significance from paired tests: exact McNemar for right/wrong tasks, bootstrap for losses.
What GCMP is
LoRA made fine-tuning practical by training small low-rank matrices on a frozen base. GCMP takes the same idea further. It trains far fewer parameters while reaching the same quality. On Qwen3.8-27B, a GCMP adapter has 2.9M trainable parameters, compared with ~159M for LoRA r=32.
Smaller adapters change the economics of fine-tuning. You can keep one adapter per customer, team, or task without a storage problem. You can load many of them onto one server and pick one per request. And because the base stays frozen, your data and adapters stay on your own hardware.
GCMP does not require quantization. It works on full-precision BF16/FP16 models and on 4-bit bases. In Training Factory, it is now the default adapter.
How we measured
An earlier version of our results compared training-log losses across two different trainers. That comparison was not valid (see the correction below). Community feedback pushed us to measure properly, so every number in this post follows the same rules:
- —Same code everywhere: every model is scored with the same lm-evaluation-harness code, prompts, and questions.
- —Decoding: greedy, with Qwen's thinking mode off.
- —Statistics: paired comparisons. Exact McNemar test for right/wrong tasks, paired bootstrap for losses.
- —Losses: computed with one formula for every model, never taken from training logs.
The benchmarks are IFEval, GSM8K, SQuAD (extraction), ARC-Challenge, HellaSwag, WinoGrande, BoolQ, MMLU, and TruthfulQA. We dropped DROP after it scored about 0.02 for every model, which points to a broken task setup rather than a real result.
The base: CQ matches BF16, NF4 does not
An adapter can only build on what the base still knows. So before comparing adapters, we compared the 4-bit bases against full precision on nine downstream tasks.
| Base model | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | HellaSwag | WinoGrande | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| BF16 (full precision) | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.737 | 0.758 | 0.542 |
| CQ 4-bit | 0.650 | 0.746 | 0.880 | 0.598 | 0.840 | 0.820 | 0.735 | 0.780 | 0.537 |
| NF4 4-bit (bitsandbytes) | 0.563 | 0.716 | 0.852 | 0.582 | 0.839 | 0.825 | 0.736 | 0.770 | 0.550 |
- —CQ vs. BF16: no significant difference on any task.
- —NF4 vs. BF16: NF4 is 12.0 points lower on GSM8K (p = 3e-6) and 3 points lower on SQuAD.
- —CQ vs. NF4: CQ is 8.7 points higher on GSM8K (p = 0.0005) and 2.8 points higher on BoolQ (p = 0.02).
Math is where 4-bit quantization error shows up most clearly, and it is where CQ and NF4 differ most. Fidelity against the BF16 model on WikiText-2 points the same way:
| Metric | CQ 4-bit | NF4 |
|---|---|---|
| Mean KL divergence ↓ | 0.0282 | 0.0330 |
| 99th-percentile KL ↓ | 0.976 | 1.181 |
| Top-1 agreement ↑ | 92.42% | 91.95% |
CQ is also smaller: an 18 GB checkpoint vs. 22.4 GB for NF4. It needs no calibration data. You point the quantizer at a Hugging Face checkpoint and it runs layer by layer.
GCMP vs. LoRA r=32
To test GCMP on its own, we used the original BF16 Qwen3.8-27B with no quantization. Both adapters were trained with Unsloth using the same settings.
| Configuration | Trainable params | Answer loss ↓ | GSM8K | SQuAD | BoolQ | ARC-C | MMLU | IFEval | TruthfulQA |
|---|---|---|---|---|---|---|---|---|---|
| No adapter | – | 1.678 | 0.683 | 0.746 | 0.874 | 0.588 | 0.839 | 0.805 | 0.542 |
| LoRA r=32 | ~159M | 1.480 | 0.903 | 0.776 | 0.896 | 0.612 | 0.844 | 0.825 | 0.518 |
| GCMP | 2.9M | 1.471 | 0.890 | 0.760 | 0.892 | 0.618 | 0.846 | 0.815 | 0.514 |
- —Benchmarks: no significant difference between GCMP and LoRA on any task (McNemar p ≥ 0.2 on every right/wrong task).
- —Answer-token loss: slightly lower for GCMP (−0.009, 95% CI [−0.010, −0.008]).
- —Validation loss: lower for GCMP at all 10 checkpoints, on both BF16 (1.2385 vs. 1.2434) and NF4 (1.2435 vs. 1.2484) bases.
- —Effect of fine-tuning: both adapters raise GSM8K by about 21 points and lower TruthfulQA by about 2.5 points, which is typical of Alpaca tuning.
We are not claiming GCMP beats LoRA on tasks. It matches LoRA at a fraction of the size. Benchmarks for LoRA and GCMP on CQ and NF4 bases are still running. We will publish them when they finish.
Why adapter size matters
For a single fine-tune, 320 MB vs. 11 MB makes little difference. It starts to matter when you have many fine-tunes:
| Configuration | 1 adapter | 100 adapters | 1,000 adapters |
|---|---|---|---|
| LoRA r=32 | ~320 MB | ~32 GB | ~320 GB |
| GCMP | ~11 MB | ~1.1 GB | ~11 GB |
Smaller adapters cost less to store, load faster, and use less GPU memory to keep resident. That last point decides how many fine-tuned variants one GPU can serve.
Serving with vLLM
CQ models run in vLLM through a plugin. To serve one GCMP adapter per server:
CQ_GCMP_ADAPTER=./outputs/my-gcmp \
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cqSeveral adapters can share one base model. Each request picks its adapter by model name:
CQ_GCMP_ADAPTERS="support=./outputs/support,sql=./outputs/sql" \
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cqcurl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model": "sql", "messages": [{"role": "user", "content": "..."}]}'Requests for different adapters run in the same batch. CQ supports pipeline parallelism across GPUs; tensor parallelism is not supported yet. See the multi-adapter serving guide for details.
Speed and memory
All runs use Alpaca, 500 steps, batch 2 × gradient accumulation 4, and sequence length 512.
| Base + adapter | Trainer | GPU | s/step ↓ | Peak memory |
|---|---|---|---|---|
| CQ + LoRA r=32 | Training Factory | A40 48 GB | 8.6 | 23.5 GiB |
| CQ + GCMP | Training Factory | A40 48 GB | 12.4 | 22.0 GiB |
| NF4 + LoRA r=32 | Unsloth | A40 48 GB | 3.2 | 23.8 GiB |
| NF4 + GCMP | Unsloth + GCMP | A40 48 GB | 5.6 | 21.5 GiB |
| BF16 + LoRA r=32 | Unsloth | 80 GB | 1.37 | 53.7 GiB |
| BF16 + GCMP | Unsloth + GCMP | 80 GB | 2.74 | 51.7 GiB |
GCMP uses slightly less peak memory than LoRA in every setup. It is not faster yet. Today it costs about 2× the step time of LoRA, and Unsloth is still faster than the Training Factory trainer. Our CQ trainer went from 27.6 s to 8.6 s per step in the last few weeks, and closing the remaining gap is our top priority.
A correction to our earlier post
In our earlier QLoRA post, we compared training-log losses from the CQ trainer and Unsloth directly (0.84 vs. 1.25). The two trainers tokenize, mask, and average loss differently, so those numbers are not on the same scale. We no longer use that comparison. The same-formula losses and downstream benchmarks above replace it.
Limitations
- —Results so far cover one model (Qwen3.8-27B) and one dataset (Alpaca), using 200–1,140 questions per task. That is enough to detect differences of about 5 points, but not 1.
- —Training runs on a single GPU. CQ serving supports pipeline parallelism but not tensor parallelism. vLLM serving of GCMP currently requires a CQ base.
- —MoE expert weights and non-CQ dense layers are not adapted.
- —Measure quality, speed, and memory on your own data before relying on these numbers.
Try it
Install Training Factory and download a CQ base, then train a GCMP adapter:
python training/train.py \
--model ./models/Qwen3.8-27B-CQ-4bit \
--data training/examples/train.jsonl \
--output ./outputs/my-gcmp \
--kernel-module cq.cq_kernels \
--adapter gcmpOr train GCMP on a full-precision base with no quantization (80 GB GPU for 27B):
python training/train_unsloth_16bit_gcmp.py \
--model Qwen/Qwen3.8-27B --dtype bfloat16 \
--train-data training/examples/train.jsonl \
--output ./outputs/my-gcmp-bf16To reproduce every number in this post, follow the training and evaluation commands in the repo README. They train the six adapters, run the same evaluation suite, and produce the comparison tables with significance tests.
The Python code is open source. The CUDA kernels ship compiled with a license check, which is free during early access. Everything runs on your own GPUs. CQ models from 8B to 397B are on Hugging Face.
If you run these benchmarks on other models or datasets, we would like to hear from you, especially if your results differ from ours: contact@thousandml.com.
Guides: train and serve your first GCMP adapter, choose between BF16, CQ, and NF4, and serve many adapters from one base.