ThousandML / Journal / Release

GCMP: LoRA quality with 55× fewer parameters

Training Factory now ships GCMP, a parameter-efficient adapter that replaces LoRA. On Qwen3.8-27B it matched LoRA r=32 on every benchmark we ran, with 2.9M trainable parameters instead of ~159M. It works on a full-precision base too. Paired with a CQ 4-bit base, it lets you fine-tune a 27B model on one 48 GB GPU and ship adapters measured in megabytes.

ThousandML Training Factory8 min readReleaseAlso on Hugging Face

55×

fewer trainable parameters than LoRA r=32

~11 MB

adapter size, vs. ~320 MB for LoRA

10 / 10

checkpoints with lower validation loss than LoRA

Qwen3.8-27B, Alpaca, 500 steps. Same evaluation code, prompts, and questions for every row. Greedy decoding. Significance from paired tests: exact McNemar for right/wrong tasks, bootstrap for losses.

What GCMP is

LoRA made fine-tuning practical by training small low-rank matrices on a frozen base. GCMP takes the same idea further. It trains far fewer parameters while reaching the same quality. On Qwen3.8-27B, a GCMP adapter has 2.9M trainable parameters, compared with ~159M for LoRA r=32.

Smaller adapters change the economics of fine-tuning. You can keep one adapter per customer, team, or task without a storage problem. You can load many of them onto one server and pick one per request. And because the base stays frozen, your data and adapters stay on your own hardware.

GCMP does not require quantization. It works on full-precision BF16/FP16 models and on 4-bit bases. In Training Factory, it is now the default adapter.

How we measured

An earlier version of our results compared training-log losses across two different trainers. That comparison was not valid (see the correction below). Community feedback pushed us to measure properly, so every number in this post follows the same rules:

  • —Same code everywhere: every model is scored with the same lm-evaluation-harness code, prompts, and questions.
  • —Decoding: greedy, with Qwen's thinking mode off.
  • —Statistics: paired comparisons. Exact McNemar test for right/wrong tasks, paired bootstrap for losses.
  • —Losses: computed with one formula for every model, never taken from training logs.

The benchmarks are IFEval, GSM8K, SQuAD (extraction), ARC-Challenge, HellaSwag, WinoGrande, BoolQ, MMLU, and TruthfulQA. We dropped DROP after it scored about 0.02 for every model, which points to a broken task setup rather than a real result.

The base: CQ matches BF16, NF4 does not

An adapter can only build on what the base still knows. So before comparing adapters, we compared the 4-bit bases against full precision on nine downstream tasks.

Base modelGSM8KSQuADBoolQARC-CMMLUIFEvalHellaSwagWinoGrandeTruthfulQA
BF16 (full precision)0.6830.7460.8740.5880.8390.8050.7370.7580.542
CQ 4-bit0.6500.7460.8800.5980.8400.8200.7350.7800.537
NF4 4-bit (bitsandbytes)0.5630.7160.8520.5820.8390.8250.7360.7700.550
  • —CQ vs. BF16: no significant difference on any task.
  • —NF4 vs. BF16: NF4 is 12.0 points lower on GSM8K (p = 3e-6) and 3 points lower on SQuAD.
  • —CQ vs. NF4: CQ is 8.7 points higher on GSM8K (p = 0.0005) and 2.8 points higher on BoolQ (p = 0.02).

Math is where 4-bit quantization error shows up most clearly, and it is where CQ and NF4 differ most. Fidelity against the BF16 model on WikiText-2 points the same way:

MetricCQ 4-bitNF4
Mean KL divergence ↓0.02820.0330
99th-percentile KL ↓0.9761.181
Top-1 agreement ↑92.42%91.95%

CQ is also smaller: an 18 GB checkpoint vs. 22.4 GB for NF4. It needs no calibration data. You point the quantizer at a Hugging Face checkpoint and it runs layer by layer.

GCMP vs. LoRA r=32

To test GCMP on its own, we used the original BF16 Qwen3.8-27B with no quantization. Both adapters were trained with Unsloth using the same settings.

ConfigurationTrainable paramsAnswer loss ↓GSM8KSQuADBoolQARC-CMMLUIFEvalTruthfulQA
No adapter–1.6780.6830.7460.8740.5880.8390.8050.542
LoRA r=32~159M1.4800.9030.7760.8960.6120.8440.8250.518
GCMP2.9M1.4710.8900.7600.8920.6180.8460.8150.514
  • —Benchmarks: no significant difference between GCMP and LoRA on any task (McNemar p ≥ 0.2 on every right/wrong task).
  • —Answer-token loss: slightly lower for GCMP (−0.009, 95% CI [−0.010, −0.008]).
  • —Validation loss: lower for GCMP at all 10 checkpoints, on both BF16 (1.2385 vs. 1.2434) and NF4 (1.2435 vs. 1.2484) bases.
  • —Effect of fine-tuning: both adapters raise GSM8K by about 21 points and lower TruthfulQA by about 2.5 points, which is typical of Alpaca tuning.

We are not claiming GCMP beats LoRA on tasks. It matches LoRA at a fraction of the size. Benchmarks for LoRA and GCMP on CQ and NF4 bases are still running. We will publish them when they finish.

Why adapter size matters

For a single fine-tune, 320 MB vs. 11 MB makes little difference. It starts to matter when you have many fine-tunes:

Configuration1 adapter100 adapters1,000 adapters
LoRA r=32~320 MB~32 GB~320 GB
GCMP~11 MB~1.1 GB~11 GB

Smaller adapters cost less to store, load faster, and use less GPU memory to keep resident. That last point decides how many fine-tuned variants one GPU can serve.

Serving with vLLM

CQ models run in vLLM through a plugin. To serve one GCMP adapter per server:

CQ_GCMP_ADAPTER=./outputs/my-gcmp \
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq

Several adapters can share one base model. Each request picks its adapter by model name:

CQ_GCMP_ADAPTERS="support=./outputs/support,sql=./outputs/sql" \
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq
curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "sql", "messages": [{"role": "user", "content": "..."}]}'

Requests for different adapters run in the same batch. CQ supports pipeline parallelism across GPUs; tensor parallelism is not supported yet. See the multi-adapter serving guide for details.

Speed and memory

All runs use Alpaca, 500 steps, batch 2 × gradient accumulation 4, and sequence length 512.

Base + adapterTrainerGPUs/step ↓Peak memory
CQ + LoRA r=32Training FactoryA40 48 GB8.623.5 GiB
CQ + GCMPTraining FactoryA40 48 GB12.422.0 GiB
NF4 + LoRA r=32UnslothA40 48 GB3.223.8 GiB
NF4 + GCMPUnsloth + GCMPA40 48 GB5.621.5 GiB
BF16 + LoRA r=32Unsloth80 GB1.3753.7 GiB
BF16 + GCMPUnsloth + GCMP80 GB2.7451.7 GiB

GCMP uses slightly less peak memory than LoRA in every setup. It is not faster yet. Today it costs about 2× the step time of LoRA, and Unsloth is still faster than the Training Factory trainer. Our CQ trainer went from 27.6 s to 8.6 s per step in the last few weeks, and closing the remaining gap is our top priority.

A correction to our earlier post

In our earlier QLoRA post, we compared training-log losses from the CQ trainer and Unsloth directly (0.84 vs. 1.25). The two trainers tokenize, mask, and average loss differently, so those numbers are not on the same scale. We no longer use that comparison. The same-formula losses and downstream benchmarks above replace it.

Limitations

  • —Results so far cover one model (Qwen3.8-27B) and one dataset (Alpaca), using 200–1,140 questions per task. That is enough to detect differences of about 5 points, but not 1.
  • —Training runs on a single GPU. CQ serving supports pipeline parallelism but not tensor parallelism. vLLM serving of GCMP currently requires a CQ base.
  • —MoE expert weights and non-CQ dense layers are not adapted.
  • —Measure quality, speed, and memory on your own data before relying on these numbers.

Try it

Install Training Factory and download a CQ base, then train a GCMP adapter:

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/train.jsonl \
  --output ./outputs/my-gcmp \
  --kernel-module cq.cq_kernels \
  --adapter gcmp

Or train GCMP on a full-precision base with no quantization (80 GB GPU for 27B):

python training/train_unsloth_16bit_gcmp.py \
  --model Qwen/Qwen3.8-27B --dtype bfloat16 \
  --train-data training/examples/train.jsonl \
  --output ./outputs/my-gcmp-bf16

To reproduce every number in this post, follow the training and evaluation commands in the repo README. They train the six adapters, run the same evaluation suite, and produce the comparison tables with significance tests.

The Python code is open source. The CUDA kernels ship compiled with a license check, which is free during early access. Everything runs on your own GPUs. CQ models from 8B to 397B are on Hugging Face.

If you run these benchmarks on other models or datasets, we would like to hear from you, especially if your results differ from ours: contact@thousandml.com.

Guides: train and serve your first GCMP adapter, choose between BF16, CQ, and NF4, and serve many adapters from one base.