04Model comparison

Every model we serve, and what it costs in memory.

Published benchmark scores, the memory each model needs at a given precision, and the licence you get. All of these are open-weight models you download, pin and run yourself.

Size it before you sign it Reproduce the numbers

01Read this first

Three things that decide the answer.

Total parameters set the memory, not active onesA sparse model activates a fraction of its weights per token, but every weight still has to sit in VRAM. A 753B model needs the memory of a 753B model.

Context window costs memory per conversationWeights are fixed. The key-value cache is not: a longer context means more memory per user, which means smaller batches and fewer people on the same box.

Benchmark names are not interchangeableSWE-bench Verified and SWE-bench Pro are different suites, and agentic scores depend on the scaffold. Two numbers in the same column are only comparable if the harness matched.

02The table

What we deploy, and what each one needs.

Model	Parameters	Weights in VRAM	Smallest node	Context	Licence	Published headline
Kimi K2.7Frontier quality, strongest for agent work	1T total	~1.0 TB at FP8	8x H200	256k	Modified MIT	Scores for this release are not published. The preceding K2.6 release led open models on aggregate intelligence indices.
GLM 5.2Best all-round open model on published indices	753B / ~40B active	~750 GB at FP8 ~380 GB at Q4	8x H200 8x H100 at Q4	1M	MIT	91.2 GPQA Diamond, 81.0 Terminal-Bench 2.1, 62.1% SWE-bench Pro (vendor)
GLM 4.6The value option when 5.2 is more than you need	355B	~355 GB at FP8 ~180 GB at Q4	8x H100 4x H100 at Q4	200k	MIT	Superseded by 5.2. Vendor scores published on the model card.
DeepSeek V4 FlashSparse attention, small footprint for its class	284B	~284 GB at FP8 ~145 GB at Q4	4x H100 2x H100 at Q4	128k	MIT	Inherits the V4 architecture. The V4 Pro flagship reports 80.6% SWE-bench Verified and 90.1 GPQA Diamond.
DeepSeek V3.2Strong on mathematics and structured derivation	685B / 37B active	~685 GB at FP8 ~350 GB at Q4	8x H200 8x H100 at Q4	128k	MIT	88.5 MMLU. Behind the V4 line on agentic coding.
Qwen3-CoderBuilt for code, used behind Senpiper Build	480B / 35B active	~480 GB at FP8 ~245 GB at Q4	8x H100 4x H100 at Q4	256k	Apache 2.0	Vendor scores published on the model card. Qwen3-235B reports 80.6 MMLU-Pro and 69.5 LiveCodeBench.
GPT-OSS 120BThe one that fits on a single 80 GB card	117B	~63 GB at MXFP4	1x H100	128k	Apache 2.0	0.878 LiveCodeBench. Weak on agentic terminal work at 0.235 Terminal-Bench Hard.

Memory figures are weights only, computed from the parameter count at the stated precision. Add the key-value cache for your context window and batch size, which the sizing calculator does for you. Benchmark scores are as published by the model vendors or independent harnesses at the date of the last review, and they move. Where a score is not published for a release, we say so rather than borrowing one from a different version.

03Picking one

The workload decides, not the leaderboard.

Tell us what the box is for and the shortlist is usually two models, not seven. Pick the job below to see where we would start.

Where we would startGLM 5.2Leads the open field on published reasoning scores, and the 1M context handles long policy documents without chunking around them.GPT-OSS 120B if the budget is one 80 GB card. It gives up agentic ability, not general chat quality.

04Reproduce it

The harnesses we run, so you can run them too.

Every score in the table above comes from one of these. They are all open source, and on your own box you can run them against your own prompts, which matters more than any public number.

lm-evaluation-harnessEleutherAI/lm-evaluation-harness

The knowledge and reasoning scores: MMLU-Pro, GPQA, IFEval and around sixty other academic suites. It is the backend for the Hugging Face Open LLM Leaderboard and it serves through vLLM, so it runs against the same deployment your users will hit.

SWE-benchSWE-bench/SWE-bench

Real GitHub issues resolved against real repositories. Verified is the human-checked subset most vendors quote; Pro is the harder multi-language successor with a standardised scaffold. Quote which one you mean.

LiveCodeBenchLiveCodeBench/LiveCodeBench

Code generation on problems published after each model's training cut-off, which is the point: contamination is the main reason older coding benchmarks flatter newer models.

vLLM benchmarksvllm-project/vllm

Throughput and latency on your actual hardware. This is where the tokens per second and concurrent user figures on the sizing page come from, served per the official vLLM recipe for each model.

05Why open weights

You can pin a version and prove nothing left.

A hosted model can change under you between one procurement cycle and the next. A downloaded model cannot. For a regulated buyer that is usually worth more than the point or two a closed model leads by.

01The exact weights you certified are the weights running in year three.

02You can fine-tune on your own data without sending it to a vendor.

03Cost is your hardware and your power bill, not a per-token meter.

04Swapping model is a redeploy, so you are not locked to one vendor's roadmap.

Sizing and hardwarePick a machine and a model, and see how many people it serves.Sovereign AIAsk questions across your own documents, policies and operational records.TrustWhere the data sits, who can reach it, and what your security team will ask.