Three numbers tell this story: 54 GB. 18 GB. 3.9 GB.
PrismML announced Bonsai 27B on July 12-13: two quantized variants of Qwen3.6 27B, one 1-bit and one ternary. The full-precision 16-bit build of a 27B model needs ~54 GB of memory. Even a 4-bit build lands at 18 GB. Neither fits a phone, and most laptops struggle. PrismML’s 1-bit variant compresses the footprint to 3.9 GB; the ternary variant to 5.9 GB. Both land inside the memory budget of consumer hardware.
But the headline number is less interesting than what it implies: how they got there, and what it means for agent deployment shape.
The memory math: 54 GB to 3.9 GB, step by step
Laying out the progression:
- 16-bit full precision: 27B × 2 bytes ≈ 54 GB. Industry standard for both training and inference.
- 4-bit quantization: ~18 GB. The standard INT4 / GPTQ / AWQ path. Still too large for a phone, and tight on most laptops.
- Ternary ({-1, 0, +1}) with FP16 group-wise scaling: 1.71 effective bits per weight, 5.9 GB. PrismML’s quality-tier variant.
- 1-bit ({-1, +1}) with the same scaling: 1.125 effective bits per weight, 3.9 GB. The phone-budget variant.
The key phrase is “effective bits per weight.” Ternary isn’t a naive 16-bit-to-3-bit shrink. Each weight is encoded in 1.71 bits on average. Binary gives 14x compression per weight. The trade-off is unavoidable loss in numerical representation.
Why the iPhone couldn’t fit a 27B model: the KV cache wall
This is more subtle than “the model is too big.” PrismML states it directly:
A phone never exposes its full memory to an app — a 12 GB iPhone offers about 6 GB for the model to use on-device, and the model shares that budget with its KV cache and activations.
iPhone 17 Pro ships 12 GB of physical RAM. A single app gets roughly 6 GB. That 6 GB has to cover three things: model weights, KV cache, and runtime activations. Open a 262K-token conversation and the KV cache eats multiple GB; the remaining model-weight budget sits around 4 GB. 1-bit Bonsai at 3.9 GB just fits. The 4-bit build at 18 GB blows past the cap entirely.
It wasn’t that nobody tried to put a 27B model on a phone — that constraint locked the path for three years. Bonsai 1-bit 27B is the first to walk through.
The real benchmark signal: not “scored high,” but “what didn’t drop”
PrismML ran a 15-task suite — knowledge, reasoning, math, coding, instruction following, tool calling, vision — all in thinking mode where the model’s full reasoning is exercised:
- Ternary retains 95% of the full-precision baseline
- 1-bit retains 90%
The averages understate the story. The capability-by-capability breakdown is what matters:
- math: nearly untouched
- coding: nearly untouched
- tool calling: within a few points of full precision
Those are the three capabilities agentic workloads actually depend on. The total score drops 5%-10%, but writing code, calling tools, and doing math reasoning are almost fully preserved. The compression scheme loses the most on knowledge / chat benchmarks; it preserves what agents need.
The contrast is sharper. The same Qwen3.6 27B base, run through the most aggressive conventional low-bit build, scores lower than 1-bit Bonsai while occupying 2.5x the memory.
Intelligence density: 0.53 / GB, a new axis, not just better efficiency
PrismML introduced this metric back with 1-bit Bonsai 8B: how much capability per GB of model. Bonsai 27B 1-bit lands at:
- 0.53 per GB
- more than 10x the full-precision baseline
- roughly 2.7x the best conventional low-bit alternative
Translated: same RAM budget, full-precision gives you “1 unit of capability”; 1-bit Bonsai gives you “10 units.” That is a leftward shift of the entire Pareto frontier on the capability-vs-footprint axis.
The architectural reversal: hybrid on-device / cloud agents
This, to me, is the most important part of the announcement. PrismML writes:
Cloud APIs will remain the right choice for many products. But for agentic workloads, cloud-only execution imposes structural constraints: every step is a remote request, per-token cost accumulates with every iteration, and every plan, tool call, and intermediate result crosses the network including the user’s private files, screen, and data.
In engineering terms: an agent running one task might make hundreds of model calls; every call is a remote request; every token is metered; every intermediate result — including the user’s private files, screen contents, and local data — traverses the network. Three constraints, each amplified 100x in the agent regime.
Local execution flips the math. A hundred-step loop has zero marginal cost. User data never leaves the device. Two product categories open up: offline assistants, and privacy-sensitive on-device agents.
And the deeper shift is hybrid deployment:
hybrid deployments that route non-frontier and privacy-sensitive tasks to a capable local model and reserve frontier cloud models for the hardest steps
Agents no longer route every step to one giant cloud model. Routine + privacy-sensitive work runs on a local 27B. The genuinely hard steps go to a frontier cloud model. This is a re-layering of agent deployment, not just a smaller model.
262K context, speculative decoding, Apache 2.0
A few more engineering details that matter:
- 262K-token context: covers a full technical book or a long meeting transcript
- vision tower in 4-bit: multimodal isn’t marketing — the vision path got its own compression pass
- speculative decoding: lossless draft-and-verify acceleration
- Apache 2.0: commercial use basically unrestricted
Throughput:
- 1-bit: 163 tok/s on RTX 5090, 87 tok/s on M5 Max
- Ternary: 134 tok/s on RTX 5090, 58 tok/s on M5 Max
Native support: Apple Silicon via MLX, NVIDIA via CUDA with custom low-bit kernels. PrismML’s team is Caltech-origin, backed by Khosla Ventures, Cerberus, and Google, with continuing support from Samsung.