Introducing efficiency focused “Thinking CAP” model series.
TL;DR
One of our core aims at BottleCap is efficiency in AI. To fulfil this mission, we fine-tuned Qwen3.6-27B to reduce unnecessary reasoning while preserving answer quality.
The result:
- 46% fewer reasoning tokens on average
- Comparable benchmark performance
- Fewer reasoning loops and failure cases
- Lower latency and inference cost
- Shorter, more to the point answers
Across twelve out-of-domain benchmarks, the model produced nearly identical accuracy while using roughly half as many thinking tokens.
We are releasing the checkpoint publicly on HuggingFace under a permissive Apache 2.0 licence. Anyone can freely download the model and instantly replace their local Qwen model to save money & time.
Contact us if you want further optimizations to save even more: enterprise@bottlecapai.com
Why do reasoning models overthink?
Reasoning models have changed our expectations around language model performance. Given enough time and enough tokens, they can solve problems that were previously inaccessible to older generations of models.
The downside is that they often think far longer than necessary. Even relatively simple questions can trigger thousands of reasoning tokens:
- revisiting already established assumptions,
- repeatedly reformulating the same argument,
- getting stuck in loops,
- spending more time explaining than solving,
- unnecessary filler words and verbose style for thinking that users do not even read.
This behaviour improves benchmark performance in some settings, but it also introduces costs: higher latency, higher inference spend, lower throughput, increased energy consumption, more compute, and more opportunities for failure.
Long reasoning traces have gradually become associated with intelligence, when in reality they often represent inefficiency. We wanted to cut through all of that, on the machine level. The question we asked was simple: how much of the modern model’s reasoning is actually necessary?
The objective
Our goal was intentionally conservative: do not try to make the model smarter or teach it new capabilities. We wanted to preserve knowledge, reasoning ability, answer quality, conversational style, instruction following, and safety behaviour.
The only thing we wanted to change was the amount of computation spent reaching an answer. In other words: keeping the same model, and making sure it overthinks less.
Training approach
Starting from the Qwen3.6-27B (Qwen Team, 2026) base model, we trained on a curated set of problems covering multiple domains and difficulty levels. The training objective rewarded efficient reasoning rather than simply rewarding correctness.
Importantly, the intervention was designed to remain as non-invasive as possible. The resulting model behaves very similarly to the original checkpoint — same style, same capabilities, same knowledge — but with substantially shorter reasoning traces. The model learns to stop once it has enough information to answer confidently.
Evaluation methodology
Evaluating reasoning models is more difficult than evaluating standard language models. At the recommended sampling temperature of 1.0, output quality and length can vary substantially between runs, so single-seed numbers give an incomplete picture. To reduce noise, we evaluated using full benchmark datasets, five independent random seeds, and statistical significance testing across all comparisons.
We evaluated both in-domain tasks (held-out portions of datasets related to training) and out-of-domain tasks (designed to test generalisation). The suite covers scientific reasoning and math, knowledge-based question answering, long-context tasks, system-prompt adherence, coding and agentic tasks, multi-turn conversational behaviour, and safety and model guardrails.
Results
Out-of-domain token efficiency
| Benchmark | acc (base) | acc (Ours) | Δ acc | tok (base) | tok (Ours) | matched Δ% | looping (base→Ours) |
|---|---|---|---|---|---|---|---|
| gpqa_diamond | 85.5% ±1.4 | 83.8% ±1.9 | -1.6pp | 10,777 | 3,351 | -67.8% | 0.4% → 0.4% |
| supergpqa | 64.0% ±0.2 | 64.0% ±0.1 | -0.1pp | 8,246 | 3,384 | -58.4% | 0.2% → 0.1% |
| mmlu_pro | 85.9% ±0.2 | 85.4% ±0.2 | -0.5pp | 3,455 | 1,290 | -53.7% | 0.1% → 0.1% |
| mmlu_redux | 93.9% ±0.1 | 93.9% ±0.1 | +0.0pp | 947 | 406 | -44.8% | 0.0% → 0.0% |
| ceval | 90.6% ±0.7 | 90.3% ±0.6 | -0.3pp | 1,279 | 663 | -47.1% | 0.0% → 0.0% |
| HMMT | 88.0% ±3.7 | 84.7% ±3.7 | -3.3pp | 39,277 | 27,388 | -38.0% | 0.0% → 0.7% |
| livecodebench | 80.7% ±0.6 | 84.3% ±1.0 | +3.6pp | 15,744 | 10,158 | -41.1% | 2.0% → 2.2% |
| longbench_v2 | 62.6% ±3.6 | 60.2% ±1.7 | -2.5pp | 1,765 | 1,091 | -39.1% | 12.4% → 5.0% |
| realworldqa | 82.4% ±0.7 | 81.9% ±1.2 | -0.4pp | 2,959 | 913 | -48.5% | 0.4% → 0.1% |
| AA-LCR | 76.2% ±3.0 | 74.2% ±2.2 | -2.0pp | 2,455 | 1,337 | -45.5% | 3.8% → 1.8% |
| llm-system-prompts | 80.6% ±1.2 | 81.5% ±1.8 | +0.9pp | 1,737 | 976 | -40.0% | 11.6% → 8.2% |
| Claw-Eval | 87.0% ±1.9 | 84.4% ±1.2 | -2.6pp | 919 | 689 | -25.2% | — |
| macro (mean) | -0.7pp | -45.8% |
Settings. Base Qwen/Qwen3.6-27B vs bottlecapai/Qwen3.6-27B-Efficient (“Ours”). 5 seeds per condition; thinking on; mean ± 95% CI across seeds. Decoding: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0. Max generation tokens: 100,000 for the general suite (gpqa_diamond, mmlu_pro, longbench_v2, realworldqa) and AA-LCR; 250,000 for HMMT (Nov 2025); 32,768 for supergpqa and livecodebench; 16,384 for ceval and mmlu_redux; 15,000 for llm-system-prompts; 49,152 for Claw-Eval. matched Δ% is the average paired per-question difference in thinking tokens (negative = reduction). looping% is the share of responses stuck repeating the same reasoning chain. macro is the equal-weight mean across benchmarks.
Across all benchmarks: a -45.8% average reduction in reasoning tokens with an approximately 0.7 percentage-point average accuracy difference. The headline result is straightforward — roughly half the reasoning, with effectively unchanged performance.
In-domain evals
Holdout test splits of datasets whose train splits are part of the finetuning mix — quality retention on in-distribution tasks (in contrast to the out-of-domain benchmarks above).
| Benchmark | acc (base) | acc (Ours) | Δ acc | tok (base) | tok (Ours) | matched Δ% | looping (base→Ours) |
|---|---|---|---|---|---|---|---|
| gsm8k | 93.3% ±1.5 | 96.5% ±0.3 | +3.2pp | 3,175 | 648 | -74.1% | 0.0% → 0.0% |
| arc_challenge | 97.0% ±0.3 | 97.6% ±0.4 | +0.6pp | 966 | 335 | -51.5% | 0.0% → 0.0% |
| arc_easy | 99.3% ±0.2 | 99.4% ±0.2 | +0.1pp | 566 | 260 | -44.5% | 0.0% → 0.0% |
| commonsense_qa | 86.7% ±0.7 | 88.2% ±0.9 | +1.5pp | 1,118 | 273 | -64.1% | 0.0% → 0.0% |
| openbookqa | 96.0% ±0.5 | 96.7% ±0.6 | +0.7pp | 858 | 248 | -59.5% | 0.0% → 0.0% |
| qasc | 91.7% ±0.7 | 92.2% ±0.5 | +0.5pp | 1,258 | 348 | -61.9% | 0.0% → 0.0% |
| sciq | 97.0% ±0.2 | 97.5% ±0.2 | +0.5pp | 766 | 276 | -48.3% | 0.0% → 0.0% |
| macro (mean) | +1.0pp | -57.7% |
Settings. 5 seeds per condition; thinking on; mean ± 95% CI across seeds. Decoding: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0 (bottlecapai/Qwen3.6-27B-Efficient uses the base’s sampling). Max generation tokens: 15,000 for GSM8K; 8,192 for the MCQ sets. Data: GSM8K is the full 1,319-row test split; the MCQ sets are capped at 1,000 rows (OpenBookQA = 500 and QASC = 926 are smaller, so full). acc — exact-match on the final answer (GSM8K) / last-letter multiple-choice match (MCQ); matched Δ% and looping% are as defined for the token-efficiency table above.
With focused training on holdout portions of the target datasets, we achieve a -57.7% reduction in reasoning tokens with a 1.0% increase in accuracy.
Production matters more than benchmarks
Unlimited token budgets are not how language models are actually deployed. Real systems operate under constraints: latency targets, context limits, cost ceilings, and throughput requirements. Under those conditions, efficiency matters.
A model that reaches an answer in 2,000 tokens will often outperform one that reaches the same answer in 5,000. Reasoning compression therefore becomes more valuable as deployment scale increases. Our model spends its token budget solving the problem rather than narrating the solution.
A very welcome bonus: more to the point answers
Initially, we aimed to keep the answer style and quality completely unchanged, so that the users would never realise they are using a different model without examining the thinking trace.
However, we accidentally applied our shortening also to the text of the answer after the </think> token. We have since fixed it and trained versions that preserve the original answer length.
In internal testing though, we all loved the original model. The style of the answer was still there, but it was much faster and easier to read and generate, and even felt more human — perhaps because humans naturally have a ‘penalty’ for long answers by getting tired writing long texts, something which we introduced into LLMs as well.
We therefore decided to publish this concise model, and save the other checkpoints for future releases.
Download the model, play with it, break it, let us know feedback. And thanks for being here with us. It’s not always about the latest model. It’s also about making the existing models work better for you. That’s what BottleCap’s here for.
Contact us
We are open to feedback, questions, requests, and business partnerships. Please contact us at hey@bottlecapai.com.
