← Back to Atlas Chart

Three Local Models, One Bug: Two Fixed It, One Never Answered

AI / Benchmarks

Same bug, same instructions, three models. Bars animate in on scroll.

Three local coding models got the same buggy function and the same instructions. Two fixed it correctly in under a minute. The third burned through 16,000 tokens of reasoning and never produced an answer at all.

This follows an easier test from last month that gave three local models one planted bug and watched all three pass, a clean sweep with no real story in it. This time the function had two independent bugs, not one, checked against 9 test cases built to catch a partial fix.

Results

Model Result Time
gpt-oss:20b 9/9, both bugs fixed 32.7s
Gemma 4 12B 9/9, both bugs fixed 42.2s
qwen3:14b, reasoning on 0/9, no answer produced 223s (cut off)
qwen3:14b, reasoning off 9/9, both bugs fixed 4s

Key finding

Reasoning mode broke qwen3:14b outright. With thinking enabled, the model never converged on an answer, even after 16,384 tokens and over 70,000 characters of chain-of-thought, and it ended up hallucinating that the prompt was repeated Chinese text it never was. With thinking disabled, the same model produced a fully correct fix for both bugs in 4 seconds.

Before writing this up

A model producing zero output looks like a harness bug, not a real finding, so I checked before believing it. I reran qwen3:14b alone at 16,384 tokens, four times the original budget. Same result: no answer, this time after 70,201 characters of chain-of-thought. Then I ran the identical model with reasoning turned off. Instant, correct fix, 4 seconds. That ruled out a token-budget mistake on my end and pointed straight at reasoning mode itself.

The two bugs

The function merges overlapping intervals. It had two separate defects: it never sorted the input, so unsorted intervals merged wrong, and it used a strict less-than where it needed less-than-or-equal, so touching intervals like [1,3] and [3,5] never merged into one. Nine test cases covered both, plus a few baseline sanity checks that even the buggy version already passed, so a partial fix would still show up as a partial score.

Methodology

All three models got the identical prompt: fix the bug, return only the corrected function. gpt-oss:20b and qwen3:14b support Ollama's thinking mode; Gemma 4 doesn't, so it ran as a plain chat completion. The original test used a model called gemma4-claude-lite that's no longer installed on this machine; Gemma 4 12B is the closest available substitute, and results for it should be read as directional rather than a rerun of that exact test.

Caveats

Single run per model, temperature 0, no repeated sampling. Nine test cases is enough to catch a partial fix on this specific bug, not a general debugging benchmark. One weird result from one model on one task is a data point, not a verdict on reasoning mode as a whole.

Sources