← Back to Atlas Chart

Fact-Checking a Viral AI YouTuber's GPU Advice

AI / Benchmarks

Benchmark results comparison. Bars animate in on scroll.

A popular AI YouTuber recommended a specific model for 16GB VRAM cards. I tested it against a model already on my disk. His pick lost: 8 points lower on a real coding benchmark, and three times slower, on identical hardware.

The guide's recommendation was Qwen3.8-27B, squeezed down to fit 16GB of VRAM. The model that beat it, gpt-oss:20b, was already installed on my RTX 5070 Ti. Both ran through HumanEval+: 164 coding problems, executed and graded automatically, no cherry-picking.

Results

Model Pass@1 Speed
gpt-oss:20b 90.9% (149/164) 151 tokens/sec
Qwen3.8-27B, reasoning off 84.8% (139/164) 49 tokens/sec
Qwen3.8-27B, xhigh reasoning forced on 82.9% (136/164) 48 tokens/sec

Key finding

Forcing the "smarter" model to reason harder made it score worse, not better. With full reasoning engaged (about 4,200 characters of chain-of-thought per problem, roughly 70 minutes of total runtime), Qwen3.8-27B scored lower than with reasoning silently disabled. On problems simple enough not to need deep thought, extended reasoning let the model second-guess a correct first answer into a wrong edge case.

Methodology

  • gpt-oss:20b: already installed, MXFP4 quantization, sparse MoE with ~3.6B active parameters
  • Qwen3.8-27B: the video's own recommended family, Unsloth 3-bit dynamic quant, dense 27B parameters

Two bugs I caught along the way

The numbers above aren't the first ones I got. Early in this benchmark, my code-extraction script mis-indented completions that included a nested helper function, turning valid code into a syntax error. I caught it by reading the raw failures instead of trusting the pass rate, fixed the indentation logic, and rescored the same cached model outputs without spending a single extra model call. That fix moved one model's score up several points.

Later, testing Qwen3.8-27B with reasoning enabled, my harness reported 0% visible thinking across all 164 problems. That's not what reasoning looks like. I'd been calling Ollama's plain completion endpoint, which never engages a model's chat-template reasoning logic. Switching to the chat endpoint with an explicit system prompt fixed it, and the reasoning-on number in the table above only exists because I checked.

Neither bug changed which model won. Both are why I don't fully trust a benchmark number, including my own, until I've read a sample of what actually failed.

Caveats

This isn't a perfect test. HumanEval and HumanEval+ are old enough that any 2026-era model has likely seen these exact problems during training. That inflates every score here somewhat, mine included. I tested one quantization per model, not a sweep. It's a single run at temperature 0, pass@1 only, no repeated sampling. Take the ranking as directional, not final.

Sources