Valid JSON returned, naive vs. optimized prompt (2 models)
A KDnuggets piece claimed five specific prompt changes measurably fix LLM output, demonstrated against a deliberately messy meeting transcript with a reassigned task, a merged task, and a task that's supposed to stay unassigned. I didn't take that on faith. I ran its exact test, its exact transcript, on two models already on my own machine.
The article's core claims: a naive "list the action items" prompt produces fluent prose your code can't parse, and it's specifically bad at three things: tracking an assignment that changes mid-conversation, merging a task folded into an earlier one, and correctly leaving a task unassigned rather than guessing an owner for it. A role-and-schema prompt is supposed to fix all three. I wanted to know if that held up outside the article's own demo, on models I actually run.
The setup
Same transcript as the source article: Priya, Tom and Jake in a standup. The mobile-layout review gets assigned to Priya, then reassigned to Jake two lines later. A tablet-breakpoint check gets folded into that same review rather than becoming its own item. And a 40-ticket support backlog needs triage, but the transcript explicitly leaves the owner undecided: Priya says she'll pick someone later, once she checks who's free.
I ran two prompts against qwen3:14b and gpt-oss:20b, both at temperature 0 so results are reproducible:
- Naive: "Extract the action items from this meeting transcript," transcript pasted below it, nothing else.
- Optimized: the article's own layered version: executive-assistant role, an explicit instruction to trace the final owner before answering, three corrective rules (final-owner-wins, merge-don't-duplicate, unassigned-not-guessed), and a required JSON schema.
Results
| Model | Prompt | Valid JSON | Support-queue owner | Runtime |
|---|---|---|---|---|
| qwen3:14b | Naive | No -- numbered prose | Correctly left to Priya to assign later | 29.1s |
| qwen3:14b | Optimized | Yes -- 3 clean objects | "unassigned" | 36.2s |
| gpt-oss:20b | Naive | No -- markdown table | Fabricated: assigned to Priya | 34.4s |
| gpt-oss:20b | Optimized | Yes -- 3 clean objects | "unassigned" | 65.2s |
Key finding
The JSON claim held up completely: 0 of 2 naive responses parsed as JSON, 2 of 2 optimized responses did, on the first try, no retries. But the more interesting result was gpt-oss:20b's naive run, which failed to parse and fabricated an owner outright. The transcript has Priya explicitly saying she hasn't decided who'll triage the support queue yet. The naive prompt's output confidently assigned that ticket to "Priya" anyway, in a clean-looking markdown table that would pass a quick read. That's the exact failure mode the source article warns about: a plausible-looking answer that's wrong in a way a glance won't catch. The optimized prompt, on the same model, correctly returned "unassigned" for that item.
Where it didn't need to work as hard
Both models actually got the reassigned mobile-review owner right (Jake, not Priya) even on the naive prompt, and both correctly folded the tablet-breakpoint check into that same item rather than duplicating it. The article's role-and-reasoning framing is aimed at exactly this kind of ambiguity, but on these two specific models it turned out not to be load-bearing for those two details. Both models got there anyway. The value I actually measured was concentrated in two places: JSON validity, and the unassigned-item guardrail. That's a narrower claim than "five strategies fix everything," and it's the one my own numbers support.
Caveats
Two models, one transcript, one run each at temperature 0. This isn't a sweep, and a different transcript or a higher temperature could shift which failure modes show up. I didn't test few-shot selection or the automated iterative-optimization strategy from the source article, only structured output, role-assignment, and the reasoning instruction bundled into one "optimized" prompt rather than isolating each one. That means I can say the bundle works; I can't yet say which single piece of it is doing the work, or whether it would hold on a task where the naive prompt already got the tricky parts right.
Sources
- Article being tested: KDnuggets
- qwen3:14b: Ollama library
- gpt-oss:20b: Ollama library