mager-bench
← all challenges

debug

Find and fix 3 bugs in broken Python code — tests careful reading and correctness reasoning

the prompt

$ cat debug.prompt
The following Python function is supposed to return the top-N most frequent words in a string, sorted by frequency descending and alphabetically for ties. It has exactly 3 bugs. Find each bug, explain what it does wrong, fix it, and show the corrected function.

```python
from collections import Counter

def top_words(text: str, n: int = 10) -> list[str]:
    words = text.lower().split()
    words = [w.strip('.,!?;:\'"') for w in words]
    counts = Counter(words)
    ranked = sorted(counts.items(), key=lambda x: (-x[1], x[0]))
    return [word for word, count in ranked[:n] if count > 1]
```

Test cases that expose the bugs:
- `top_words('the cat sat on the mat the', 3)` should return `['the', 'cat', 'mat']`
  (3 most frequent; cat/mat/sat tie at 1 occurrence — alphabetical breaks the tie)
- `top_words('a a b b c', 2)` should return `['a', 'b']`
- `top_words('hello world', 5)` should return `['hello', 'world']`

For each bug: (1) quote the buggy line, (2) explain what it does wrong, (3) show the fix.

rubric

correctness

Is the `count > 1` filter identified as a bug (drops valid hapax words)? Is the empty-string entry problem identified (stripping punctuation-only tokens creates '' in counter)? Is a third genuine defect found (e.g. no deduplication of empty strings, or missing n=0 guard)? Does the corrected function pass all three test cases?

quality

Are fixes minimal (not a full rewrite)? Does the model explain WHY each line is wrong, not just show a diff? Is the corrected function clean Python?

documentation

Is each bug explained clearly enough that a junior dev would understand? Are code quotes used to anchor each explanation? Is the structure (bug → explanation → fix) followed for each of the 3?

results

GPT-OSS 120B

6.7
correctness 6.0quality 7.0documentation 7.05210ms

# Correctly diagnoses the count>1 filter and empty-string-after-strip issues, and the fixed function passes all three tests, but the claimed 'third bug' (incomplete punctuation set) is really the same underlying issue as bug #2 rather than a distinct third defect (e.g., dedup or n<=0 guard), so only two truly independent bugs are identified despite the exercise requiring three.

inspect full trace →

Gemini 2.5 Flash

6.7
correctness 6.0quality 6.0documentation 8.032164ms

# Correctly identifies the count>1 filter bug and passes all three test cases, but mischaracterizes bug 2 (frames it as general tokenization robustness rather than the actual empty-string bug from punctuation-only tokens) and invents an unrelated third bug (negative n handling) instead of addressing the real empty-string/dedup issue, resulting in fixes that are more than minimal (full regex rewrite) rather than surgical patches.

inspect full trace →

Claude Sonnet 4.6

5.3
correctness 6.0quality 5.0documentation 5.024025ms

# patched after judge fix

inspect full trace →

Llama 3.3 70B

5.0
correctness 6.0quality 4.0documentation 5.01842ms

# Correctly identifies and fixes the count>1 filter bug (verified against all three test cases), but the other two 'bugs' are really the same generic complaint about punctuation-stripping rather than the intended distinct defects (e.g., empty-string entries in the counter, missing n=0 guard); the fix works but is a broad rewrite (regex tokenization) rather than a minimal targeted patch, and explanations for bugs 2/3 are vague and redundant rather than precisely diagnosing the actual failure mode.

inspect full trace →

Claude Haiku 4.5

4.7
correctness 5.0quality 5.0documentation 4.07262ms

# Correctly finds the count>1 filter bug and the empty-string bug, but 'Bug 2' is not a genuine separate defect—it's the same line and same fix as Bug 1, and the explanation even contradicts itself by admitting the fix collapses back to Bug 1's fix; a real third bug (e.g., n<=0 edge case) was never found, so the function passes tests but the bug analysis is padded and confusing rather than truly three distinct defects.

inspect full trace →