debug

Find and fix 3 bugs in broken Python code — tests careful reading and correctness reasoning

the prompt

$ cat debug.prompt

The following Python function is supposed to return the top-N most frequent words in a string, sorted by frequency descending and alphabetically for ties. It has exactly 3 bugs. Find each bug, explain what it does wrong, fix it, and show the corrected function.

```python
from collections import Counter

def top_words(text: str, n: int = 10) -> list[str]:
    words = text.lower().split()
    words = [w.strip('.,!?;:\'"') for w in words]
    counts = Counter(words)
    ranked = sorted(counts.items(), key=lambda x: (-x[1], x[0]))
    return [word for word, count in ranked[:n] if count > 1]
```

Test cases that expose the bugs:
- `top_words('the cat sat on the mat the', 3)` should return `['the', 'cat', 'mat']`
  (3 most frequent; cat/mat/sat tie at 1 occurrence — alphabetical breaks the tie)
- `top_words('a a b b c', 2)` should return `['a', 'b']`
- `top_words('hello world', 5)` should return `['hello', 'world']`

For each bug: (1) quote the buggy line, (2) explain what it does wrong, (3) show the fix.

rubric

correctness

Is the `count > 1` filter identified as a bug (drops valid hapax words)? Is the empty-string entry problem identified (stripping punctuation-only tokens creates '' in counter)? Is a third genuine defect found (e.g. no deduplication of empty strings, or missing n=0 guard)? Does the corrected function pass all three test cases?

quality

Are fixes minimal (not a full rewrite)? Does the model explain WHY each line is wrong, not just show a diff? Is the corrected function clean Python?

documentation

Is each bug explained clearly enough that a junior dev would understand? Are code quotes used to anchor each explanation? Is the structure (bug → explanation → fix) followed for each of the 3?

results

Gemini 2.5 Flash

6.7

correctness 6.0quality 6.0documentation 8.032164ms

# Correctly identifies the count>1 filter bug and passes all three test cases, but mischaracterizes bug 2 (frames it as general tokenization robustness rather than the actual empty-string bug from punctuation-only tokens) and invents an unrelated third bug (negative n handling) instead of addressing the real empty-string/dedup issue, resulting in fixes that are more than minimal (full regex rewrite) rather than surgical patches.

inspect full trace →

Claude Haiku 4.5

5.7

correctness 5.0quality 6.0documentation 6.06280ms

# Correctly identifies the count>1 filter and empty-string bugs and passes all test cases, but 'Bug 3' is not a distinct third defect—it's the same buggy line and fix as Bug 1 restated, so the model effectively only found 2 genuine bugs despite the task requiring 3.

inspect full trace →

Claude Sonnet 4.6

5.3

correctness 6.0quality 5.0documentation 5.024025ms

# patched after judge fix

inspect full trace →

Llama 3.3 70B

5.0

correctness 6.0quality 4.0documentation 5.01842ms

# Correctly identifies and fixes the count>1 filter bug (verified against all three test cases), but the other two 'bugs' are really the same generic complaint about punctuation-stripping rather than the intended distinct defects (e.g., empty-string entries in the counter, missing n=0 guard); the fix works but is a broad rewrite (regex tokenization) rather than a minimal targeted patch, and explanations for bugs 2/3 are vague and redundant rather than precisely diagnosing the actual failure mode.

inspect full trace →