mager-bench
← all challenges

refactor

Refactor messy code and explain each change

the prompt

$ cat refactor.prompt
Refactor the following Python function and explain each change you made:

```python
def p(d):
    r = []
    for i in range(len(d)):
        x = d[i]
        if x % 2 == 0:
            r.append(x * x)
        else:
            r.append(x * x * x)
    return r
```

Requirements: rename everything meaningfully, use a list comprehension or equivalent, add a docstring, add type hints. Then write a short explanation of each change.

rubric

correctness

Does the refactored function produce the same output as the original?

quality

Are the names meaningful? Is the list comprehension idiomatic? Is it more readable?

documentation

Is the docstring clear? Is the explanation of changes clear and specific (not generic)?

results

Claude Haiku 4.5

9.0
correctness 10.0quality 9.0documentation 8.04800ms

# Correct, idiomatic refactor with meaningful names, type hints, and a clear docstring/example; the explanation is specific but includes an unsubstantiated performance claim (10-15% speedup).

inspect full trace →

Claude Sonnet 4.6

9.0
correctness 9.0quality 9.0documentation 9.015452ms

# Accurate, well-documented refactor with meaningful names, idiomatic comprehension, correct type hints, and specific, non-generic explanations for each change.

inspect full trace →

Gemini 2.5 Flash

9.0
correctness 10.0quality 9.0documentation 8.09695ms

# Correct, well-named refactor with idiomatic comprehension and a thorough docstring with examples, but the final explanation point (type hints) is cut off mid-sentence, leaving the response incomplete.

inspect full trace →

Llama 3.3 70B

8.7
correctness 10.0quality 8.0documentation 8.01944ms

# Correct and clean refactor with good docstring and type hints, though the loop variable 'x' remains unrenamed despite the explanation claiming it was renamed, and some explanation points are somewhat generic.

inspect full trace →