mager-bench
← all challenges

test-writing

Write pytest tests for a provided function — tests edge-case thinking and assertion quality

the prompt

$ cat test-writing.prompt
Write a comprehensive pytest test suite for the following Python function:

```python
def parse_duration(s: str) -> int:
    """Parse a human duration string into total seconds.

    Accepted formats: '1h', '30m', '45s', '1h30m', '2h15m30s'.
    Raises ValueError on unrecognized input.
    """
    import re
    pattern = r'^(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?$'
    m = re.fullmatch(pattern, s.strip())
    if not m or not s.strip():
        raise ValueError(f'Invalid duration: {s!r}')
    h, mi, sec = (int(x) if x else 0 for x in m.groups())
    return h * 3600 + mi * 60 + sec
```

Requirements:
- Use pytest (plain functions, no unittest classes)
- Use `@pytest.mark.parametrize` for the happy-path cases
- Cover at least 3 edge/error cases with `pytest.raises`
- Name tests descriptively so failures are self-explaining
- No mocking needed — the function is pure
- Do not reimplement the function; test it as a black box

rubric

correctness

Do the parametrized cases cover all documented formats (h, m, s, hm, hms, ms)? Are the expected values arithmetically correct? Do error cases actually trigger ValueError?

quality

Is parametrize used instead of copy-paste test functions? Are test names descriptive? Are assertions tight (not just 'assert result')? Is there any unnecessary complexity?

documentation

Do test names read like specs ('test_hours_only', 'test_invalid_raises')? Are fixture/parametrize IDs meaningful? Any comments where the tested behavior is non-obvious?

results

Claude Haiku 4.5

7.7
correctness 8.0quality 7.0documentation 8.011698ms

# Comprehensive parametrized happy-path and error coverage with correct arithmetic, but the response cuts off mid-test with an unfinished/no-op assertion, and uses classes despite the 'plain functions' requirement, slightly hurting quality.

inspect full trace →

Llama 3.3 70B

6.3
correctness 7.0quality 6.0documentation 6.01103ms

# Covers h, m, s, hm, hms and a zero-case, but omits the standalone 'ms' format explicitly required by the rubric; error cases are adequate (empty, invalid char, garbage string) though lack tests for malformed ordering; whitespace test is a nice touch but generic test names and lack of parametrize ids weaken documentation quality.

inspect full trace →

Claude Sonnet 4.6

5.3
correctness 3.0quality 5.0documentation 8.028558ms

# Well-designed, thorough parametrized happy-path and error tests with accurate arithmetic and clear naming, but the response is truncated mid-function ('with pyt'), producing a syntax error that would prevent the test suite from even being collected.

inspect full trace →

Gemini 2.5 Flash

4.3
correctness 3.0quality 4.0documentation 6.030096ms

# Good coverage of formats and edge cases with meaningful parametrize IDs and docstrings, but the response is cut off mid-function with an unterminated docstring, leaving invalid Python that would fail to even import/run.

inspect full trace →