mager-bench
← all challenges

readme-writer

Write a README for a CLI tool — tests documentation ability directly

the prompt

$ cat readme-writer.prompt
Write a README.md for a command-line tool called `snapdiff` that compares two directories and reports added, removed, and changed files. The tool is written in Python, installable via pip. Include: a one-line description, installation instructions, usage examples with flags (--ignore-hidden, --output json|text, --depth N), output format explanation, and a short 'How it works' section. Use proper Markdown formatting.

rubric

correctness

Does the README cover all the required sections? Are the flags documented correctly?

quality

Is the README well-structured and scannable? Would a new user understand the tool from this alone?

documentation

Is the Markdown formatting correct? Are code blocks used for commands? Is the writing clear and concise?

results

GPT-OSS 120B

9.0
correctness 9.0quality 9.0documentation 9.032506ms

# Comprehensive, well-organized README covering all required sections with correctly documented flags, clear examples, and proper Markdown formatting, though some details (e.g., version, license) are invented without task basis.

inspect full trace →

Claude Sonnet 4.6

9.0
correctness 9.0quality 9.0documentation 9.026268ms

# Comprehensive, well-organized README covering all required sections with correct flag documentation, clear examples, and proper Markdown/code-block usage, though it invents extra details (e.g., MD5, placeholder repo links) not specified in the prompt.

inspect full trace →

Claude Haiku 4.5

8.7
correctness 8.0quality 9.0documentation 9.09373ms

# Thorough, well-formatted README covering all required sections with clear examples, though depth flag semantics (-1 for unlimited vs default unlimited) are slightly confusing and some details are fabricated/unverifiable.

inspect full trace →

Gemini 2.5 Flash

8.0
correctness 8.0quality 8.0documentation 8.012093ms

# All required sections and flags are covered clearly and correctly formatted, though depth default (-1) and hashing algorithm ambiguity feel slightly invented/inconsistent.

inspect full trace →

Llama 3.3 70B

6.0
correctness 6.0quality 6.0documentation 6.01290ms

# Covers all required sections and flags but output format section lacks concrete example output, missing sample JSON/text output blocks and a title-only H1 without top-level structure like a Table of Contents or License section.

inspect full trace →