mager-bench
← all challenges

readme-writer

Write a README for a CLI tool — tests documentation ability directly

the prompt

$ cat readme-writer.prompt
Write a README.md for a command-line tool called `snapdiff` that compares two directories and reports added, removed, and changed files. The tool is written in Python, installable via pip. Include: a one-line description, installation instructions, usage examples with flags (--ignore-hidden, --output json|text, --depth N), output format explanation, and a short 'How it works' section. Use proper Markdown formatting.

rubric

correctness

Does the README cover all the required sections? Are the flags documented correctly?

quality

Is the README well-structured and scannable? Would a new user understand the tool from this alone?

documentation

Is the Markdown formatting correct? Are code blocks used for commands? Is the writing clear and concise?

results

Claude Sonnet 4.6

9.0
correctness 9.0quality 9.0documentation 9.026268ms

# Comprehensive, well-organized README covering all required sections with correct flag documentation, clear examples, and proper Markdown/code-block usage, though it invents extra details (e.g., MD5, placeholder repo links) not specified in the prompt.

inspect full trace →

Claude Haiku 4.5

8.7
correctness 8.0quality 9.0documentation 9.09906ms

# Comprehensive, well-organized README covering all required sections with clear examples, though some invented details (hash/line-count logic, placeholder GitHub URL) go beyond the stated scope.

inspect full trace →

Gemini 2.5 Flash

8.0
correctness 8.0quality 8.0documentation 8.012093ms

# All required sections and flags are covered clearly and correctly formatted, though depth default (-1) and hashing algorithm ambiguity feel slightly invented/inconsistent.

inspect full trace →

Llama 3.3 70B

6.0
correctness 6.0quality 6.0documentation 6.01290ms

# Covers all required sections and flags but output format section lacks concrete example output, missing sample JSON/text output blocks and a title-only H1 without top-level structure like a Table of Contents or License section.

inspect full trace →