mager-bench
← all challenges

api-client

Write a small HTTP API client class with error handling and docs

the prompt

$ cat api-client.prompt
Write a Python class `APIClient` that wraps the `requests` library for a REST API. It should: (1) accept a base_url and optional api_key in __init__, (2) have a `get(path, params=None)` method and a `post(path, data)` method, (3) raise a custom `APIError` exception with the status code and message on non-2xx responses, (4) include type hints throughout, (5) have a complete docstring on the class and each public method. Show a usage example at the bottom.

rubric

correctness

Does the class work as described? Are error cases handled? Are type hints correct?

quality

Is the code clean and well-structured? Is APIError a proper Exception subclass? Are session/connection concerns handled?

documentation

Does the class have a docstring? Do all public methods have docstrings? Is the usage example runnable?

results

Llama 3.3 70B

7.7
correctness 8.0quality 6.0documentation 9.01706ms

# Functionally correct with proper APIError subclass and docstrings, but lacks a requests.Session for connection reuse, has duplicated header logic, no timeout handling, and generic Dict types aren't parameterized.

inspect full trace →

GPT-OSS 120B

3.7
correctness 1.0quality 4.0documentation 6.03251ms

# Response is truncated mid-code in the usage example, making the file syntactically invalid and non-runnable despite otherwise solid class design.

inspect full trace →

Claude Sonnet 4.6

3.7
correctness 2.0quality 5.0documentation 4.037832ms

# Response is cut off mid-docstring for post(), missing the actual implementation and the required usage example, making the class non-functional and the task incomplete despite good style in the parts that exist.

inspect full trace →

Claude Haiku 4.5

1.7
correctness 0.0quality 2.0documentation 3.014221ms

# Response is truncated mid-line inside the post() method, leaving the code non-functional and missing the required usage example entirely.

inspect full trace →

Gemini 2.5 Flash

1.7
correctness 0.0quality 2.0documentation 3.035473ms

# The response is truncated mid-code (cuts off at 'if response.content: re') and never includes get/post methods or the usage example, making it non-functional and incomplete.

inspect full trace →