mager-bench
← all challenges

api-client

Write a small HTTP API client class with error handling and docs

the prompt

$ cat api-client.prompt
Write a Python class `APIClient` that wraps the `requests` library for a REST API. It should: (1) accept a base_url and optional api_key in __init__, (2) have a `get(path, params=None)` method and a `post(path, data)` method, (3) raise a custom `APIError` exception with the status code and message on non-2xx responses, (4) include type hints throughout, (5) have a complete docstring on the class and each public method. Show a usage example at the bottom.

rubric

correctness

Does the class work as described? Are error cases handled? Are type hints correct?

quality

Is the code clean and well-structured? Is APIError a proper Exception subclass? Are session/connection concerns handled?

documentation

Does the class have a docstring? Do all public methods have docstrings? Is the usage example runnable?

results

Llama 3.3 70B

7.0
correctness 7.0quality 6.0documentation 8.01859ms

# Solid, working implementation with correct type hints and proper APIError subclass, but doesn't use requests.Session for connection reuse, uses naive string concatenation for URLs instead of urljoin, and lacks timeout/error handling for non-JSON responses.

inspect full trace →

Claude Haiku 4.5

5.0
correctness 4.0quality 6.0documentation 5.010540ms

# Well-structured and thoroughly documented up to the point of truncation, but the response cuts off mid-docstring in the post method, omitting its body and the required usage example entirely, and it never uses a requests.Session for connection reuse.

inspect full trace →

Gemini 2.5 Flash

5.0
correctness 3.0quality 6.0documentation 6.019050ms

# Core class logic is solid and well-typed, but the usage example is cut off mid-string leaving the code non-runnable and incomplete.

inspect full trace →

Claude Sonnet 4.6

3.0
correctness 1.0quality 3.0documentation 5.027637ms

# Response is truncated mid-method, missing the post() method entirely and the usage example, so the code doesn't even run as-is despite otherwise clean structure and good docstrings for what exists.

inspect full trace →