mager-bench
← all challenges

async-fetch

Write async Python for concurrent HTTP fetching with timeout and retry

the prompt

$ cat async-fetch.prompt
Write a Python async function `fetch_all(urls: list[str], timeout: float = 5.0, retries: int = 2) -> list[dict]` that fetches all URLs concurrently using `aiohttp` and returns a list of result dicts.

Each result dict must have:
  - `url`: the original URL
  - `status`: HTTP status code (int), or None on error
  - `body`: response text (str), or None on error
  - `error`: error message string, or None on success

Requirements:
1. All URLs must be fetched concurrently (not sequentially)
2. Each fetch must respect the `timeout` (per-request, not total)
3. Retry up to `retries` times on network errors or 5xx responses (exponential backoff: 0.5s, 1s, ...)
4. Never raise — errors go into the result dict
5. Type-hint everything; add a docstring
6. Show a `__main__` block that fetches 3 URLs and prints results

rubric

correctness

Is `asyncio.gather` (or equivalent) used for actual concurrency? Is timeout applied per-request via `aiohttp.ClientTimeout`? Does retry logic actually retry on 5xx and network errors (not just any exception)? Is exponential backoff implemented (not fixed sleep)? Does the function never raise on individual URL errors?

quality

Is the session created once (not per-request)? Is the retry loop clean (not deeply nested)? Are asyncio patterns idiomatic (async with, await)? Is the main block runnable with `asyncio.run`?

documentation

Does the docstring explain all parameters including units (timeout in seconds)? Are result dict keys documented? Is the backoff behavior described?

results

Claude Sonnet 4.6

8.3
correctness 8.0quality 8.0documentation 9.029212ms

# Solid implementation with proper concurrency, per-request timeout, exponential backoff, and never-raise behavior, though the response is truncated mid-explanation and uses a slightly odd backoff scheme (0s on first attempt) plus TCPConnector(limit=0) which removes any concurrency control.'

inspect full trace →

Claude Haiku 4.5

8.3
correctness 9.0quality 8.0documentation 8.08387ms

# Correctly implements per-request timeout via session-level ClientTimeout, real concurrency via asyncio.gather, retries with exponential backoff on 5xx/timeouts/ClientError, and never raises; minor style nit is repeated backoff/continue blocks across exception branches and the backoff behavior isn't mentioned in the top-level docstring.

inspect full trace →

GPT-OSS 120B

7.7
correctness 6.0quality 8.0documentation 9.03368ms

# Good concurrency (session reuse + asyncio.gather) and proper 5xx/network-error retry filtering with exponential backoff, but the retry-count logic has an off-by-one bug (allows one extra attempt beyond `retries`), and timeout is passed as a raw float rather than an explicit aiohttp.ClientTimeout, slightly diverging from the rubric's expectations; docstrings are thorough and the main block is correctly runnable.

inspect full trace →

Llama 3.3 70B

5.7
correctness 4.0quality 6.0documentation 7.01760ms

# Critical bug: timeouts raise asyncio.TimeoutError which is not caught by the except aiohttp.ClientError clause, and gather() lacks return_exceptions=True, so the function can still raise on timeout, violating the 'never raise' requirement; also timeout is passed as a raw float rather than an explicit aiohttp.ClientTimeout object as the rubric expects, though backoff and session reuse are implemented correctly.

inspect full trace →

Gemini 2.5 Flash

1.3
correctness 0.0quality 1.0documentation 3.038391ms

# Response is truncated/incomplete code (cuts off mid-line at 'results = a'), so it doesn't even run, let alone satisfy requirements like gather, main block, or returning final results.

inspect full trace →