mager-bench
← all challenges

async-fetch

Write async Python for concurrent HTTP fetching with timeout and retry

the prompt

$ cat async-fetch.prompt
Write a Python async function `fetch_all(urls: list[str], timeout: float = 5.0, retries: int = 2) -> list[dict]` that fetches all URLs concurrently using `aiohttp` and returns a list of result dicts.

Each result dict must have:
  - `url`: the original URL
  - `status`: HTTP status code (int), or None on error
  - `body`: response text (str), or None on error
  - `error`: error message string, or None on success

Requirements:
1. All URLs must be fetched concurrently (not sequentially)
2. Each fetch must respect the `timeout` (per-request, not total)
3. Retry up to `retries` times on network errors or 5xx responses (exponential backoff: 0.5s, 1s, ...)
4. Never raise — errors go into the result dict
5. Type-hint everything; add a docstring
6. Show a `__main__` block that fetches 3 URLs and prints results

rubric

correctness

Is `asyncio.gather` (or equivalent) used for actual concurrency? Is timeout applied per-request via `aiohttp.ClientTimeout`? Does retry logic actually retry on 5xx and network errors (not just any exception)? Is exponential backoff implemented (not fixed sleep)? Does the function never raise on individual URL errors?

quality

Is the session created once (not per-request)? Is the retry loop clean (not deeply nested)? Are asyncio patterns idiomatic (async with, await)? Is the main block runnable with `asyncio.run`?

documentation

Does the docstring explain all parameters including units (timeout in seconds)? Are result dict keys documented? Is the backoff behavior described?

results

Claude Sonnet 4.6

8.3
correctness 8.0quality 8.0documentation 9.029212ms

# Solid implementation with proper concurrency, per-request timeout, exponential backoff, and never-raise behavior, though the response is truncated mid-explanation and uses a slightly odd backoff scheme (0s on first attempt) plus TCPConnector(limit=0) which removes any concurrency control.'

inspect full trace →

Claude Haiku 4.5

7.3
correctness 7.0quality 7.0documentation 8.09035ms

# Solid overall (proper session reuse, per-request ClientTimeout, asyncio.gather, error containment) but backoff is linear (0.5*(n+1)) rather than truly exponential, and persistent 5xx failures after retries exhaust silently return status with error=None; retry logic is also duplicated across multiple except blocks instead of being unified.

inspect full trace →

Llama 3.3 70B

5.7
correctness 4.0quality 6.0documentation 7.01760ms

# Critical bug: timeouts raise asyncio.TimeoutError which is not caught by the except aiohttp.ClientError clause, and gather() lacks return_exceptions=True, so the function can still raise on timeout, violating the 'never raise' requirement; also timeout is passed as a raw float rather than an explicit aiohttp.ClientTimeout object as the rubric expects, though backoff and session reuse are implemented correctly.

inspect full trace →

Gemini 2.5 Flash

1.3
correctness 0.0quality 1.0documentation 3.038391ms

# Response is truncated/incomplete code (cuts off mid-line at 'results = a'), so it doesn't even run, let alone satisfy requirements like gather, main block, or returning final results.

inspect full trace →