mager-bench

Baseline correctness + code style

# Correct, clean, well-documented solution with extra ValueError handling not required by the spec but reasonable; docstring example formatting (wrapped list) is slightly misleading vs actual doctest output but harmless since not run as doctest.

Algorithm + full docstring (Args/Returns/Raises + examples)

# Correct, overflow-safe binary search with clean logic, thorough docstring/comments, and good extra test coverage beyond the required 3 examples, though duplicates aren't explicitly tested.

Code clarity + whether the model can explain its changes

# Correct, idiomatic refactor with meaningful names, type hints, and a clear docstring/example; the explanation is specific but includes an unsubstantiated performance claim (10-15% speedup).

Pure documentation ability — no code at all

# Comprehensive, well-organized README covering all required sections with clear examples, though some invented details (hash/line-count logic, placeholder GitHub URL) go beyond the stated scope.

Edge-case thinking + pytest parametrize discipline

# Comprehensive parametrized happy-path and error coverage with correct arithmetic, but the response cuts off mid-test with an unfinished/no-op assertion, and uses classes despite the 'plain functions' requirement, slightly hurting quality.

Idiomatic Go table-driven tests + benchmark

# Comprehensive table-driven tests with correct map comparison via maps.Equal and a proper benchmark loop, but lacks a doc comment on TestWordCount and makes some debatable assumptions about punctuation/apostrophe handling.

Async concurrency patterns + retry/timeout handling

# Solid overall (proper session reuse, per-request ClientTimeout, asyncio.gather, error containment) but backoff is linear (0.5*(n+1)) rather than truly exponential, and persistent 5xx failures after retries exhaust silently return status with error=None; retry logic is also duplicated across multiple except blocks instead of being unified.

CTE + window function fluency on a real schema

# Logic for revenue, avg, and top_customer (via DISTINCT ON) is correct, but revenue_pct is computed with a CROSS JOIN CTE rather than an actual window function (SUM(...) OVER()), directly violating an explicit requirement despite the model's rationalization; comments and explanations are clear and well-organized otherwise.

ExUnit describe blocks + assert_raise + unicode handling

# patched after judge fix

Careful reading + correctness reasoning over broken code

# Correctly identifies the count>1 filter and empty-string bugs and passes all test cases, but 'Bug 3' is not a distinct third defect—it's the same buggy line and fix as Bug 1 restated, so the model effectively only found 2 genuine bugs despite the task requiring 3.

Class design + error handling + type hints + docs

# Well-structured and thoroughly documented up to the point of truncation, but the response cuts off mid-docstring in the post method, omitting its body and the required usage example entirely, and it never uses a requests.Session for connection reuse.

Vegas slot machine — reels, pay table, betting, win animations

# judge: llama-3.3-70b

DDA raycaster + textures + door + minimap + Z-buffer — the signature challenge

# Response is severely truncated mid-function, leaving out nearly all rendering, player, HUD, and code logic — completely unusable and unscoreable as a working implementation.

Baseline correctness + code style

# Correct, clean, idiomatic implementation with thorough docstring, doctest-style examples, sensible input validation, and helpful usage demos, though extra validation exceeds task scope slightly.

Pure documentation ability — no code at all

# Comprehensive, well-organized README covering all required sections with correct flag documentation, clear examples, and proper Markdown/code-block usage, though it invents extra details (e.g., MD5, placeholder repo links) not specified in the prompt.

Code clarity + whether the model can explain its changes

# Accurate, well-documented refactor with meaningful names, idiomatic comprehension, correct type hints, and specific, non-generic explanations for each change.

Algorithm + full docstring (Args/Returns/Raises + examples)

# Correct O(log n) binary search with overflow-safe mid calculation and solid edge-case handling, though the added O(n) sortedness check slightly undermines efficiency and the response ends with a truncated 'sample output' section; documentation is thorough with accurate docstring examples and meaningful comments.

Async concurrency patterns + retry/timeout handling

# Solid implementation with proper concurrency, per-request timeout, exponential backoff, and never-raise behavior, though the response is truncated mid-explanation and uses a slightly odd backoff scheme (0s on first attempt) plus TCPConnector(limit=0) which removes any concurrency control.'

ExUnit describe blocks + assert_raise + unicode handling

# patched after judge fix

CTE + window function fluency on a real schema

# patched after judge fix

Edge-case thinking + pytest parametrize discipline

# Well-designed, thorough parametrized happy-path and error tests with accurate arithmetic and clear naming, but the response is truncated mid-function ('with pyt'), producing a syntax error that would prevent the test suite from even being collected.

Careful reading + correctness reasoning over broken code

# patched after judge fix

Idiomatic Go table-driven tests + benchmark

# Test cases and benchmark are well-designed and exceed the requirement, but the file incorrectly declares package wordcount_test and reimplements a stub WordCount that always returns nil, meaning the delivered file will fail most subtests unless manually edited to remove the stub and fix the package, contradicting the stated 'package: wordcount' requirement.

Class design + error handling + type hints + docs

# Response is truncated mid-method, missing the post() method entirely and the usage example, so the code doesn't even run as-is despite otherwise clean structure and good docstrings for what exists.

Vegas slot machine — reels, pay table, betting, win animations

# judge: llama-3.3-70b

DDA raycaster + textures + door + minimap + Z-buffer — the signature challenge

#

Baseline correctness + code style

# Correct, clean implementation with clear docstring, doctest example, and additional demo block.

Code clarity + whether the model can explain its changes

# Correct and clean refactor with good docstring and type hints, though the loop variable 'x' remains unrenamed despite the explanation claiming it was renamed, and some explanation points are somewhat generic.

Algorithm + full docstring (Args/Returns/Raises + examples)

# Core binary search logic is correct, but raising ValueError for empty list deviates from the spec's expectation of returning -1, and mid-calculation doesn't address overflow-safe alternative despite rubric interest; comments are somewhat redundant but docstring and examples are otherwise complete and accurate.

Class design + error handling + type hints + docs

# Solid, working implementation with correct type hints and proper APIError subclass, but doesn't use requests.Session for connection reuse, uses naive string concatenation for URLs instead of urljoin, and lacks timeout/error handling for non-JSON responses.

Edge-case thinking + pytest parametrize discipline

# Covers h, m, s, hm, hms and a zero-case, but omits the standalone 'ms' format explicitly required by the rubric; error cases are adequate (empty, invalid char, garbage string) though lack tests for malformed ordering; whitespace test is a nice touch but generic test names and lack of parametrize ids weaken documentation quality.

Pure documentation ability — no code at all

# Covers all required sections and flags but output format section lacks concrete example output, missing sample JSON/text output blocks and a title-only H1 without top-level structure like a Table of Contents or License section.

ExUnit describe blocks + assert_raise + unicode handling

# Good coverage of required scenarios and correct assert_raise usage, but the custom-suffix expected value ('he~') is inconsistent with the truncation formula implied by the default-suffix test ('he...'), suggesting a copy-paste error, and the zero-length/negative-length assumptions are unverified given no reference implementation; the 'Examples' section is written as plain comments rather than a proper @moduledoc, weakening the documentation requirement.

Async concurrency patterns + retry/timeout handling

# Critical bug: timeouts raise asyncio.TimeoutError which is not caught by the except aiohttp.ClientError clause, and gather() lacks return_exceptions=True, so the function can still raise on timeout, violating the 'never raise' requirement; also timeout is passed as a raw float rather than an explicit aiohttp.ClientTimeout object as the rubric expects, though backoff and session reuse are implemented correctly.

Careful reading + correctness reasoning over broken code

# Correctly identifies and fixes the count>1 filter bug (verified against all three test cases), but the other two 'bugs' are really the same generic complaint about punctuation-stripping rather than the intended distinct defects (e.g., empty-string entries in the counter, missing n=0 guard); the fix works but is a broad rewrite (regex tokenization) rather than a minimal targeted patch, and explanations for bugs 2/3 are vague and redundant rather than precisely diagnosing the actual failure mode.

Idiomatic Go table-driven tests + benchmark

# Test logic and case coverage are solid and benchmark is correct, but the code contains a fatal compile error (`t *test.T` instead of `*testing.T`) that would prevent the file from building; also lacks a dedicated comment on TestWordCount explaining its coverage.

CTE + window function fluency on a real schema

# The country_sales CTE groups by (country, name), so downstream MAX() aggregation in country_data yields the top customer's revenue/order_count/avg instead of the country's true totals (correct only when a country has a single customer), and revenue_pct is computed via a CROSS JOIN scalar CTE rather than a window function as required; despite clear comments and a single final SELECT, the core aggregation logic is fundamentally broken.

DDA raycaster + textures + door + minimap + Z-buffer — the signature challenge

# The response is truncated mid-code, leaving the file syntactically invalid and missing nearly all required features (textures, HUD, minimap, input, game loop, door/exit logic, Z-buffer usage).

Vegas slot machine — reels, pay table, betting, win animations

# Response is truncated mid-function and non-functional, missing animateReels, updateUI, game-over logic, count-up animation, and proper win handling entirely.

Baseline correctness + code style

# Correct, clean, well-documented implementation with thorough examples and sensible edge-case handling, only minor nit being the added ValueError not explicitly required by the task.

Code clarity + whether the model can explain its changes

# Correct, well-named refactor with idiomatic comprehension and a thorough docstring with examples, but the final explanation point (type hints) is cut off mid-sentence, leaving the response incomplete.

CTE + window function fluency on a real schema

# Logic is correct and uses ROW_NUMBER + window function for revenue_pct appropriately, but recomputing country aggregates via a second orders/customers join (instead of deriving from customer_spend) is slightly redundant.

Algorithm + full docstring (Args/Returns/Raises + examples)

# Algorithm is correct and handles edge cases well, but Raises section is empty (task requested a Raises section, ideally documenting behavior like non-sorted input or type errors), and comments are somewhat verbose/restating code in places.

Pure documentation ability — no code at all

# All required sections and flags are covered clearly and correctly formatted, though depth default (-1) and hashing algorithm ambiguity feel slightly invented/inconsistent.

Careful reading + correctness reasoning over broken code

# Correctly identifies the count>1 filter bug and passes all three test cases, but mischaracterizes bug 2 (frames it as general tokenization robustness rather than the actual empty-string bug from punctuation-only tokens) and invents an unrelated third bug (negative n handling) instead of addressing the real empty-string/dedup issue, resulting in fixes that are more than minimal (full regex rewrite) rather than surgical patches.

ExUnit describe blocks + assert_raise + unicode handling

# Covers all required scenarios and uses assert_raise/describe blocks well, but contains an internal inconsistency (empty string + custom suffix test contradicts its own assumed implementation) and the doctest Examples section has several numerically incorrect examples (e.g. 'hello world',7 => 'hello...' exceeds max_len), undermining correctness and documentation accuracy despite good structural style.

Class design + error handling + type hints + docs

# Core class logic is solid and well-typed, but the usage example is cut off mid-string leaving the code non-runnable and incomplete.

Edge-case thinking + pytest parametrize discipline

# Good coverage of formats and edge cases with meaningful parametrize IDs and docstrings, but the response is cut off mid-function with an unterminated docstring, leaving invalid Python that would fail to even import/run.

Idiomatic Go table-driven tests + benchmark

# The test file will not compile because 'reflect' is imported but never actually referenced (only mentioned in a comment), causing a Go compile error; additionally the apostrophe-handling test case ('It's') mismatches the model's own regex-based implementation output, meaning even after fixing the import the test would fail — despite good table-driven structure, many descriptive test names, and a solid benchmark.

Async concurrency patterns + retry/timeout handling

# Response is truncated/incomplete code (cuts off mid-line at 'results = a'), so it doesn't even run, let alone satisfy requirements like gather, main block, or returning final results.

DDA raycaster + textures + door + minimap + Z-buffer — the signature challenge

# Response is cut off almost immediately in the CSS, containing no actual JavaScript implementation or functionality at all.

Vegas slot machine — reels, pay table, betting, win animations

# Response is truncated mid-CSS and contains no JavaScript or functional game logic at all, making it completely non-functional.