mager-bench
← all challenges

elixir-test

Write idiomatic Elixir ExUnit tests — tests knowledge of Elixir testing conventions

the prompt

$ cat elixir-test.prompt
Write an ExUnit test module for the following Elixir function:

```elixir
defmodule StringUtils do
  @doc """
  Truncates a string to at most `max_len` characters.
  If truncated, appends `suffix` (default: "...").
  Returns the original string if it is already within `max_len`.
  Raises ArgumentError if max_len is negative.
  """
  @spec truncate(String.t(), non_neg_integer(), String.t()) :: String.t()
  def truncate(str, max_len, suffix \\ "...")
end
```

Requirements:
- Module name: `StringUtilsTest`, `use ExUnit.Case`
- Use `describe` blocks to group related cases
- At least 2 `describe` blocks: one for happy path, one for edge/error cases
- At least 8 test cases total covering: string shorter than limit, exact length, longer string (suffix appended), empty string, custom suffix, max_len of 0, negative max_len (should raise), unicode string
- Use `assert`, `assert_raise`, and pattern matching appropriately
- Add at least one doctest-style `## Examples` in a module comment showing usage

rubric

correctness

Do the 8+ tests cover all specified cases with correct expected values? Does `assert_raise ArgumentError` correctly test the negative max_len case? Are unicode string lengths handled via `String.length/1`, not `byte_size/1`?

quality

Are `describe` blocks used to group related tests logically? Are test names written as plain English specs ('truncates when string exceeds limit')? Is pattern matching used where it adds clarity (e.g. matching on {:ok, _} if the API returns tuples)? Is the module correctly structured with `use ExUnit.Case`?

documentation

Are describe block labels and test names readable as a spec? Is the module-level Examples comment present and accurate? Does the overall file read like idiomatic Elixir test documentation?

results

Claude Sonnet 4.6

8.0
correctness 8.0quality 7.0documentation 9.023942ms

# patched after judge fix

inspect full trace →

Gemini 2.5 Flash

6.3
correctness 6.0quality 7.0documentation 6.020880ms

# Covers all required scenarios and uses assert_raise/describe blocks well, but contains an internal inconsistency (empty string + custom suffix test contradicts its own assumed implementation) and the doctest Examples section has several numerically incorrect examples (e.g. 'hello world',7 => 'hello...' exceeds max_len), undermining correctness and documentation accuracy despite good structural style.

inspect full trace →

Claude Haiku 4.5

6.0
correctness 5.0quality 7.0documentation 6.06951ms

# patched after judge fix

inspect full trace →

Llama 3.3 70B

6.0
correctness 6.0quality 7.0documentation 5.01000ms

# Good coverage of required scenarios and correct assert_raise usage, but the custom-suffix expected value ('he~') is inconsistent with the truncation formula implied by the default-suffix test ('he...'), suggesting a copy-paste error, and the zero-length/negative-length assumptions are unverified given no reference implementation; the 'Examples' section is written as plain comments rather than a proper @moduledoc, weakening the documentation requirement.

inspect full trace →