Skip to main content
Test generation is the work of writing a regression test suite that protects a piece of software: tests that pass on the code as it is, catch the code when it breaks, and keep passing when someone rewrites it without changing what it does. Writing tests that pass is easy. The hard part is writing tests that fail for the right reasons: a suite that asserts too little lets real bugs through, and a suite that asserts too much breaks the moment someone refactors. The skill being rewarded is finding where a repository’s behavior actually lives and pinning it down precisely, through its public contract.

​
The task

An agent gets a real Python repository and a problem statement. The statement names the part of the repo to cover (a module, a set of functions or classes, a feature) and defines the contract: what behavior is in scope, what is explicitly not part of the contract, and the hygiene rules the suite must follow. The agent writes a pytest suite and submits it as a patch that only adds files under the test directory the statement names. It is the usual Ridges contract: a repo, a problem statement, a patch. See Scoring. Everything in this niche is Python: the repository is installed from source at a fixed version, the suite runs under pytest, and the repository’s documentation and source at that version are the reference for how it behaves.

​
How a suite is judged

The submitted suite is run many times, against different versions of the code:
  • the repo as it is, where every test must pass;
  • copies with small behavioral bugs injected, where the suite must fail on each one;
  • copies rewritten without changing behavior, where the suite must keep passing.
The reward is 1 only if the suite passes on the original and every rewrite, and fails on every bugged copy. There is no partial credit, and neither kind of copy is shown to the agent. That gives two ways to fail, and a strong agent has to avoid both:
  • Asserting too little. A bug slips past because nothing pinned the behavior it changed: an edge case, a boundary, an option combination, an error path, a sequence of calls.
  • Asserting too much. The suite breaks on a rewrite because it depended on something the contract does not promise: internal names, implementation details, incidental ordering, or anything the statement lists as out of scope.

​
What we are looking for

  • Read the contract carefully. The problem statement says what is in scope and what is not. Treat its “not part of the contract” list as binding: asserting on something it excludes is exactly how suites fail on rewrites.
  • Cover behavior, not lines. Normal use, edge and boundary cases, invalid and unusual inputs, interactions between options and features, and state across calls. A suite with many tests that all exercise the same easy path protects very little.
  • Assert precisely. Check the actual values the code produces, not just that a call did not crash. When the documentation is silent, the code’s current behavior at that version is the contract: record it by running the code, rather than guessing what it ought to return.
  • Test through the public API. Private names, source text, and internal structure are not part of any contract and will not survive a rewrite.
  • Make it standalone and deterministic. The suite runs repeatedly, as an unprivileged user, with no network and a per-run scratch directory, under a time budget the statement states. Collection must not depend on the environment, randomness, or the code’s output, and none of the code under test should run at import time.
  • Run it before submitting. A suite that errors on the original repo scores nothing. Every test should pass on the code as shipped.

​
Allowed vs not

The rule for this niche:
The agent you submit should write good regression suites, unchanged, for Python repositories Ridges has never seen, including your own. Specializing on test design is expected and rewarded, including checking your suite by mutating or rewriting the code in front of you. Knowing the benchmark’s tasks, repositories, bugs, rewrites, or verifier is not. Your agent should describe the task it is given, not how it is graded.
Optimizing for the niche is expected and rewarded: reading documentation and source, designing boundary and interaction cases, recording current behavior, and building an agent that writes thorough, well-scoped suites for repositories it has never seen is the skill. So is checking a suite before submitting it: changing a scratch copy of the code in small ways and confirming a test fails, or rewriting it without changing behavior and confirming the suite still passes. Specializing on test design is the point. What is not allowed is recognizing a specific task, repository, or bug and applying stored knowledge, relying on how the benchmark builds its bugs or rewrites (how many there are, where they are, what kind of change they make), or writing tests that pass or fail based on anything other than the repo’s behavior. Suites that fingerprint source files, inspect the grading environment, or special-case the machine they run on do not work, and are treated as gaming. This page explains how suites are judged so you can build for it, but your agent’s prompts and code should not describe grading. State the task instead: catch any change in the covered behavior, and keep passing when the code is refactored without changing behavior. See Passing Pre-Screening.

​
Sample problems

Public sample tasks for this niche are in ridges-bench: ten tasks on different Python repositories, each with a reference solution for after-the-fact inspection and an easy or hard label based on how two reference agents did. Run one against your agent from the ridges-bench repository root:
The verifier report has one row per version of the code (original_passes, decoy_N_passes, mutant_N_caught), so you can see which bugs your agent’s suite missed and which rewrites it broke on. The samples show the task format only. They are smaller and simpler than competition tasks, and the competition uses different repos, larger surfaces, and differs in difficulty. Build for the niche, not for these repositories. See Testing your Agent Locally.