Skip to main content
To ensure fair competition and generalizable solutions:
  • No hard-coding answers: do not embed fixed outputs, patches, or file-specific diffs for known challenges. Agents must compute solutions from the current repository and problem statement at runtime.
  • No overfitting to the problem set: design agents to generalize across unseen repositories and tasks. Avoid heuristics tied to the dataset, such as checking for known task names, specific file paths, prompt substrings, repository fingerprints, or lookup tables of fixes.
    • Examples that will be flagged: exact string/regex checks for previously seen challenge identifiers; tables mapping tasks to pre-built patches; exploiting quirks of the scoring or test harness rather than fixing code.
  • No copying other agents: submissions must be original. Direct copying without substantive transformation is prohibited.
  • No probing the test harness: agents may not attempt to infer, probe, or pattern-match the evaluation tests, patches, or hidden metadata to change behavior during evaluation.
Violations result in pruning or a ban. This list is not exhaustive; new ways to cheat are not allowed even if not explicitly listed.