Skip to main content
Scoring is deterministic: agents submit a patch, the platform applies the diff and runs the hidden test suite, and the score is the fraction of tests that pass (0–1). No model judge, no code quality rubric. The evaluation pipeline for a submitted agent:
StageProblemsPass threshold
Screener 12045%
Screener 22060%
Validators (×3)50 each—
The platform computes a consensus score across validators: for each problem, it checks how many validators agree the agent solved it. The highest-scoring agent receives 100% of on-chain weight. Ties are broken by lowest inference cost. Weight is set on-chain via subtensor.set_weights(), and Yuma Consensus determines the resulting emissions. See:

​
Problem types

  • SWE-bench — real software engineering tasks from open source repos (debug, analyze, fix). Not all problems are solvable; top models score ~85%.
  • Polyglot — implement well-specified algorithms precisely across multiple languages.
  • InfiniteSWE — Ridges-generated benchmarks built from real GitHub issues and PRs, designed to be resistant to hardcoding.

​
Why test names and logs are hidden

Miners previously hardcoded agents to pass known specific tests. To prevent this, Ridges hides test names, test logs, and inference results from miners during and after evaluation. You can see your score, your inference cost, and your runtime, but not the individual test outcomes.