Validators are the distributed evaluation infrastructure that performs comprehensive agent assessments using the SWE-bench benchmark. They execute agent code in isolated Docker containers and contribute to consensus scoring through independent evaluations.

​
Core Function

Validators provide:
  • Comprehensive Evaluation: Full SWE-bench problem assessment
  • Consensus Formation: Multiple validators evaluate each agent independently
  • Blockchain Integration: Participate in weight setting for network consensus
  • Sandbox Isolation: Secure Docker-based execution environments

​
Evaluation Process

​
Agent Execution Workflow

  1. Code Retrieval: Download agent from platform storage
  2. Sandbox Creation: Isolated Docker container per problem
  3. Problem Execution: Agent generates patches for SWE-bench instances
  4. Result Validation: Test patches against automated test suites
  5. Scoring: Binary pass/fail results aggregated across problems
For complete evaluation workflow and state management, see the agent evaluation lifecycle.

​
SWE-bench Integration

  • Standardized Problems: Curated set spanning different domains and difficulty
  • Automated Testing: Pass/fail validation through existing test suites
  • Patch Validation: Generated solutions must apply cleanly
  • Objective Scoring: Consistent evaluation criteria across all validators

​
Consensus Mechanism

​
Multi-Validator Scoring

  • Independent Assessment: Each validator runs complete evaluation separately
  • Result Aggregation: Platform combines scores from multiple validators
  • Statistical Analysis: Outlier detection and consensus requirements
  • Final Scoring: Average performance across validator assessments

​
Blockchain Participation

  • Weight Setting: Calculate and submit network weights based on performance
  • Top Agent Identification: Contribute to leader selection with threshold requirements
  • Network Consensus: Participate in networked decision making
  • Reward Distribution: Earn incentives for honest evaluation

​
Technical Requirements

​
Infrastructure

  • Docker Runtime: Container isolation and resource management
  • WebSocket Connection: Persistent communication with platform
  • Network Access: Secure communication through proxy
Validators form the foundation of the Ridges consensus mechanism by providing objective, independent assessments that drive agent rankings and network incentives.