A plugin-based framework for evaluating agent harnesses, inference engines, and kernels, with more areas to come.
English | 中文
Quick Start · Documentation · Playbooks · Plugins
ARSI-Evaluator is a unified, plugin-based framework for agent harness capability evaluation, inference engine performance evaluation, and kernel performance evaluation. It can be extended to support more evaluation scenarios.
The framework uses YAML configuration and composable components to define evaluation workflows. It supports local, containerized, and distributed execution, records results, helps diagnose failures, and lets you resume interrupted runs.
ARSI-Evaluator architecture overview. For design details, see Source architecture and execution flow.
- Evaluation scenarios: Evaluate agent harnesses, inference engines, and kernels using supported benchmarks or custom tasks. Benchmark list
- Agent integrations: Use the built-in SimpleAgent or connect agent harnesses such as DSH, Dify, Pi, and DeepAgents through AgentBundle. Agent integrations
- Execution modes: Run locally, in Docker, or across Ray nodes, with configurable concurrency, environment warmup, and network settings. Execution modes
- Reliable evaluation runs: Save execution evidence, diagnose failures, resume interrupted runs, and retry tasks that failed due to network errors. Resume and recovery · Network retries
- Metrics and trajectory analysis: Calculate evaluation metrics, generate reports offline, and inspect execution trajectories and feedback with Tracer. Results and reports
- Plugins and SDK: Add custom evaluation components or integrate evaluations using the Node.js SDK. Plugin development · SDK integration
ARSI-Evaluator covers three types of evaluation. See the benchmark list for the full list and run requirements.
| Evaluation area | Representative benchmarks | What it evaluates |
|---|---|---|
| Model and agent capabilities | GSM8K, MMLU, SWE-bench, Terminal-Bench, WorkBuddy, OfficeQA, and others | Evaluate question answering, coding, terminal, office, and web tasks. Custom Harbor tasks are also supported. |
| Inference engine performance | AgentX, AISBench | Replay agent request traces and load-test model services to measure throughput, latency, and prefix cache metrics. AgentX guide · AISBench guide (Chinese) |
| Kernel performance | kernel | Verify kernel correctness and measure execution time and speedup. Kernel evaluation guide |
Give your coding agent the following instructions to install and verify the CLI and Skill.
Install the ARSI-Evaluator CLI and the accompanying arsi-evaluator Skill from https://github.com/ARSI-LAB/ARSI-Evaluator.
Follow the manual installation steps in the README to install the CLI, then install the Skill for yourself in the current project.
Run arsi-evaluator --help to verify the CLI and confirm that you can discover and read the Skill. Stop after verifying both.
You need Node.js 24.x (at least 24.22.1), pnpm 11 or 12, and Git. If pnpm is missing, run npm install -g pnpm.
git clone https://github.com/ARSI-LAB/ARSI-Evaluator.git arsi-evaluator
cd arsi-evaluator
pnpm install --frozen-lockfile
pnpm build
pnpm add -g .
arsi-evaluator --helpThe help output confirms the CLI is installed.
Run the following command in your target project to install the Skill for your coding agent.
npx skills add ARSI-LAB/ARSI-Evaluator --skill arsi-evaluatorFor other installation methods and troubleshooting, see the installation guide.
Run two GSM8K questions with the built-in SimpleAgent. Use Bash from the repository root and configure a model service that supports OpenAI Chat Completions and tool calling.
export MODEL_NAME="your-model-id"
export MODEL_ENDPOINT="https://your-model-service.example/v1"
export MODEL_API_KEY="your-api-key"
arsi-evaluator inspect playbook/gsm8k/gsm8k-quick-start.yaml
arsi-evaluator run playbook/gsm8k/gsm8k-quick-start.yamlUse inspect to check the experiment configuration and run to start the evaluation. When it finishes, open report.md in the results directory. See the quick-start guide for the full walkthrough and command-line examples.
For other evaluation scenarios, see the Terminal-Bench 4.0 quick start and the AgentX performance quick start. Find more experiment configurations in the Playbook examples.
For evaluations that need Docker images, you can check the environment and prepare images and runtime dependencies ahead of time to reduce setup time during the evaluation.
arsi-evaluator doctor playbook/terminalbench/terminalbench-4.0.yaml
arsi-evaluator warmup playbook/terminalbench/terminalbench-4.0.yaml
arsi-evaluator run playbook/terminalbench/terminalbench-4.0.yamlUse doctor to check the environment, warmup to prepare images and dependencies, and run to start the evaluation. Warmup makes no model calls and performs no scoring. The evaluation can reuse the prepared images. See the warmup guide for details.
The project publishes benchmark evaluation images in its ModelScope image repository. Use --network to configure the image source. See the image download guide for instructions.
Tracer lets you inspect agent trajectories, performance metrics, and optimization feedback. It shows model calls, tool events, and their timing, and provides rule-based diagnostics and optional LLM analysis. Add --tracer to enable it for an evaluation.
arsi-evaluator run playbook/gsm8k/gsm8k-quick-start.yaml --tracerTracer starts the local Collector on demand. On the machine running the Collector, open http://127.0.0.1:8787 in your browser to view trajectories and feedback.
Tracer trajectory analysis view. See the Tracer guide for configuration and troubleshooting.
These runs used DeepSeek V4.1 Flash 1017. tbd marks a pending evaluation. Scores follow each benchmark’s rules.
| Benchmark | Agent Harness | Result |
|---|---|---|
| GSM8K | SimpleAgent | 92.34% (1,218 / 1,319) |
| MMLU | choice-logprobs | 88.83% (12,474 / 14,042) |
| WorkBuddy | SimpleAgent | Office 0.8502 · Code 0.9009 · Web 0.7957 |
| Terminal-Bench 4.0 | SimpleAgent | tbd |
| SWE-bench Verified | SimpleAgent | tbd |
| OfficeQA Pro V2 | SimpleAgent | 56.67% (51 / 90) |
- Configure experiments: Configuration guide · Solvers and model services
- Choose an evaluation scenario: Benchmark guides · AgentX performance evaluation · Playbook examples
- Run and manage evaluations: Daily operations · Concurrency and environment preparation · Image cleanup
- Extend and integrate: Plugin development and loading · SDK integration
- Troubleshoot and recover: FAQ · Error code reference · Retrying network failures
- RSI is a plugin-based recursive self-improvement (RSI) framework for agents and AI infrastructure. It uses feedback from ARSI-Evaluator to generate and refine candidate solutions across iterations.
- AgentBundle packages and manages configurations, skills, and plugins across agent harnesses, making it easier to integrate and evaluate different agents in ARSI-Evaluator.
Thanks to the maintainers and contributors of Cordis and Harbor. ARSI-Evaluator uses Cordis to manage plugins and their lifecycles and supports running and verifying evaluation tasks using the Harbor protocol.


