Skip to content

About

Many ways to evaluate. One framework to bring them together. Unify benchmarks, agent evaluations, and performance testing across coding, terminal, office, web, and more.

Topics

Resources

Stars

41 stars

Watchers

0 watching

Forks

Repository files navigation

ARSI-Evaluator

Many ways to evaluate. One framework to bring them together.

A plugin-based framework for evaluating agent harnesses, inference engines, and kernels, with more areas to come.

Node.js 24.x pnpm 11.22.0 License: Apache-2.0

English | 中文

Quick Start · Documentation · Playbooks · Plugins

ℹ️ Introduction

ARSI-Evaluator is a unified, plugin-based framework for agent harness capability evaluation, inference engine performance evaluation, and kernel performance evaluation. It can be extended to support more evaluation scenarios.

The framework uses YAML configuration and composable components to define evaluation workflows. It supports local, containerized, and distributed execution, records results, helps diagnose failures, and lets you resume interrupted runs.

ARSI-Evaluator architecture and evaluation workflow

ARSI-Evaluator architecture overview. For design details, see Source architecture and execution flow.

⚙️ Features

  • Evaluation scenarios: Evaluate agent harnesses, inference engines, and kernels using supported benchmarks or custom tasks. Benchmark list
  • Agent integrations: Use the built-in SimpleAgent or connect agent harnesses such as DSH, Dify, Pi, and DeepAgents through AgentBundle. Agent integrations
  • Execution modes: Run locally, in Docker, or across Ray nodes, with configurable concurrency, environment warmup, and network settings. Execution modes
  • Reliable evaluation runs: Save execution evidence, diagnose failures, resume interrupted runs, and retry tasks that failed due to network errors. Resume and recovery · Network retries
  • Metrics and trajectory analysis: Calculate evaluation metrics, generate reports offline, and inspect execution trajectories and feedback with Tracer. Results and reports
  • Plugins and SDK: Add custom evaluation components or integrate evaluations using the Node.js SDK. Plugin development · SDK integration

🧪 Supported Benchmarks

ARSI-Evaluator covers three types of evaluation. See the benchmark list for the full list and run requirements.

Evaluation area Representative benchmarks What it evaluates
Model and agent capabilities GSM8K, MMLU, SWE-bench, Terminal-Bench, WorkBuddy, OfficeQA, and others Evaluate question answering, coding, terminal, office, and web tasks. Custom Harbor tasks are also supported.
Inference engine performance AgentX, AISBench Replay agent request traces and load-test model services to measure throughput, latency, and prefix cache metrics. AgentX guide · AISBench guide (Chinese)
Kernel performance kernel Verify kernel correctness and measure execution time and speedup. Kernel evaluation guide

📦 Installation

Install with a Coding Agent

Give your coding agent the following instructions to install and verify the CLI and Skill.

Install the ARSI-Evaluator CLI and the accompanying arsi-evaluator Skill from https://github.com/ARSI-LAB/ARSI-Evaluator.
Follow the manual installation steps in the README to install the CLI, then install the Skill for yourself in the current project.
Run arsi-evaluator --help to verify the CLI and confirm that you can discover and read the Skill. Stop after verifying both.

Manual Installation

1. Install the CLI

You need Node.js 24.x (at least 24.22.1), pnpm 11 or 12, and Git. If pnpm is missing, run npm install -g pnpm.

git clone https://github.com/ARSI-LAB/ARSI-Evaluator.git arsi-evaluator
cd arsi-evaluator
pnpm install --frozen-lockfile
pnpm build
pnpm add -g .
arsi-evaluator --help

The help output confirms the CLI is installed.

2. Install the Agent Skill (Optional)

Run the following command in your target project to install the Skill for your coding agent.

npx skills add ARSI-LAB/ARSI-Evaluator --skill arsi-evaluator

For other installation methods and troubleshooting, see the installation guide.

🚀 Quick Start

Run two GSM8K questions with the built-in SimpleAgent. Use Bash from the repository root and configure a model service that supports OpenAI Chat Completions and tool calling.

export MODEL_NAME="your-model-id"
export MODEL_ENDPOINT="https://your-model-service.example/v1"
export MODEL_API_KEY="your-api-key"

arsi-evaluator inspect playbook/gsm8k/gsm8k-quick-start.yaml
arsi-evaluator run playbook/gsm8k/gsm8k-quick-start.yaml

Use inspect to check the experiment configuration and run to start the evaluation. When it finishes, open report.md in the results directory. See the quick-start guide for the full walkthrough and command-line examples.

For other evaluation scenarios, see the Terminal-Bench 4.0 quick start and the AgentX performance quick start. Find more experiment configurations in the Playbook examples.

🔥 Warmup Before Evaluation (Optional)

For evaluations that need Docker images, you can check the environment and prepare images and runtime dependencies ahead of time to reduce setup time during the evaluation.

arsi-evaluator doctor playbook/terminalbench/terminalbench-4.0.yaml
arsi-evaluator warmup playbook/terminalbench/terminalbench-4.0.yaml
arsi-evaluator run playbook/terminalbench/terminalbench-4.0.yaml

Use doctor to check the environment, warmup to prepare images and dependencies, and run to start the evaluation. Warmup makes no model calls and performs no scoring. The evaluation can reuse the prepared images. See the warmup guide for details.

The project publishes benchmark evaluation images in its ModelScope image repository. Use --network to configure the image source. See the image download guide for instructions.

🛰️ Tracer for Trajectory and Feedback Analysis

Tracer lets you inspect agent trajectories, performance metrics, and optimization feedback. It shows model calls, tool events, and their timing, and provides rule-based diagnostics and optional LLM analysis. Add --tracer to enable it for an evaluation.

arsi-evaluator run playbook/gsm8k/gsm8k-quick-start.yaml --tracer

Tracer starts the local Collector on demand. On the machine running the Collector, open http://127.0.0.1:8787 in your browser to view trajectories and feedback.

Tracer waterfall showing the timing of model calls and tool events

Tracer trajectory analysis view. See the Tracer guide for configuration and troubleshooting.

📊 Evaluation Results

These runs used DeepSeek V4.1 Flash 1017. tbd marks a pending evaluation. Scores follow each benchmark’s rules.

Benchmark Agent Harness Result
GSM8K SimpleAgent 92.34% (1,218 / 1,319)
MMLU choice-logprobs 88.83% (12,474 / 14,042)
WorkBuddy SimpleAgent Office 0.8502 · Code 0.9009 · Web 0.7957
Terminal-Bench 4.0 SimpleAgent tbd
SWE-bench Verified SimpleAgent tbd
OfficeQA Pro V2 SimpleAgent 56.67% (51 / 90)

📖 Guides

🔗 Related Projects

  • RSI is a plugin-based recursive self-improvement (RSI) framework for agents and AI infrastructure. It uses feedback from ARSI-Evaluator to generate and refine candidate solutions across iterations.
  • AgentBundle packages and manages configurations, skills, and plugins across agent harnesses, making it easier to integrate and evaluate different agents in ARSI-Evaluator.

🙏 Acknowledgments

Thanks to the maintainers and contributors of Cordis and Harbor. ARSI-Evaluator uses Cordis to manage plugins and their lifecycles and supports running and verifying evaluation tasks using the Harbor protocol.

About

Many ways to evaluate. One framework to bring them together. Unify benchmarks, agent evaluations, and performance testing across coding, terminal, office, web, and more.

Topics

Resources

Stars

41 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages