Watch the benchmark harness run

This is a scripted replay of the actual test loop researcher Yusuke Endoh used: Claude Code (Opus 4.6) implements init, add, commit, and log for a mini Git clone, 20 times per language, timed and priced per run. Hit play, or jump straight to the finish.

Benchmark explorer — all 15 language configs

Real results from 600 runs (40 per config, v1+v2 tasks). Click a header to sort, click a chip to filter by family, click a row for the story behind the number.

Ruby or Python? Weight what matters to your team

The benchmark says Ruby edges Python by a hair on agent-authored code. But your team isn’t an agent. Pick the priorities that matter for your actual project and see how the case shifts.

Your CI pipeline is about to have opinions on your language

GitHub’s new Copilot desktop app (public preview, rolling out through summer 2026) lets teams run multiple coding agents at once against real repos — triaging issues, opening PRs, and looping on review feedback largely unattended. GitHub also shifted Copilot billing to usage-based, per-token pricing this year. Put those two facts together and the benchmark above stops being trivia: if your agents are writing glue scripts, Rakefiles, CI steps, or Lambda handlers, the language they write in changes your bill — not by a rounding error, but by up to 2.3× (Ruby/Steep’s $0.84 vs. plain Ruby’s $0.36).

Practical takeaway: for AI-agent-authored automation code specifically — the boring scripts nobody hand-reviews line by line — dynamic languages without a strict type checker bolted on came out cheapest and fastest across the board. Save the static typing for the code your humans own.

# Route agent-authored automation to a cheap, fast language runner,
# reserve statically-typed jobs for human-owned services.
name: agent-automation
on: [issue_comment]
jobs:
  run-agent-task:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: "Let Copilot / Claude Code write + run the fix"
        run: |
          # Ruby & Python both hit 40/40 pass rate in the 2026 benchmark;
          # Ruby ran ~2% cheaper, Python's ecosystem is broader.
          bundle exec rake agent:apply_patch
          bundle exec rspec --fail-fast