MoonshotAI: Kimi

6/46 tasks solved at reward >= 0.95 (pass@0.95); mean reward x100 = 37.8; official LHTB Harbor harness

agent=kimi-k3; 3-LLM judge panel (majority vote); turn-weighted weighted pass rate (0-100); post-publication run

harness=claude_code; harness_variant=thinking-max; attempted=149/152; official leaderboard snapshot=2026-10-01T20:39:17.826877+00:00; values are percentages (0-100); equal-task aggregation; unattempted tasks contribute zero; pass threshold score>=0.999. This is an official leaderboard export, not an HF-verified evaluation.

harness=kimi_code; harness_variant=thinking-max; attempted=149/152; official leaderboard snapshot=2026-10-01T20:39:17.826877+00:00; values are percentages (0-100); equal-task aggregation; unattempted tasks contribute zero; pass threshold score>=0.999. This is an official leaderboard export, not an HF-verified evaluation.

lighteval, LEXam paper prompts, one response per question, no tools; DeepSeek-R1-0528 judge; mean of the German and English means over the 518 questions, 0-100

with-skills; BenchFlow harness; OpenHands agent; 87 tasks x 3 trials; full 261/261 coverage

Harness: Hermes; Protocol: 75 tasks x 3 trials; Metric: pass@1 (%)

ResearchHarness evaluation with tools enabled, code execution, and a file-system workspace; completed 38/40 ResearchClawBench tasks.

Pass³% | N=3 | 161 tasks

medium preset, avg final funds (USD) across seeds 1,2,3

0/46 tasks solved at reward >= 0.95 (pass@0.95); mean reward x100 = 25.5; official LHTB Harbor harness

lighteval, LEXam paper prompts, one response per question, no tools; DeepSeek-R1-0528 judge; mean of the German and English means over the 518 questions, 0-100

With tools

GPQA Diamond

512 frames

ResearchHarness evaluation with tools enabled, code execution, and a file-system workspace; completed 39/40 ResearchClawBench tasks.

64 frames

Pass³% | N=3 | 161 tasks

avg final funds (USD) across seeds 1,2,3. Kimi K2.5 (via OpenRouter moonshotai/kimi-k2.5)

agent: Terminus 2

harness: react_toolbelt_agent; reasoning_effort: thinking