Rippling Tests 15 AI Models on Real Payroll Data

Rippling evaluated 15 AI models through 2,100 scored agent runs each on real personnel, payroll, and financial records. Tasks included headcount queries by department and tenure, salary increases of 10 percent for qualifying employees, new hire onboarding, termination scheduling with approvals, and payment entry from spreadsheets. Every run either passed production correctness checks or failed, with unfinished attempts counted as failures.

Model Performance Results

Opus 4.6 recorded a 91.0 percent pass rate at a cost of $1,453 and 154 seconds on the slowest 10 percent of tasks. GPT-5.5 med scored 89.5 percent for $1,435. GPT-5.5 low reached 88.8 percent at $1,308 and 130 seconds on the slowest 10 percent. GLM 5.2 achieved 88.7 percent for $621 and 243 seconds on the slowest 10 percent, according to SaaStr.

Seven models fell between 88.5 percent and 89.5 percent pass rates. GLM 5.2 and Fable 5 differed by one tenth of a point in accuracy while one cost seven times the other.

Cost and Speed Comparisons

Opus 4.6 outperformed newer Anthropic models on accuracy while costing 42 percent less than Opus 5. Fable 5 ranked fifth overall despite the highest price. Grok 4.5 scored 0.2 points behind Opus 5 at 68 percent lower cost. A re-run of 2,100 attempts on Grok 4.6 showed accuracy falling from 87.3 percent to 85.9 percent and typical response time rising from 71 seconds to 131 seconds.

Tuning and Version Effects

Rippling spent five months tuning instructions and tools for Opus 4.6, an effort estimated to add one or two points of accuracy. All other models ran without tuning. Version numbers did not guarantee improvement, as newer models sometimes underperformed prior versions on the same tasks, according to SaaStr.

The spread across the leading models measured 2.5 points total. Results from different runs on different days cannot be compared directly.