Decide, Solve or Think
RungBench · 1,073 rule-application items

Decide, Solve or Think

Routing rule-application decisions between one-pass decision models, symbolic solvers and thinking models.

A decision model answers a typed question in one forward pass and returns a probability per option. A thinking model writes a chain of thought first. We find which decisions the fast model gets right, and route the rest.

≤ 44%for every decision model once a computed value feeds a later step; most thinking models keep 79–100%
96–97%for the strongest decision models when they read the facts and a symbolic solver applies the rule
98.4% · 5.2 sthree untuned routing rules with Jev, against 96.9% at 156 s for a local thinking model on every item

One case, five questions

Every item states a rule and a case, and the gold answer comes from running the rule. Each case is asked five questions, each one step deeper. Decision models read the case and score one thing well. Once they have to add computed values, they fall. Pick a question.

Nutrition information · cheese
per 100 gper 45 g serving decoy
Fruit & vegetables 18%
Rule (per 100 g)
A nutrient earns 1 point per threshold its value is strictly above.
salt: 0.2 · 0.4 · 0.6 · 0.8 · 1 …
sugars: 3.4 · 6.8 · 10 … energy: 335 · 670 …
sat. fat: 1 · 2 · 3 … protein: 2.4 · 4.8 · 7.2 · 9.6 · 12 …
N = energy + sugars + sat. fat + salt points
P = fibre + fruit/veg + protein points
score = N − P → grade A if ≤ 0, B if 1–2, C if 3–10 …
decision modelsthinking models

The task type sets the boundary

Single-step decisions (read, check, one number) are System 1: the best decision models match thinking models. Once a computed value feeds a later step, every decision model falls. With source and length held fixed, that step costs decision models 42 points and thinking models 8.

decision models, median and range thinking models, median and range selected model

Accuracy falls along each derivation

Accuracy against derivation length C per source. Decision models drop at the first chained step; thinking models stay flat and pay in tokens instead. Capirca stays a check at every length, and the strong decision models do not drop.

decision models: median, band = rangethinking models: median, band = range

Read with the decision model, compute with a program

Decision models read stated facts well (median 94% on read items). So we split the decision: the model extracts each rule input as a typed question, a short rule program computes, and the model answers with the result in its state.

① Read

One choice question per rule input, about the label only.
energy? 335 kJ
sugars? 4.4 g
sat. fat? 0.9 g
salt? 0.71 g (not 0.32)

② Compute

The rule as a program, no model.
points: 0 + 1 + 0 + 3
N = 4

③ Answer

The result goes into the state as one line; the decision model answers “Total negative points N?”
→ 4 ✓ (alone: 9%)
alone with solver (final answer) program's own answer thinking-model range (87–100)
Text items (783). The program's own answer beats the final read for all eight models: they overrule a correct program more often than they rescue a wrong one.
The model reads new phrasings; a parser does not. A parser written for the released text from its dev split reaches 98.2%, then drops when the same items are rewritten.

Three rules route each decision

The router reads only the decision model's own outputs: its one-pass answer and the facts it extracted, each with a confidence. One knob, t. At t = 0 nothing is tuned: return the program whenever it runs, think only when it cannot.

1
Certificate. The program runs and agrees with the one-pass answer → return it.
2
Trust the program. They disagree, every extracted fact has confidence ≥ t → return the program.
3
Think. The program cannot run, or a disagreement rests on an unsure fact → thinking model.
With an API thinking model, Claude Opus 5.5: Jev's three rules reach 98.4% at a mean 0.8 s per item, against 100.0% at 4.7 s for Opus on every item. Times for the local thinking models are estimated from generated tokens at 40 tokens/s.
No program? Use the model's own confidence. Send a task to the thinking model when Jev's probability for its answer is below 0.8. JevBench was never used in designing the router.
RouterRungBench sentAcc.JevBench sentAcc.
Jev alone0%65.80%85.3
Confidence < 0.8 (ours)57%95.223%96.5
Thinking model on every task100%96.9100%97.8
Thinking model: Qwen3.5-9B on RungBench, DeepSeek V4.1 Flash (published outcomes) on the 231 public JevBench tasks. The threshold recovers 94% and 90% of the gap; most of what it sends is multi-step.

All results

Accuracy (%) and median time per item. Click a column to sort. “~” marks local thinking-model times estimated from generated tokens.

ModelSizeText (783)Multimodal (290)All (1,073)Time (s)