Refund calculation
How we decide whether your model beats Jev, and so whether the credit comes back, with numbers anyone can recompute.
Every model trained on one decision is measured against Jev, the market reference model, on your own examples. If it does not beat Jev, the credit comes back to your balance on its own. "Beat" is not "one metric by any margin": the refund calculation asks for a gain that holds up under resampling and no metric that collapses.
The test examples
Before training, we set aside examples the model never sees:
- 20% of your labeled examples, drawn per answer so each answer keeps its share, up to 2,000.
- When your data has groups (the same repository, customer or document across rows), the test takes whole groups until it reaches that 20%, and the other rows of those groups stay out of training too. The model is never measured on a group it saw, so the comparison with Jev stays fair.
- When your data has no answers, the examples a person labeled in the review step instead (drawn from the evaluation groups, when there are groups).
The draw is seeded by the dataset, so every version of a model is measured on the same examples. Jev answers the same examples through the same question: its answers are used only to measure, never to train.
Four metrics
Each system gets four scores on the test examples, all higher-is-better:
| Metric | What it says |
|---|---|
macro_f1 | How often the answer is right, with every answer weighing the same, so a rare answer counts as much as a common one. |
brier_skill | How good the probabilities are: 1 - mean Brier / 2, from 0 to 1. Rewards being confident when right and unsure when wrong. |
selective_acc_80 | Accuracy over the 80% most confident answers: what you get when you automate the confident ones and send the rest to a person. |
worst_class_f1 | F1 of the weakest answer with 5 or more examples. No answer may collapse. |
The rule
The weighted gain over Jev is
G = 0.5 × Δmacro_f1 + 0.3 × Δbrier_skill + 0.2 × Δselective_acc_80where Δ is our score minus Jev's. The model beats Jev only if both hold:
- The gain is real. We resample the test examples 2,000 times (a paired bootstrap: the same draw for both systems)
and recompute
Geach time. The 5th percentile of those 2,000 values must be above zero:G > 0with 95% confidence. - Nothing collapses. No metric may be worse than Jev's by more than its margin: 0.02 for the first three, 0.05 for
worst_class_f1.
Anything else is a refund, ties included. When probabilities tie for the top, both systems get the same benefit of the doubt: the answer counts as right if the right one is among the tied.
Deterministic
The bootstrap's seed comes from the SHA-256 of the test examples' texts. Same examples, same answers, same numbers, every time.
Verify it yourself
Each model has an audit file (GET /v1/models/{id}/audit): for every test example, the SHA-256 of its text, the right
answer and the probabilities of both systems, plus the report with every number above. The verifier rebuilds the whole
report with the Python standard library and fails if any number differs:
curl -O https://typestrict.ai/verify_audit.py
python3 verify_audit.py audit.jsonThe audit holds no text of yours, only hashes and probabilities, so it stays with the model's record even after its files expire (see Retention).
Several decisions in one model
A multi-task plan is compared with Jev decision by decision, each on its own held-out examples, with the same rule. The model beats Jev when it beats it on at least two thirds of the decisions (4 of 5, 2 of 3, both of 2); below that, the credit comes back. The result page shows the decision that settled it (the first one won, or the first one lost), and the audit file has every decision's numbers. Multiple-choice decisions count too: Jev gets the same options each example offers.
A training that fails returns the credit too, whatever the kind of model.
Keep the model even if it loses
When you start a training you can choose to keep the model even if it does not beat Jev (the Keep the model even if it
does not beat Jev box on the base model step, or "options": { "keep_if_loses": true } in the API). It is off unless
you turn it on, and it gives up the guarantee for that training:
- If the model beats Jev, nothing changes: verdict
beats. - If it does not beat Jev, it is delivered anyway: verdict
kept. Its files stay for 30 days and can be downloaded or published like any model, and there is no refund: the credit stays spent. - If the training fails on our side, the credit still comes back.
The comparison, the scores and the audit file are the same as for any model, so a kept model shows by how much it lost.