Refund calculation
How we decide whether your model beats Jev, and so whether the credit comes back, with numbers anyone can recompute.
Every model trained on one decision is measured against Jev, the market reference model, on your own examples. If it does not beat Jev, the credit comes back to your balance on its own. "Beat" is not "one metric by any margin": the refund calculation asks for a gain that holds up under resampling and no metric that collapses.
The test examples
Before training, we set aside examples the model never sees:
- 20% of your labeled examples, drawn per answer so each answer keeps its share, up to 2,000.
- When your data has no answers, the examples a person labeled in the review step instead.
The draw is seeded by the dataset, so every version of a model is measured on the same examples. Jev answers the same examples through the same question: its answers are used only to measure, never to train.
Four metrics
Each system gets four scores on the test examples, all higher-is-better:
| Metric | What it says |
|---|---|
macro_f1 | How often the answer is right, with every answer weighing the same, so a rare answer counts as much as a common one. |
brier_skill | How good the probabilities are: 1 - mean Brier / 2, from 0 to 1. Rewards being confident when right and unsure when wrong. |
selective_acc_80 | Accuracy over the 80% most confident answers: what you get when you automate the confident ones and send the rest to a person. |
worst_class_f1 | F1 of the weakest answer with 5 or more examples. No answer may collapse. |
The rule
The weighted gain over Jev is
G = 0.5 × Δmacro_f1 + 0.3 × Δbrier_skill + 0.2 × Δselective_acc_80where Δ is our score minus Jev's. The model beats Jev only if both hold:
- The gain is real. We resample the test examples 2,000 times (a paired bootstrap: the same draw for both systems)
and recompute
Geach time. The 5th percentile of those 2,000 values must be above zero:G > 0with 95% confidence. - Nothing collapses. No metric may be worse than Jev's by more than its margin: 0.02 for the first three, 0.05 for
worst_class_f1.
Anything else is a refund, ties included. When probabilities tie for the top, both systems get the same benefit of the doubt: the answer counts as right if the right one is among the tied.
Deterministic
The bootstrap's seed comes from the SHA-256 of the test examples' texts. Same examples, same answers, same numbers, every time.
Verify it yourself
Each model has an audit file (GET /v1/models/{id}/audit): for every test example, the SHA-256 of its text, the right
answer and the probabilities of both systems, plus the report with every number above. The verifier rebuilds the whole
report with the Python standard library and fails if any number differs:
curl -O https://typestrict.ai/verify_audit.py
python3 verify_audit.py audit.jsonThe audit holds no text of yours, only hashes and probabilities, so it stays with the model's record even after its files expire (see Retention).
When the guarantee does not apply
- Several decisions in one model (a multi-task plan): each task is measured against the base model it started from, not against Jev, and there is no refund for the result.
- A training that fails returns the credit too, whatever the kind of model.