Score every checkpoint, name the best one
Upload a test set, build one in the dashboard, or let Nucleus generate it. Set the criteria and a pass threshold, and one run scores every checkpoint of your fine-tuned model against them.
Benchmark scorethreshold 93.0%
Serving price$0.31 / 1M tokens
Every checkpoint, one runsupport-eval.jsonl · 1,200 cases
threshold 93.0%
Bring your benchmarks
Upload domain-specific test sets and compare accuracy, speed and cost.
Build in the dashboard
Define questions, criteria and pass thresholds, with no code required.
Auto-generated benchmarks
Nucleus generates domain-specific tests and scores each checkpoint.
A before-and-after number
Pass compare_to and the response carries baseline_score beside score, so the gain from fine-tuning reads straight off it.
One set, one run, one verdict per checkpoint
A run takes the benchmark and scores each checkpoint of the fine-tuned model against its criteria. A checkpoint under the threshold is held, and the highest score is named the best. In this run the third checkpoint clears 93.0% and the fourth, the newest, falls back under it.
Results from LLMs powered by Nucleus
prediction accuracy on equipment failures
Predictive Maintenance
monthly revenue misattributed
Ad Performance Tracking
rerouted in a single fleet optimisation
Intelligent Fleet Optimisation
average time to resolution
Critical Incident Response
Where a threshold settled it
Each is an example workflow, powered by Nucleus, and every figure comes from the one it describes.
Every incident past its four-hour SLA, escalated by name
Line 3's next failure, booked into the maintenance window
Why Block 4B's rice yield came in 14% under target
Everything an evaluation carries
Ways in
- Upload a JSONL test set, purpose evaluation
- Build the benchmark in the dashboard, no code
- Generate one from a run or a domain description
- Connect a system of record as an outcome source
Scoring
- Metrics exact_match, rouge, pass_rate and grader
- score on 0 to 1, with n, the cases scored
- compare_to scores the base model in the same run
- A breakdown across the benchmark's criteria
Criteria and thresholds
- Criteria with a case count, held on the benchmark
- One pass threshold per benchmark
- A checkpoint under it is reported as held
- best_checkpoint names the highest scoring one
API
- POST /v1/benchmarks holds the cases and the bar
- POST /v1/evaluations answers 202 with an operation
- Pass training_run to score every checkpoint at once
- List by benchmark for the history across releases
Evaluations billed on training meters
Evaluation runs are metered as token consumption on the same meters as everything else. There is no separate charge type.
Evaluations
Metered as token consumption, on the same meter your fine-tuning jobs run on.
Storage
Benchmarks, datasets and checkpoints are metered by the GB-hour, so you only pay while you keep them.
Also in Evaluate
Outcome benchmarks
Business results as the test set
Evaluate
Everything in Evaluate, in one place.
Read on
Benchmarks
Hold the cases, the criteria and the pass threshold on one set, and reference it from every run.
Evaluations
Score every checkpoint of a training run against a benchmark, and name the best one.
Managed fine-tuning
Go from a raw JSONL dataset to a served private model with the managed job API. Step 5 evaluates the result.
Score your next checkpoint
Read the evaluations reference, or see what a run costs on the training meters.