Score every checkpoint, name the best one
Upload a test set, build one in the dashboard, or let Nucleus generate it. Set the criteria and a pass threshold, and one run scores every checkpoint of your fine-tuned model against them.
Bring your benchmarks
Upload domain-specific test sets and compare accuracy, speed and cost.
Build in the dashboard
Define questions, criteria and pass thresholds, with no code required.
Auto-generated benchmarks
Nucleus generates domain-specific tests and scores each checkpoint.
A before-and-after number
Pass compare_to and the response carries baseline_score beside score, so the gain from fine-tuning reads straight off it.
One set, one run, one verdict per checkpoint
A run takes the benchmark set and scores each checkpoint of the fine-tuned model against its criteria. A checkpoint under the threshold is held, and the highest score is named the best. In this run the third checkpoint clears 93.0 and the fourth, the newest, falls back under it.
Results from LLMs powered by Nucleus
prediction accuracy on equipment failures, predictive maintenance
monthly revenue re-attributed in marketing, ad performance
rerouted in a single fleet optimisation, fleet optimisation
average time to resolution, incident response
Three workflows from the use cases page
Each is an example workflow, powered by Nucleus, and every figure comes from the one it describes.
Six vendor contracts ranked before the renewal meeting
Line 3's next failure, booked into the maintenance window
Why Block 4B's rice yield came in 14% under target
Everything an evaluation carries
Ways in
- Upload a JSONL test set, purpose evaluation
- Build questions, criteria and thresholds in the dashboard
- Domain benchmarks generated by Nucleus
Scoring
- Metrics exact_match, rouge, pass_rate and grader
- A domain score per checkpoint
- compare_to scores the base model in the same run
- score, baseline_score and n on the response
Criteria and thresholds
- A case count per criterion
- One pass threshold per set
- A checkpoint under the threshold is held
- The highest score is named the best
API
- POST /v1/evaluations answers 202 with an operation
- GET /v1/evaluations/{id}
- Poll the operation or the evaluation until succeeded
- Any checkpoint by its nucleus:// path
Evaluations billed on training meters
Evaluation runs are metered as token consumption on the same meters as everything else. There is no separate charge type.
Training tokens
Fine-tuning jobs and training-loop steps are metered per million tokens processed, whichever way you train.
Also in Evaluate
Outcome benchmarks
Business results as the test set
Read on
Docs · Evaluations
Score a model against a dataset and compare it with a baseline.
Guide · Managed fine-tuning
Go from a raw JSONL dataset to a served private model with the managed job API. Step 5 evaluates the result.
Docs · Checkpoints
Save, inspect, export and manage the weights produced by training runs.
Score your next checkpoint
Read the evaluations reference, or see what a run costs on the training meters.