Know how good your model really is
Upload a test set, build one in the dashboard, or let Nucleus generate it. Every checkpoint gets a domain score against your criteria and your pass threshold. The best one is named.
Bring your benchmarks
Upload a domain-specific test set and compare accuracy, speed and cost across checkpoints.
Build in the dashboard
Define the questions, the criteria and the pass threshold in the dashboard, with no code to write.
Auto-generated benchmarks
Nucleus generates domain-specific tests and scores each checkpoint against them.
A before-and-after number
Score the base model on the same set in the same run, and the delta from fine-tuning reads straight off the response.
Score against a test set, or against the results you record
Evaluations score a checkpoint against a benchmark set you upload, build or generate. Outcome benchmarks score it against the results your systems already record.
An evaluation runs a checkpoint over your test set and scores the outputs, giving you one number for how well it performs. Set the criteria and the pass threshold once, and every checkpoint from a run is scored against them. Pass compare_to and the base model is scored on the same set in the same run, so the delta from fine-tuning is read straight off the response.
Results from LLMs powered by Nucleus
prediction accuracy on equipment failures
yield shortfall traced to one cause
monthly revenue misattributed
average time to resolution
Where a measurement changed the decision
Line 3's next failure, booked into the maintenance window
$73.4K a month was being credited to the wrong channel
Why Block 4B's rice yield came in 14% under target
What's included: Evaluate at a glance
Ways into a test set
- Upload a JSONL test set
- Build it in the dashboard
- Auto-generated by Nucleus
- Hold out a slice of your training data
Scoring
- Criteria with a pass threshold
- A domain score per checkpoint
- Four metrics: exact_match, rouge, pass_rate, grader
- Baseline comparison with compare_to
- Accuracy, speed and cost side by side
Runs and results
- Asynchronous runs, 202 Accepted with an operation ID
- Poll the operation or the evaluation until it succeeds
- score, baseline_score and n on the response
- Any checkpoint by its nucleus:// path
Outcome benchmarks
- The system of record connected
- The outcome defined
- Every checkpoint scored against real results
- Drift over time
Evaluations billed on training meters
Evaluation runs are metered as token consumption on the same meters as everything else, so the bill stays on three quantities: training tokens, inference tokens and storage.
Evaluations
Metered as token consumption, on the same meter your fine-tuning jobs run on.
Storage
Datasets, checkpoints and exported weights are metered by the GB-hour, so you only pay while you keep them.
Pick your path
Evaluations
Score every checkpoint
Outcome benchmarks
Business results as the test set
Read on
Evaluations API reference
Create an evaluation, pass compare_to, and read score beside baseline_score.
Managed fine-tuning
Seven calls from a JSONL dataset to a served private model, with the evaluation at step five.
National Neuroscience Institute
An immersive VR training environment for brain aneurysm surgery, with comprehensive performance analytics.
Put a number on your next checkpoint
Read the Evaluations reference, or see how evaluation runs are metered.