Know how good your model really is
Upload a test set, build one in the dashboard, or let Nucleus generate it. Every checkpoint gets a domain score against your criteria and your pass threshold. The best one is named.
Bring your benchmarks
Upload a domain-specific test set and compare accuracy, speed and cost across checkpoints.
Build in the dashboard
Define the questions, the criteria and the pass threshold in the dashboard, with no code to write.
Auto-generated benchmarks
Nucleus generates domain-specific tests and scores each checkpoint against them.
A before-and-after number
Score the base model on the same set in the same run, and the delta from fine-tuning reads straight off the response.
Score against a test set, or against the results you record
Evaluations score a checkpoint against a benchmark set you upload, build or generate. Outcome benchmarks score it against the results your systems already record.
An evaluation runs a checkpoint over your test set and scores the outputs, giving you one number for how well it performs. Set the criteria and the pass threshold once, and every checkpoint from a run is scored against them. Pass compare_to and the base model is scored on the same set in the same run, so the delta from fine-tuning is read straight off the response.
Results from LLMs powered by Nucleus
prediction accuracy on equipment failures
Predictive Maintenance
yield shortfall traced to one cause
Agriculture Yield Analysis
monthly revenue misattributed
Ad Performance Tracking
average time to resolution
Critical Incident Response
Where a measurement changed the decision
Line 3's next failure, booked into the maintenance window
$73.4K a month was being credited to the wrong channel
Why Block 4B's rice yield came in 14% under target
What's included: Evaluate at a glance
Ways into a benchmark
- Upload a JSONL test set, purpose evaluation
- Build it in the dashboard, no code
- Generated by Nucleus from a run or a description
- Connect a system of record as an outcome source
Scoring
- Criteria with a case count, and one pass threshold
- Four metrics: exact_match, rouge, pass_rate, grader
- Baseline comparison with compare_to
- score on 0 to 1, with the cases scored
Runs and results
- Pass a training_run to score every checkpoint at once
- Asynchronous runs, 202 Accepted with an operation ID
- best_checkpoint names the highest scoring one
- Any single model by its nucleus:// path
Over time
- The system of record connected
- The outcome field defined
- Every checkpoint scored against real results
- List by benchmark for drift across releases
Evaluations billed on training meters
Evaluation runs are metered as token consumption on the same meters as everything else, so the bill stays on three quantities: training tokens, inference tokens and storage.
Evaluations
Metered as token consumption, on the same meter your fine-tuning jobs run on.
Storage
Datasets, checkpoints and exported weights are metered by the GB-hour, so you only pay while you keep them.
Pick your path
Evaluations
Score every checkpoint
Outcome benchmarks
Business results as the test set
Read on
Evaluations
Create an evaluation, pass compare_to, and read score beside baseline_score.
Managed fine-tuning
Seven calls from a JSONL dataset to a served private model, with the evaluation at step five.
National Neuroscience Institute
An immersive VR training environment for brain aneurysm surgery, with comprehensive performance analytics.
Put a number on your next checkpoint
Read the Evaluations reference, or see how evaluation runs are metered.