Benchmark your model on the outcomes your business already records
The results your CRM, ticketing or ERP already record become the test set. Every checkpoint is scored against them, so the score reads as a business outcome.
The test set is your history
Each row is a real case with the result your system wrote down for it.
The score is a field you already track
Choose the column that holds the outcome and the score is the share of cases the model got right.
Every checkpoint, against your threshold
Each checkpoint of a run is scored beside the base model, and the best one is named.
From a system of record to a score
Four steps, each one a thing you can see in the dashboard or read off the API.
Point Nucleus at the CRM, the ticketing queue or the ERP where the result is written down. One call to /v1/outcome-sources connects it, from the same catalogue of 95 systems the training connectors draw on, and every refresh writes a fresh evaluation file. Set it to hourly, daily or weekly and the cases keep themselves current.
Results from LLMs powered by Nucleus
monthly revenue misattributed
Ad Performance Tracking
downtime prevented this month
Predictive Maintenance
average time to resolution
Critical Incident Response
daily fuel saved across the fleet
Intelligent Fleet Optimisation
Where the outcome already lives in a system
Every incident past its four-hour SLA, escalated by name
Line 3's next failure, booked into the maintenance window
$73.4K a month was being credited to the wrong channel
Outcome benchmarks at a glance
Sources
- Any of the 95 systems in the connector catalogue
- CRM, ticketing and ERP as the system of record
- A JSONL evaluation file, when you already have one
- Refresh hourly, daily, weekly or on demand
The mapping
- case_field, what the model is asked about
- outcome_field, the result already recorded
- A filter to select which records are scored
- Every refresh writes a fresh set of cases
Scoring
- Metrics exact_match, rouge, pass_rate and grader
- Baseline comparison with compare_to
- One pass threshold, held on the benchmark
- score on 0 to 1, with the cases scored
Over time
- Pass a training_run to score every checkpoint
- best_checkpoint names the highest scoring one
- List by benchmark for the history in order
- Releases and retrains read on one line
Evaluations billed on training meters
Evaluation runs are metered as token consumption on the same meters as everything else. There is no separate charge type.
Evaluations
Metered as token consumption, on the same meter your fine-tuning jobs run on.
Storage
Benchmarks, datasets and checkpoints are metered by the GB-hour, so you only pay while you keep them.
More from Evaluate
Evaluations
Score every checkpoint
Evaluate
Everything in Evaluate, in one place.
Read on
Outcome sources
Connect the system of record and name the field that holds the result.
Benchmarks
Hold the cases, the criteria and the pass threshold on one set.
Automation workflows, powered by Nucleus
Nine scripted workflows and the figures each one produced.
Score your model on the results you already record
Bring the system that holds the outcome. We will connect it and show you the score.