xAGI LABS / DATA PROGRAMS
Expert data for model evaluation
Turn a domain task into a reviewable rubric, representative examples and a scoped evaluation program.
Choose the decisions to evaluate
Start with a model failure or business workflow, then define what a useful answer or action would look like. Domain-specific evaluation may require source checking, factual reasoning, tool use or a sequence of decisions. Identify the context available to the model and the material it must not see. Generic preference scores alone may not explain why a system fails.
Write a rubric before collecting judgments
Specify the scoring dimensions, evidence requirements and treatment of partial correctness, uncertainty and unsafe actions. Define reviewer qualifications and a process for resolving disagreements. Plan calibration examples and checks for inconsistent labels. Any specialist data program is subject to actual expert availability and agreed review methods; this page does not imply an existing expert workforce.
Keep evaluation useful over time
Agree training and holdout boundaries, duplicate checks, versioning and criteria for refreshing examples. Track failure categories that can lead to a targeted collection or annotation task. Start with a small evaluation design and measure whether the rubric distinguishes relevant model behavior before committing to a larger dataset.
A worked evaluation example
Illustrative planning template, version 1.0 · 17 October 2026. These examples are not a dataset listing, measured project results or a delivery commitment.
Task: answer whether a source permits model training. Evidence: the supplied license excerpt permits internal evaluation only. Constraint: do not infer rights that are not stated.
| Dimension | Score 0 | Score 1 | Score 2 |
|---|---|---|---|
| Grounded correctness | 0: contradicted or unsupported | 1: partly supported; material omission | 2: supported and addresses the task |
| Constraint adherence | 0: violates an explicit constraint | 1: follows some constraints | 2: follows all stated constraints |
| Uncertainty | 0: invents missing information | 1: notes uncertainty without a next step | 2: identifies missing evidence and a useful next step |
Example A: “Yes, it is public, so you can train on it.” Scores 0 / 0 / 0: it contradicts the evidence and invents permission. Example B: “Training permission is not established. The excerpt permits internal evaluation only; obtain the complete terms or written permission before deciding.” Scores 2 / 2 / 2 under this illustrative rubric. This is a scoring exercise, not legal advice.
Before production labeling, have qualified reviewers independently score calibration examples, record disagreements, agree adjudication rules and re-check a held-out sample. Report agreement and failure categories rather than promising accuracy from a rubric alone.
Download the worked rubric (Markdown) →
Prepared by the xAGI Labs team for project scoping. A project-specific domain reviewer and acceptance criteria are agreed before collection or labeling.
Start with a useful conversation.
Tell us the model task or source you want to explore. We establish feasibility, quality and rights before discussing delivery.
Book a scoping call ↗