Use datasets in evaluations

Use datasets as the data source for evaluations. You can pass datasets directly or convert experiment results into dataset format.

Pass datasets directly

Pass datasets directly to Eval():

import { initDataset, Eval } from "braintrust";
import { Levenshtein } from "autoevals";

Eval("Say Hi Bot", {
  data: initDataset("My App", { dataset: "My Dataset" }),
  task: async (input) => {
    return "Hi " + input;
  },
  scores: [Levenshtein],
});

from autoevals import Levenshtein
from braintrust import Eval, init_dataset

Eval(
    "Say Hi Bot",
    data=init_dataset(project="My App", name="My Dataset"),
    task=lambda input: "Hi " + input,
    scores=[Levenshtein],
)

Assign to environments

Environments are attached to dataset snapshots, not datasets directly. To assign a snapshot to an environment:

Go to Datasets.
Open the dataset.
Click Snapshots in the toolbar.
Find the snapshot you want to assign, then select Environments.
Toggle the environments in the submenu.

You can also tag and untag environments from Loop when working on a dataset page.

# 1. Create a snapshot
curl -X POST https://api.braintrust.dev/v1/dataset_snapshot \
  -H "Authorization: Bearer $BRAINTRUST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "DATASET_ID",
    "name": "prod snapshot",
    "xact_id": "DATASET_VERSION_OR_XACT_ID"
  }'

# 2. Upsert the environment association
curl -X PUT https://api.braintrust.dev/environment-object/dataset/DATASET_ID/ENVIRONMENT \
  -H "Authorization: Bearer $BRAINTRUST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "object_version": "SNAPSHOT_XACT_ID"
  }'

Once assigned, load the dataset for that environment in your evals:

import { Eval, initDataset } from "braintrust";

// Load by environment
Eval("My App", {
  data: initDataset({ project: "My App", dataset: "My Dataset", environment: "production" }),
  task: async (input) => { /* ... */ },
  scores: [],
});

// Load by version name
Eval("My App", {
  data: initDataset({ project: "My App", dataset: "My Dataset", version_name: "version 1" }),
  task: async (input) => { /* ... */ },
  scores: [],
});

// Load by xact_id (less readable, but precise)
Eval("My App", {
  data: initDataset({ project: "My App", dataset: "My Dataset", version: "8234923849293849..." }),
  task: async (input) => { /* ... */ },
  scores: [],
});

from braintrust import Eval, init_dataset

# Load by environment
Eval(
    "My App",
    data=init_dataset(project="My App", name="My Dataset", environment="production"),
    task=lambda input: ...,
    scores=[],
)

# Load by version name
Eval(
    "My App",
    data=init_dataset(project="My App", name="My Dataset", version_name="version 1"),
    task=lambda input: ...,
    scores=[],
)

# Load by xact_id (less readable, but precise)
Eval(
    "My App",
    data=init_dataset(project="My App", name="My Dataset", version="8234923849293849..."),
    task=lambda input: ...,
    scores=[],
)

Convert experiment results

Convert experiment results into dataset format using asDataset()/as_dataset(). This is useful for iterative improvement workflows where you want to use the results of one experiment as the baseline for future experiments:

import { init, Eval } from "braintrust";
import { Levenshtein } from "autoevals";

const experiment = init("My App", {
  experiment: "my-experiment",
  open: true,
});

Eval<string, string>("My App", {
  data: experiment.asDataset(),
  task: async (input) => {
    return `hello ${input}`;
  },
  scores: [Levenshtein],
});

from autoevals import Levenshtein
from braintrust import Eval, init

experiment = braintrust.init(
    project="My App",
    experiment="my-experiment",
    open=True,
)

Eval(
    "My App",
    data=experiment.as_dataset(),
    task=lambda input: f"hello {input}",
    scores=[Levenshtein],
)

Next steps

Run evaluations for the full evaluation workflow.
Track performance to see how dataset rows perform across experiments.

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Use datasets in evaluations

Pass datasets directly

Assign to environments

Convert experiment results

Next steps

​Pass datasets directly

​Assign to environments

​Convert experiment results

​Next steps

Pass datasets directly

Assign to environments

Convert experiment results

Next steps