Remote evals and sandboxes

Documentation Index

Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

• Remote evals — Run evals on your own infrastructure, controlled from Braintrust. Your evaluation code runs on your machine or server. The Braintrust playground triggers execution, sends parameters, and displays results.
• Sandboxes — Run evals in an isolated cloud sandbox, controlled from Braintrust. You push an execution artifact (a code bundle or container snapshot) and Braintrust invokes it on demand from the playground. No server to keep running.
  Sandboxes are in beta and the API, configuration, and behavior are likely to change in the near future. Requires a Pro or Enterprise plan. Self-hosted deployments require data plane version v2.0.

​

Common use cases

​

Run a remote eval

​

1. Write your eval

# pnpm
pnpm add braintrust openai autoevals
# npm
npm install braintrust openai autoevals

import { Eval, wrapOpenAI } from "braintrust";
import OpenAI from "openai";
import { z } from "zod";

const client = wrapOpenAI(new OpenAI());

Eval("my-project", {
  data: [{ input: "hello", expected: "HELLO" }],
  task: async (input, { parameters }) => {
    const completion = await client.chat.completions.create({
      ...parameters.main.build({ input }),
      model: parameters.model,
    });
    return completion.choices[0].message.content ?? "";
  },
  scores: [],
  parameters: {
    model: {
      type: "model",
      description: "Model to evaluate",
      default: "gpt-5-mini",
    },
    main: {
      type: "prompt",
      name: "Main prompt",
      description: "The prompt used to process input",
      default: {
        messages: [{ role: "user", content: "{{input}}" }],
        model: "gpt-5-mini",
      },
    },
    prefix: z.string().describe("Optional prefix to prepend to input").default(""),
  },
});

​

2. Expose the eval server

bt eval path/to/eval.ts --dev

bt eval path/to/eval.py --dev

# Requires Braintrust Ruby SDK v0.2.1+
require "braintrust"
require "braintrust/server"
require "openai"

Braintrust.init(blocking_login: true)
Braintrust.instrument!(:openai)

client = OpenAI::Client.new(api_key: ENV.fetch("OPENAI_API_KEY"))

simple_eval = Braintrust::Eval::Evaluator.new(
  task: ->(input:) {
    response = client.chat.completions.create(
      model: "gpt-5-mini",
      messages: [{role: "user", content: input}]
    )
    response.choices.first.message.content
  },
  scorers: [
    Braintrust::Scorer.new("exact_match") { |expected:, output:| output == expected ? 1.0 : 0.0 }
  ]
)

run Braintrust::Server::Rack.app(
  evaluators: {"simple-eval" => simple_eval}
)

# Gemfile
gem "rack"
gem "puma"

bundle install
bundle exec rackup eval_server.ru -p 8300 -o 0.0.0.0

# config/initializers/braintrust_server.rb
Braintrust::Contrib::Rails::Server::Engine.configure do |config|
  config.auth = :none
end

​

3. Configure in your project

1. In your project, go to Settings > Remote evals.
2. Select Create remote eval source.
3. Enter the name and URL of your remote eval server.
4. Select Create remote eval source.

​

4. Run from a playground

1. Open a playground in your project.
2. Select + Task.
3. Choose Remote eval from the task type list.
4. Select your eval and configure parameters using the UI controls.
5. Provide data inline or select a dataset, optionally add scorers, and click Run.

​

Demo

​

Run a sandbox eval

• Lambda — AWS Lambda-based. The default for braintrust push. Supports both Python and TypeScript. No extra configuration needed.
• Modal — Container-based via Modal. Requires a pre-built Modal container image. Executes TypeScript evals only.

​

1. Write your eval

# pnpm
pnpm add braintrust openai autoevals
# npm
npm install braintrust openai autoevals

import { Eval, wrapOpenAI } from "braintrust";
import OpenAI from "openai";
import { z } from "zod";

const client = wrapOpenAI(new OpenAI());

Eval("my-project", {
  data: [{ input: "hello", expected: "HELLO" }],
  task: async (input, { parameters }) => {
    const completion = await client.chat.completions.create(
      parameters.main.build({ input }),
    );
    return completion.choices[0].message.content ?? "";
  },
  scores: [],
  parameters: {
    main: {
      type: "prompt",
      name: "Main prompt",
      description: "The prompt used to process input",
      default: {
        messages: [{ role: "user", content: "{{input}}" }],
        model: "gpt-5-mini",
      },
    },
    prefix: z.string().describe("Optional prefix to prepend to input").default(""),
  },
});

​

2. Register your sandbox

braintrust push my_eval.py           # Python
npx braintrust push my_eval.eval.ts  # TypeScript

braintrust push my_eval.py --requirements requirements.txt

npx braintrust eval my_eval.eval.ts --push

braintrust push my_eval.py --if-exists replace           # Python
npx braintrust push my_eval.eval.ts --if-exists replace  # TypeScript

​

3. Run from a playground

1. Open a playground in your project.
2. Select + Task.
3. Open the Remote eval submenu and select your sandbox.
4. Select your eval and configure parameters using the UI controls.
5. Provide data inline or select a dataset, optionally add scorers, and click Run.

​

Limitations

• The dataset defined in your eval is ignored when running from the playground. Datasets are managed through the playground.
• Scorers defined in your eval are concatenated with scorers added in the playground.
• For Lambda sandboxes, each individual invocation is capped at 15 minutes, but Braintrust manages dataset iteration outside the Lambda, so large dataset evals are not constrained by that limit.
• For Modal sandboxes, each sandbox is capped at a 60-minute lifetime by default. Self-hosted deployments can override this with the MODAL_SANDBOX_TIMEOUT_S environment variable (in seconds).

​

Next steps

• Test prompts and models without custom code
• Create parameters to manage configurable settings
• Interpret results from your experiments