Our read
Use Braintrust to run the old and new prompt over the same dataset built from real traces, with code scorers for hard rules and an LLM judge for meaning. Then read the failures yourself. The published Box and Coursera examples show automatic scores missing partial credit and vague answers, so treat the score as a filter, not a release decision.
Why a prompt edit needs a test, not a spot check
A prompt change that fixes the bug in front of you can quietly break something else. Dropbox describes exactly this in its Braintrust case study: as more engineers worked on its Dash assistant, everyone edited the prompt after a bug report, each fix caused a new problem elsewhere, and spreadsheets stopped being workable.
Braintrust positions its evals as the answer: run experiments against real datasets, compare prompts and models side by side, and score outputs with LLMs, code or humans. The workflow below follows that shape. It is an editorial sequence built from the vendor's product description and three published customer stories, not a procedure Braintrust prescribes, and every outcome quoted comes from the vendors or customers themselves.
Step 1: build the test set from real traces
Braintrust says production traces can be turned into eval datasets with one click, so regression tests come from real failures and edge cases rather than invented examples. Coursera shows what careful curation looks like: the team reviews anonymized chatbot transcripts and human-graded assignments, pays particular attention to conversations with thumbs-up or thumbs-down feedback, and adds LLM-generated synthetic cases to probe edge conditions.
Editorial advice: include the cases the new prompt is meant to fix and also a sample of ordinary cases the old prompt already handled. A set made only of failures cannot show that the change broke something. Retention matters here. The pricing page lists 14 days of included retention on Starter and 30 days on Pro, so copy the traces you need into a dataset before they expire.
Step 2: run both versions on the same inputs
Run the current prompt and the candidate over the same dataset with the same scorers, and change only the prompt. Box used the same logic at larger scale: one dataset as a fixed baseline, parallel experiments across many models, each compared on accuracy, latency, cost, token use and tool use. Its team also runs evaluations programmatically overnight, so the morning starts with what regressed or improved.
Dropbox splits the work in two tiers: 150 targeted tests before a pull request merges, then a suite of more than 10,000 tests after merge. For your own pipeline, the integration list includes Pytest, OpenTelemetry, LangChain and the OpenAI Agents SDK, which suggests the run can sit in code you already have. How a prompt version is stored and promoted is not detailed in the pages we read, so try that on the free plan before you rely on it.
Step 3: choose scorers and know their blind spots
Code scorers are deterministic but only check what you wrote down. Box grades citations with pure code: expected file against cited file. Coursera uses heuristic checks for format and structure. Neither says whether the answer is useful.
LLM judges cover meaning, with their own weaknesses. Box feeds each rubric item to a judge separately, yet an all-or-nothing rule marked an answer with four of five expected facts, well explained, as a failure. Dropbox found that classic metrics such as ROUGE, BLEU and cosine similarity missed citations, grounding and tone, and moved to LLM judges that began broad and became more specific. Box also warns against having an LLM write an eval set and then judge it: the result can score perfectly and tell you nothing.
Step 4: read the failures yourself
Box found its partial-credit problem by tabbing through failures question by question and opening the trace logs. Coursera's grading prototype looked fine on valid submissions until the team saw that vague answers still earned high scores, which led it to add negative test cases. In both stories, a person reading examples exposed what the aggregate number hid.
Plan limits shape this step. Human review scores are limited to one per project on Starter and unlimited on Pro, which costs $249 per month; included scores are 10,000 on Starter and 50,000 on Pro. Braintrust also describes task-specific trace views for building annotation interfaces without frontend work, useful if reviewers need a layout built for your content.
Step 5: decide, and know what the score does not prove
A threshold is a team decision, not a property of the tool. Box gated internal release on scoring above 80% on a set of about eighty questions, then widened to a 330-question golden dataset for general availability. Early scores sat around sixty percent and that was acceptable because the failures were legible and could be sorted by tag.
A passing offline score covers only the questions you wrote. Box checks production: if negative responses cluster somewhere, it looks at whether the eval set shows the same drop under that tag, and if not, the set has a gap. Dropbox runs an online judge on sampled production queries with the same rubric. These are vendor-reported practices, not independent measurements, and they are different companies with different metrics, so do not read them as a benchmark between teams. Ship when the comparison, the human read and a production watch all agree.
Method and limits
2026-11-01
This dated snapshot counts distinct companies in public customer evidence. CONFIRMED means a case study, named testimonial or press source; DECLARED means the company itself told usedby; OBSERVED means a vendor listing; SIGNAL means a job post. All four count, but their strength differs. Removed and unlevelled lines are excluded. Examples are a capped selection verified within 90 days. Vendor facts are sourced records checked within 60 days. Coverage is uneven and missing evidence is unknown. These figures measure neither market share nor product performance; capture dates are not adoption dates.
