Our read
The path has four moves: find the faulty trace, copy it with a correct expected answer into a dataset, replay that dataset on the fixed version, then let online evaluators watch production. LangSmith supplies tracing, evaluators and alerts; your team supplies the right answer, the choice of evaluators and the threshold. Base traces are kept for 14 days, so copy the trace into a dataset as soon as you diagnose it (an editorial inference).
Start from the wrong answer, not from a prompt edit
Take a hypothetical case, offered as an editorial example: a support agent quotes the wrong refund window. A customer reports it, or a dashboard surfaces a low feedback score. The instinct is to edit the prompt. The better first move is to find the exact run that produced the answer.
In LangSmith, a trace is one execution of your application and can hold many steps, such as model calls and other tracked events. For chat agents, message threading lets you reread earlier turns. Instrumentation uses the Python, TypeScript, Go or Java SDKs, or OpenTelemetry. The vendor states that traces are sent asynchronously and do not slow the application.
Read the trace before blaming the model
The trace shows what the agent did step by step, with latency, cost and response quality. In our case the useful question is whether the wrong window came from a badly retrieved document, a tool called with wrong parameters, or the model itself. The fix, and therefore the test, differs for each. Insights clusters traces by topic and failure mode, which helps you see whether the defect is isolated or recurring.
If traces contain customer data, hosting matters. The managed cloud stores data in GCP us-central-1. BYOC and self-hosted options exist, and Enterprise can run on your own Kubernetes cluster. The vendor says it does not train on your data.
Copy the trace into a dataset before it expires
The pricing grid lists dataset collection and an annotation queue for human feedback. The product pages do not detail the clicks that turn a trace into an example, so confirm that step in the documentation or a demo.
Retention sets the pace. Base traces are kept 14 days and extended traces 180 days, and a base trace can be upgraded to extended for an extra fee. The vendor presents extended traces as useful when they carry feedback from users, evaluators or labelers. Our editorial inference: if the investigation runs past two weeks, the trace may be gone. Copy the input and the faulty output into a dataset at diagnosis.
One step no tool does for you is writing the expected answer. LangSmith does not know your refund window is 30 days. Someone who knows the policy has to decide, and the annotation queue exists to collect that judgment. The Engine product, billed in LSUs, can also create datasets for human review; the review stays human.
Replay: compare the faulty version with the fix
The pricing grid lists offline evals, a Playground and a Prompt Hub. This is an editorial recommendation, not a step-by-step capability described by the vendor: run the dataset on the current version, then on the fixed one. The example you added should move from fail to pass, and the other examples should not get worse.
A one-example dataset proves little. Add neighbouring questions (other windows, other products) and cases that were already correct. A code evaluator fits anything checkable mechanically, such as the exact number of days. An LLM judge suits tone or phrasing, but it is one more model to check: compare its verdicts with a human reviewer's on a few examples before you trust it.
Watch production for what the test set misses
After the fix ships, monitoring takes over. LangSmith offers online evaluators, LLM-as-judge or code, applied to production runs, plus tool and agent trajectory monitoring. Dashboards track tokens, P50 and P99 latency, errors, cost and feedback scores, with webhook or PagerDuty alerts when a metric crosses a threshold.
The threshold is a team decision: at what failure level does someone get paged? Editorial recommendation: test that an online evaluator's score really triggers a webhook or PagerDuty alert before you rely on it. Tuned Evaluators, in public beta for Plus and Cloud Enterprise plans in the US, cost 0.015 LSU per successful evaluation.
What the plan imposes, and what stays with your team
Developer costs $0 with one seat and 5,000 base traces a month. Plus costs $39 per seat with 10,000 base traces, then pay as you go at 0.005 LSU per extra trace, with 1 LSU equal to $1. Because a trace also covers evaluator and Playground runs, check how your online evals count toward volume. You can start with tracing alone and add evals later.
Cisco, Toyota, Salesforce, Orange Business and Numan appear on the vendor's customer page. Their story titles point to a platform centralizing observability, an andon-style board for agents, standardized agent evals, live intelligence built on LangChain and LangSmith, and an eval-first health coach.
Method and limits
2026-10-30
This dated snapshot counts distinct companies in public customer evidence. CONFIRMED means a case study, named testimonial or press source; DECLARED means the company itself told usedby; OBSERVED means a vendor listing; SIGNAL means a job post. All four count, but their strength differs. Removed and unlevelled lines are excluded. Examples are a capped selection verified within 90 days. Vendor facts are sourced records checked within 60 days. Coverage is uneven and missing evidence is unknown. These figures measure neither market share nor product performance; capture dates are not adoption dates.
