Layers
Tracing, evaluation, and product analytics
Three questions on the same agentic run — not three interchangeable tools.
Tools are not layers
Teams often treat tracing platforms, eval suites, and product dashboards as substitutes. They are not. Each answers a different question on the same run. Ranking vendors as if they compete for one slot hides the gap. You still need execution detail, quality signals, and a named job outcome. Together.
Tracing — what executed?
Tracing records the runtime path. You get spans for steps and tools. You get model calls, retries, timing, and errors. It answers whether the graph or pipeline ran as instrumented. A green trace is needed proof of execution. It does not, by itself, say the product job succeeded.
This is where most agent observability starts. Dedicated tracing products are strongest here. Keep that layer. Do not ask it to stand in for outcomes.
Evaluation — quality vs criteria?
Evaluation scores outputs against criteria. That can mean rubrics, golden datasets, offline suites, and online judges or scores. It answers how good the result looks vs a defined standard. A high eval score can still sit next to a failed product goal. That happens when the rubric and the job definition diverge.
Eval tooling belongs beside tracing. Neither layer says, in product terms, whether the run achieved the job you shipped for.
Product analytics — was the declared job done?
Product outcomes ask a third question. Did this run meet the goal you named, with evidence you can inspect? That means a goal next to the code. It means pass or fail tied to fields on the run. When useful, spend joins that same run id.
HTTP 200 can be true. Finished spans can be true. A solid rubric score can be true. The answer can still be empty, ungrounded, or outside the contract. Outcome analytics closes that gap. It does not replace traces or evals.
How they fit — a map, not a ranking
Think in layers on one run identity. Cost sits between execution and outcomes. Spend only means something when joined to what the run achieved. The diagram is a map of questions. It is not a leaderboard of tools.
Where Witdem sits
Witdem focuses on the outcomes layer. That means named product goals, evidence on the run, and spend joined to that job. Peers such as Langfuse, LangSmith, and Arize Phoenix often own tracing and much of evaluation. Use them for what they do well. Add outcomes on the same run. Do not treat any one product as the full stack.
Witdem does not replace a tracing or eval platform. The useful frame is complementary. Traces and scores show how the run behaved. Outcomes answer whether the shipped job held.
What this means in practice
Picture one agent run from start to finish. You ask three short questions on that same run. What executed? How good was it vs criteria? Was the named job done?
First comes tracing. You see steps, tools, models, and errors. A green path here only says the code finished.
Next comes evaluation. You score the output against criteria you chose. A high score still needs a job that you named up front.
Last comes the product outcome. You ask whether you met the goal you wrote down. You keep fields that show why. Join spend to that same run when cost reviews matter.
FAQ
Are tracing and product analytics the same? No. Tracing shows what ran. Product analytics asks whether the job you named was done. You need both on the same run.
Do eval scores replace a product goal? No. Eval scores check quality vs criteria. They do not, by themselves, say the shipped goal held.
Where does Witdem sit? On the outcomes layer: named goals, evidence, and spend×job. It sits beside tracing and evaluation.
Should I pick one tool for all layers? No. Keep strong tracing and eval tools. Add outcomes on the same run identity.
Related
Witdem and Langfuse · Witdem and LangSmith · Witdem and Arize Phoenix · Agentic workflow observability
Framework guides: Haystack · LangGraph · LangChain · OpenAI Agents