LangGraph

LangGraph observability beyond a green graph

A finished LangGraph run means nodes executed. It does not mean the product job succeeded. Observability has to cover both stories on the same run identity.

What ran vs what it meant

LangGraph graphs compose nodes, edges, and shared state. Tracing shows which nodes fired and how state moved. That is useful, but it is only half the picture.

You also need a clear product goal. Did the job you named actually hold for the user? Without that question, a green graph can hide an empty or ungrounded answer in final state.

Think in layers on one run: tracing for node spans, evaluation for quality checks, cost for spend, and outcomes for “was the job done?” Keep them joined on one identity. Do not leave the last layer only in LangSmith or a spreadsheet.

What this means in practice

Picture one LangGraph support-routing run from start to finish. You walk the same run twice: once for nodes and state updates, once for the job you promised the user.

First you look at tracing. Classify ran, retrieve ran, and answer returned. Spans look healthy. That only says the graph finished its chosen path.

Then you ask the product question. Is there a usable answer in state, and is it grounded in what you retrieved? If those checks fail, the run failed for the user — even when the graph looks healthy at runtime.

Attach cost to that same run identity. Tokens and tool spend matter more when you know whether the goal passed. A cheap miss and an expensive miss need different reviews, but both need the goal marked clearly.

Write the success rule once next to the graph code. Every engineer should score the same run the same way. That is LangGraph observability beyond a green status light.

Example: false success

Every node can finish and state updates can apply. The graph can return HTTP 200. The final state can still hold an empty or ungrounded answer. That gap is a false success.

Ops may trust the green execution path. Product still sees a failed job. Without a named goal on the run, those two views never meet in one review.

A finished graph is not the same as a successful job. Node spans show the path taken. They do not prove the answer field the user actually needed.

A product goal is a short shared rule for “done.” Evidence is the structured fields that prove pass or fail. Keep both beside LangGraph node spans on the same run.

Then a review can ask a fair question: did we spend money on a real win, or on a clean graph execution that still missed the job?

Instrument the graph

Use one integration point. Wrap the compiled graph, keep LangGraph’s native tracing, and attach meaning on the state you get back.

The snippet below reports whether an answer is present in state. That is a simple starting point. You can add richer evaluation checks later without changing the wrap shape.

from witdem_sdk.integrations.langgraph import instrument

graph = instrument(
    build_graph(),
    report_result=lambda state: {
        "result": "completed" if state.get("answer") else "unresolved",
        "result_valid": bool(state.get("answer")),
        "requirements": {"non_empty_answer": bool(state.get("answer"))},
    },
)

Declare success next to the code

Put the product goal in the repository. Then every engineer scores the same run with the same shared rule.

A contract names the artifact and the goal. Pass means the answer field is present and valid. The rule lives in git with the graph, not only behind a dashboard toggle.

version: 1
service:
  name: langgraph-answer
  runtime: langgraph
contracts:
  useful_answer:
    artifact:
      name: Answer
      valid:
        non_empty: $.answer
    product_goal:
      name: Non-empty answer
      achieved:
        all:
          - $.witdem.artifact_valid

What to look for on the run

Goal achievement, evidence fields, and spend should share one run identity. Keep them beside node spans. Do not replace detailed spans with a lone aggregate score.

When you open a run, you want the execution path and the outcome together. That is how you debug a false success without guessing across separate tools.

Run the example

Start from the state-graph example in the open-source repository. Wire the instrument helper, declare a product goal, and review one run end to end.

Read the LangGraph integration docs if you need the API shape. Keep tracing in place for node spans. Add the outcome layer beside it rather than instead of it.

FAQ

Does a green LangGraph run mean success? No. Green means nodes finished executing. Success means the product goal you named held for that specific run.

Do I replace LangSmith or native tracing? No. Keep LangSmith or native tracing for node spans. Attach outcome fields next to them on the same run so they stay complementary.

What is a product goal for a graph? A short shared rule for “done,” such as a non-empty answer present in state. Evidence is the structured fields that prove pass or fail.

Where should the rule live? In the repository next to the graph definition, so reviews reuse one definition instead of inventing “good” inside each separate tool.

Related

Agentic workflow observability · Haystack observability · LangChain observability · OpenAI Agents observability