LangChain · beta

LangChain observability beyond a green chain

A finished LangChain invoke means the runnable returned. It does not mean the product job succeeded. Observability has to cover both stories on the same run.

What ran vs what it meant

LangChain chains compose runnables, prompts, models, and parsers. Callbacks and tracers show which steps executed across the runnable graph. That layer is useful, but it is only half of observability.

You also need a clear product goal: did the job you named actually hold for the user? Without that question, a green chain can hide an empty or ungrounded answer behind healthy spans.

Think in layers on one run identity: tracing for spans, evaluation for quality criteria, cost for spend, and outcomes for “was the job done?” Keep those layers joined. Do not leave the outcome layer only in a chat transcript or a spreadsheet.

What this means in practice

Picture one LangChain chain invoke from start to finish. You review the same run twice: once for runnable execution, once for the product job you promised the user.

First you inspect tracing. The prompt prepared input, the model returned tokens, and the parser finished cleanly. Spans look healthy — yet that only says the chain completed its runnable steps.

Then you ask the product question. Is there a usable answer, and is it grounded in the context you fed the model? If those evaluation checks fail, the run failed for the user even when HTTP status looks healthy.

Attach cost to that same run identity. Token spend and tool spend matter more when you already know whether the goal passed. A cheap miss and an expensive miss need different reviews, but both need the goal marked clearly.

Write the success rule once next to the chain code so every engineer scores the same run the same way. That is LangChain observability beyond a green status light.

Example: false success

The prompt, model, and parser can all finish. The invoke can return HTTP 200. The answer can still be empty or ungrounded. That gap is a false success.

Operations may trust the green execution path while product still sees a failed job. Without a named goal on the run identity, those two readings never meet.

A finished model span is not the same as a useful answer, and a clean parser result is not the same as grounded text the user can trust.

A product goal is a short shared rule for “done.” Evidence is the structured fields that prove pass or fail. Keep both beside LangChain spans on the same run.

Then a review can ask a fair question: did we spend money on a real win, or on a clean chain that still missed the product job?

Instrument the runnable

Use one integration point: wrap the chain, keep LangChain’s callbacks and tracers, and attach outcome meaning on the data you get back.

The snippet below reports whether an answer is present. That is a simple instrumentation start. You can add richer evaluation checks later without changing the wrap shape.

from witdem_sdk.integrations.langchain import instrument

chain = instrument(
    build_chain(),
    report_result=lambda answer: {
        "result": "completed" if answer else "unresolved",
        "result_valid": bool(answer),
        "requirements": {"non_empty_answer": bool(answer)},
        "metrics": {"answer_characters": len(str(answer))},
    },
)
print(chain.invoke("What is observability?"))

Declare success next to the code

Put the product goal in the repository so every engineer scores the same run with the same shared rule.

A contract names the artifact and the goal. Pass means the answer field is present. The teaching YAML below shows that shape for readability. The live OSS example uses contract v2 with a shared useful-answer rule — same idea, declared next to the chain code rather than only in a dashboard toggle.

version: 1
service:
  name: langchain-answer
  runtime: langchain
contracts:
  useful_answer:
    artifact:
      name: Answer
      valid:
        non_empty: $.answer
    product_goal:
      name: Non-empty answer
      achieved:
        all:
          - $.witdem.artifact_valid

What to look for on the run

Goal achievement, evidence fields, and spend should share one run identity beside LangChain spans. Do not replace spans with a lone outcome score.

When you open a run, you want the execution path and the product outcome together. That is how you debug a false success without guessing across separate tools.

Run the example

Start from the runnable-pipeline example in the open-source repository. Wire the instrument helper, declare a product goal, and review one run end to end.

Read the LangChain integration docs if you need the API shape. Keep tracing in place for runnable spans, and add the outcome layer beside it rather than instead of it. The integration is in beta — expect the wrap shape to stay while details continue to tighten.

FAQ

Does a green LangChain invoke mean success? No. Green means runnables finished executing. Success means the product goal you named held for that specific run.

Do I replace LangChain callbacks or tracers? No. Keep native tracing for spans. Attach outcome fields next to them on the same run so execution and goals stay complementary.

What is a product goal for a chain? A short shared rule for “done,” such as a non-empty answer or a grounded answer check. Evidence is the structured fields that prove pass or fail.

Is the LangChain integration stable? It is beta. The instrument wrap and goal-on-run idea are the durable story; APIs may still tighten as examples mature.

Related

Agentic workflow observability · Haystack · LangGraph · OpenAI Agents