Haystack

Haystack observability beyond a green pipeline

A finished Haystack run means components executed. It does not mean the product job succeeded. Observability has to cover both stories on the same run.

What ran vs what it meant

Haystack pipelines compose retrievers, generators, and tools. Tracing shows which components fired. That is useful, but it is only half the picture.

You also need a clear product goal. Did the job you named actually hold? Without that question, a green pipeline can hide an empty or ungrounded answer.

Think in layers on one run: tracing for spans, evaluation for quality checks, cost for spend, and outcomes for “was the job done?” Keep them joined. Do not leave the last layer in a chat or a sheet.

What this means in practice

Picture one Haystack RAG run from start to finish. You walk the same run twice: once for components, once for the job you promised the user.

First you look at tracing. Retrievers ran. The generator returned. Spans look fine. That only says the pipeline finished its steps.

Then you ask the product question. Is there a usable answer? Is it grounded in the retrieved context? If those checks fail, the run failed for the user — even when HTTP status looks healthy.

Attach cost to that same run identity. Tokens and tool spend matter more when you know whether the goal passed. A cheap miss and an expensive miss are different reviews, but both need the goal marked.

Write the rule once next to the code. Every engineer should score the same run the same way. That is Haystack observability beyond a green status light.

Example: false success

Both retrievers can return hits. The generator can return HTTP 200. The answer can still be empty or ungrounded. That gap is a false success.

Ops may trust the green path. Product still sees a failed job. Without a named goal on the run, those two views never meet.

Hits from a retriever are not the same as a useful answer. A finished generator span is not the same as grounded text the user can trust.

A product goal is a short shared rule for “done.” Evidence is the fields that prove pass or fail. Keep both beside Haystack component spans.

Then a review can ask a fair question: did we spend money on a real win, or on a clean pipeline that still missed the job?

Instrument the pipeline

Use one integration point. Wrap the pipeline. Keep Haystack’s native tracer. Attach meaning on the data you get back.

The snippet below reports whether an answer is present. That is a simple start. You can add richer checks later without changing the wrap shape.

from witdem_sdk.integrations.haystack import instrument

pipeline = instrument(
    build_pipeline(...),
    report_result=lambda result: {
        "result": "completed" if result.get("answer", {}).get("answer") else "unresolved",
        "result_valid": bool(result.get("answer", {}).get("answer")),
        "requirements": {"non_empty_answer": bool(result.get("answer", {}).get("answer"))},
    },
)

Declare success next to the code

Put the product goal in the repo. Then every engineer scores the same run with the same rule.

A contract names the artifact and the goal. Pass means the answer field is present. The rule lives in git with the pipeline, not only in a dashboard toggle.

version: 1
service:
  name: haystack-answer
  runtime: haystack
contracts:
  useful_answer:
    artifact:
      name: Answer
      valid:
        non_empty: $.answer
    product_goal:
      name: Non-empty answer
      achieved:
        all:
          - $.witdem.artifact_valid

What to look for on the run

Goal achievement, evidence fields, and spend should share one run identity. Keep them beside component spans. Do not replace spans with a lone score.

When you open a run, you want the path and the outcome together. That is how you debug a false success without guessing across tools.

Run the example

Start from the parallel-retriever example in the open-source repo. Wire the instrument helper, declare a goal, and review one run end to end.

Read the Haystack integration docs if you need the API shape. Keep tracing in place. Add the outcome layer beside it.

FAQ

Does a green Haystack pipeline mean success? No. Green means components finished. Success means the product goal you named held for that run.

Do I replace Haystack’s native tracer? No. Keep native tracing for component spans. Attach outcome fields next to them on the same run.

What is a product goal for RAG? A short shared rule for “done,” such as a non-empty answer or a grounded answer check. Evidence is the fields that prove pass or fail.

Where should the rule live? In the repo next to the pipeline, so reviews reuse one definition instead of inventing “good” in each tool.

Related

Agentic workflow observability · LangGraph observability · LangChain observability · OpenAI Agents observability