OpenAI Agents · beta

OpenAI Agents observability beyond a green agent

A finished OpenAI Agents run means the agent completed. It does not mean the product job succeeded. Observability has to cover both stories on the same run.

What ran vs what it meant

OpenAI Agents compose an agent, tools, and a trace processor. Spans show which tools ran and that the agent loop finished. That layer is useful, but it is only half of observability.

You also need a clear product goal: did the job you named actually hold for the user? Without that question, a green agent can hide an empty or wrong answer behind healthy spans.

Think in layers on one run identity: tracing for spans, evaluation for quality criteria, cost for spend, and outcomes for “was the job done?” Keep those layers joined. Do not leave the outcome layer only in a chat transcript or a spreadsheet.

What this means in practice

Picture one OpenAI Agents run from start to finish. You review the same run twice: once for agent and tool execution, once for the product job you promised the user.

First you inspect tracing. The agent finished, tools returned, and no exception fired. Spans look healthy — yet that only says the agent completed its loop.

Then you ask the product question. Is there a usable answer for the named job, and is it correct enough to ship? If those evaluation checks fail, the run failed for the user even when the agent looks green.

Attach cost to that same run identity. Token spend and tool spend matter more when you already know whether the goal passed. A cheap miss and an expensive miss need different reviews, but both need the goal marked clearly.

Write the success rule once next to the agent code so every engineer scores the same run the same way. That is OpenAI Agents observability beyond a green status light.

Example: false success

The agent can finish. Tools can return. No exception may fire. The answer can still be empty or wrong for the named job. That gap is a false success.

Operations may trust the green execution path while product still sees a failed job. Without a named goal on the run identity, those two readings never meet.

A finished agent span is not the same as a useful answer, and a clean tool return is not the same as the right result for the job you named.

A product goal is a short shared rule for “done.” Evidence is the structured fields that prove pass or fail. Keep both beside OpenAI Agents spans on the same run.

Then a review can ask a fair question: did we spend money on a real win, or on a clean agent that still missed the product job?

Instrument the agent run

Use one integration point: wrap the run, keep the native OpenAI Agents trace processor, and attach outcome meaning on the data you get back.

The snippet below reports whether an answer is present. That is a simple instrumentation start. You can add richer evaluation checks later without changing the wrap shape.

from witdem_sdk.integrations.openai_agents import instrument

observed_run = instrument(
    run,
    report_result=lambda answer: {
        "result": "completed" if answer else "unresolved",
        "result_valid": bool(answer),
        "requirements": {"non_empty_answer": bool(answer)},
        "metrics": {"answer_characters": len(answer)},
    },
)
print(observed_run())

Declare success next to the code

Put the product goal in the repository so every engineer scores the same run with the same shared rule.

A contract names the artifact and the goal. Pass means the answer field is present. The teaching YAML below shows that shape for readability. The live OSS example uses contract v2 with a shared useful-answer rule — same idea, declared next to the agent code rather than only in a dashboard toggle.

version: 1
service:
  name: openai-agents-answer
  runtime: openai_agents
contracts:
  useful_answer:
    artifact:
      name: Answer
      valid:
        non_empty: $.answer
    product_goal:
      name: Non-empty answer
      achieved:
        all:
          - $.witdem.artifact_valid

What to look for on the run

Goal achievement, evidence fields, and spend should share one run identity beside OpenAI Agents spans. Do not replace spans with a lone outcome score.

When you open a run, you want the execution path and the product outcome together. That is how you debug a false success without guessing across separate tools.

Run the example

Start from the basic agent example in the open-source repository. Wire the instrument helper, declare a product goal, and review one run end to end.

Read the OpenAI Agents integration docs if you need the API shape. Keep the native trace processor in place for agent spans, and add the outcome layer beside it rather than instead of it. The integration is in beta — expect the wrap shape to stay while details continue to tighten.

FAQ

Does a green OpenAI Agents run mean success? No. Green means the agent finished executing. Success means the product goal you named held for that specific run.

Do I replace the native trace processor? No. Keep native tracing for spans. Attach outcome fields next to them on the same run so execution and goals stay complementary.

What is a product goal for an agent? A short shared rule for “done,” such as a non-empty answer or a correct answer for the named job. Evidence is the structured fields that prove pass or fail.

Is the OpenAI Agents integration stable? It is beta. The instrument wrap and goal-on-run idea are the durable story; APIs may still tighten as examples mature.

Related

Agentic workflow observability · Haystack · LangGraph · LangChain