Skip to main content
If your team runs its own AI agent and sends its traces to PostHog, Aide can read those conversations, judge whether each answer served the customer, group the failures into themes and show which tools failed. Connect PostHog under Integrations > PostHog. The reports are in AI Agent > Quality: pick your agent under From PostHog in the agent picker at the top.

​
Who can use it

It is available on every plan. Owners and admins see the Quality pages.

​
Connect PostHog

  1. Open Integrations and choose PostHog.
  2. Click Connect PostHog and sign in to PostHog. Aide asks for three read-only scopes: query:read, project:read and llm_analytics:read.
  3. If your PostHog sign-in can see more than one project, choose the one that holds your agent’s traces under Project.
To connect with a key instead, click Use a personal API key instead:
  1. Choose the Region: US or EU.
  2. Enter the Project ID. It is the number after /project/ in your PostHog address.
  3. In PostHog, create a personal API key with the same three scopes, limited to that project. Paste it under Personal API key.
  4. Click Connect with API key.

​
What Aide imports

  • The AI traces, generations and spans your agent sends to the PostHog project, with the metrics and feedback recorded on them.
  • Aide imports every 15 minutes, starting with the last 30 days.
  • Your agent’s system prompt is never stored.
  • A trace that is a single model call in a pipeline, such as a classification or a summary, is not a conversation. Aide leaves it out and never judges it.

​
What Aide judges

Aide judges two groups of conversations:
  • A random sample. A fixed share of conversations, a quarter by default, is judged at random. The Bad-response rate comes from this sample alone, so it is an honest estimate. It appears once 30 conversations in the period have been judged at random.
  • Conversations with a signal. Every conversation with a tool error, an error or negative feedback is judged too. These are left out of the bad-response rate.
A daily limit on judged conversations for each connection keeps the cost bounded.

​
Read the reports

Open AI Agent > Quality and pick your agent under From PostHog.
  • Overview shows Bad-response rate, Spend, Tokens, Conversations and Tool error rate for the Last 7 days or Last 28 days, next to the period before, so you can see whether a change helped. Below are Top themes and Tools that fail most.
  • Conversations lists your agent’s conversations with how each was judged. Filter by outcome, theme or tool.
  • Issues has two views. By theme groups bad responses that share a goal and a cause. Themes are rebuilt every Monday, and new conversations join them daily. Open a theme to see What would fix it: a tool to add and its inputs, a tool that keeps failing and its most common errors, or what the agent needed to know. By tool lists each tool with its Calls, Errors, Error rate and Most common errors. It appears when your agent’s traces include tool calls.
  • Evals works differently for your own agent. Aide can’t replay it, so it drafts tests for you to run. Open a theme in Issues, click Add to evals, then download the cases with JSON or CSV. Each case holds the conversation, what a good answer must do and what it must not do. Personal details are removed.
Learned changes is hidden for an agent from PostHog. It applies to Aide’s own agents only.

​
Organizations

If your agent’s events name a PostHog group, such as the company your customer works for, each conversation shows its organization. The overview adds an Organizations table with the conversations, how many were judged, the bad-response rate and the top issue for each. Click an organization there to see only its conversations. Each theme lists the organizations whose conversations fall in it. An agent that sends no group sees no change.

​
Privacy mode

If your agent sends traces without message content, Aide still reports tools, errors, tokens and spend. It can’t judge those answers or group them into themes.

​
How long data is kept

Conversation text is kept for 90 days. Counts, themes and eval cases are kept longer.

​
How it counts against your plan

Each conversation Aide judges counts once against your plan’s AI replies. Settings > Billing shows the total as Conversations judged in Agent analytics. Conversations that are imported but not judged don’t count. While AI is paused for the workspace, nothing is judged. See What counts as a billable AI reply.

​
If PostHog stops accepting Aide’s access

The Quality pages show a notice with Reconnect PostHog. New conversations aren’t imported until you reconnect. What was imported before stays.