Playground conversations are now labeled in the background

Once a Playground conversation has been quiet for 30 minutes, an AI judge now reads it in the background and records a set of labels about it: whether the need was met, what the conversation was about, whether the replies were accurate, on brand and within your instructions, what the shopper objected to, and what was missing, from a product in your catalog to a capability the assistant lacks. Every negative answer points at the messages or products it's based on. A conversation that gets a new message or a new rating is labeled again, and the earlier label is kept.

What this covers today:

• Playground conversations only. Conversations shoppers have on your storefront or on your public link are not labeled.
• Nothing to read yet. The labels are stored with each conversation, but no dashboard page, API or MCP tool shows them for now.
• How the data is handled. The judge is an OpenAI model reached through the Vercel AI Gateway, pinned to the European inference zone, with zero data retention and no training. Labels are deleted with their conversation.

Labeling runs once an hour, and you don't have to wait for it: the Conversations page of every site now has a Label conversations now button, next to the conversation count. It labels your organization's Playground conversations straight away, on all its sites, once they've been quiet for a minute. It needs the Conversations: Edit permission, and each click is recorded in the audit log.

See Conversation labeling for which conversations are labeled and when, and the Label reference for every label and value.