User-Agent, matched
word-boundary, case-insensitive. The edge library’s built-in list and the Vercel and ESI regexes in
integrate mirror this table exactly; doubleshift classifies reported agents with the same
list. The machine-readable copy is schema/agents.json on brandedcontent.dev.
Purpose:
query, a real-time fetch for a user asking now; search, building an AI search index;
training, a bulk crawl for a training corpus; unknown, mixed or undocumented. Robots.txt-only tokens
such as Google-Extended are left out: they never appear in a User-Agent.