# One group, on purpose. robots.txt matching is winner-take-all: a crawler with its own # group obeys only that group and ignores `*` entirely. Naming Googlebot, bingbot, # OAI-SearchBot, PerplexityBot and Claude-SearchBot here bought nothing while every rule # was `Allow: /`, and the first `Disallow:` added below would have silently skipped them. # Add a named group only with a rule that is genuinely meant for that crawler alone, and # then restate every rule from this group inside it. User-agent: * Allow: / Sitemap: https://www.docsgpt.cloud/sitemap.xml