User-agent: * Allow: /social_instagram/ Sitemap: https://innolabs.dev/sitemap.xml ############## # custom # ############## Disallow: /web/login Disallow: /web/session Disallow: /web/database Disallow: /my/ Disallow: /shop/cart Disallow: /shop/checkout Disallow: /shop/payment Disallow: /knowledge/ # Explicit allow for AI crawlers (trust signal); internal Knowledge stays out User-agent: GPTBot Allow: / Disallow: /knowledge/ User-agent: ChatGPT-User Allow: / Disallow: /knowledge/ User-agent: OAI-SearchBot Allow: / Disallow: /knowledge/ User-agent: ClaudeBot Allow: / Disallow: /knowledge/ User-agent: Claude-Web Allow: / Disallow: /knowledge/ User-agent: anthropic-ai Allow: / Disallow: /knowledge/ User-agent: PerplexityBot Allow: / Disallow: /knowledge/ User-agent: Google-Extended Allow: / Disallow: /knowledge/ User-agent: CCBot Allow: / Disallow: /knowledge/ User-agent: cohere-ai Allow: / Disallow: /knowledge/ User-agent: Applebot-Extended Allow: / Disallow: /knowledge/ # Block high-volume scrapers with no disclosed AI-product purpose User-agent: Bytespider Disallow: / User-agent: DataForSeoBot Disallow: /