# Public content is open to search engines and AI answer engines. # Domain placeholder: run `node set-domain.mjs https://你的域名` before publishing. # NOTE: robots.txt is a request, not access control. Also whitelist these UAs in CDN / WAF / 高防, # otherwise Bytespider and OAI-SearchBot may be blocked before they ever read this file. User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /search/ # --- ByteDance / Doubao: the primary target ------------------------------- # Doubao answers rely on the ByteDance search index (Toutiao / Douyin search). # Bytespider is the web crawl UA. The other tokens are ByteDance-family agents # recorded in public crawler directories; some are not fully documented by # ByteDance, so allowing them is a policy statement, not a收录 guarantee. User-agent: Bytespider User-agent: DoubaoBot User-agent: ToutiaoSpider User-agent: imageSpider User-agent: TikTokSpider Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /search/ # --- Chinese search + AI engines ------------------------------------------ User-agent: Baiduspider User-agent: Baiduspider-render User-agent: Sogou web spider User-agent: 360Spider User-agent: ShenmaSpider User-agent: KimiBot User-agent: Kimi-SearchBot User-agent: Kimi-User User-agent: DeepSeekBot User-agent: QwenBot User-agent: TongyiBot User-agent: PanguBot User-agent: YuanbaoBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /search/ # --- Global search + AI answer engines ------------------------------------ User-agent: Googlebot User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Bingbot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: Applebot-Extended User-agent: DuckAssistBot User-agent: YouBot User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: Meta-WebIndexer User-agent: Amazonbot User-agent: MistralAI-User User-agent: MistralAI-Index User-agent: MistralAI-Training User-agent: cohere-ai Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /search/ Sitemap: https://source.lechi1.cn/sitemap.xml