wordpress蜘蛛抓取规则robots设置
放行了几个常用搜索引擎，其他的全部禁止，减轻服务器负担
User-agent: * Disallow: /wp-admin/ Disallow: /wp-include/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Disallow: /wp-includes/ Disallow: /redirect/ Disallow: /wp-login.php?redirect_to=* Disallow: /go?_=* Disallow: /?s= Disallow: /?p= User-agent: Baiduspider Disallow: User-agent: Sosospider Disallow: User-agent: sogou spider Disallow: User-agent: Googlebot Disallow: User-agent: Bingbot Disallow: User-agent: PetalBot Disallow: User-agent: YisouSpider Disallow: User-agent: Sogou web spider Disallow: User-agent: * Disallow: / Sitemap: https://www.168xindai.com/sitemap-post-2021.xml
