本帖最后由 SeanKok 于 2013-6-18 18:18 编辑
360工作人员,你好,我的网站使用了360网站卫士服务,麻烦看下我的网站 http://www.cgwill.com 为什么访问正常但百度和谷歌的蜘蛛无法抓取了,而其他搜索引擎比如360搜素,SOSO,Bing 的蜘蛛抓取却是正常的。去百度站长平台和谷歌网站工具看了下,都提示无法抓取的异常。网站服务商是万网,之前和万网的客服联系了,他们说原因是我使用了代理没有将网站解析到服务器IP上,请问是不是360网站卫士把蜘蛛给屏蔽了?
robots:
# Created by Yifan @ 2008/11/3# Sorry! You will see no welcome descriptions except this because we want to limit the total output bytes.# Section A: Version 1.0 standard# Please add disallows in this section!# DO NOT add allows in this section, since "allow" is only allowed in robots.txt version 2.0.User-agent: *Disallow: /web/vcodeimg.aspx # verify code imagingDisallow: /plugin/doc/docEdit.aspx # user comment edit pageDisallow: /~/plugin/doc/docEdit.aspx # user comment edit page# Section B: Non standardCrawl-delay: 30Sitemap: http://www.cgwill.com/web/sitemapsxml.aspx?sid=10039685 # this is automatically generated by robots.aspx# Section C: Version 2.0 standard# Please check the following url:# http://www.conman.org/people/spc/robots2.html#format.directives.robot-version# You can add allows here! eg:# Allow: /# Request-rate Hints:# Request-rate for the specific time, h=hour, m=minute, omitted=secondRequest-rate: 1/1h 0030-0329 # 0830-1129, Beijing, zoom +8Request-rate: 3/1h 0330-0529 # 1130-1329, Beijing, zoom +8Request-rate: 1/1h 0530-1029 # 1330-1829, Beijing, zoom +8Request-rate: 1/6m 1030-1329 # 1830-2129, Beijing, zoom +8Request-rate: 1/1m 1330-0029 # 2130-0829 +1 day, Beijing, zoom +8# End of file
|
|
|
|
评论
直达楼层