Robots.txt æ¯åæ¾å¨ç«ç¹æ ¹ç®å½ä¸çä¸ä¸ªçº¯ææ¬æ件ã
è½ç¶å®ç设置å¾ç®åï¼ä½æ¯ä½ç¨å´å¾å¼ºå¤§ãå®å¯ä»¥æå®æç´¢å¼æèèåªæåæå®çå
容ï¼æè
æ¯ç¦æ¢æç´¢å¼æèèæåç½ç«çé¨åæå
¨é¨å
容ã
使ç¨æ¹æ³:
Robots.txt æ件åºè¯¥æ¾å¨ç½ç«æ ¹ç®å½ä¸ï¼å¹¶ä¸è¯¥æ件æ¯å¯ä»¥éè¿äºèç½è¿è¡è®¿é®çã
ä¾å¦ï¼å¦ææ¨çç½ç«å°åæ¯
http://www.baidu.com/é£ä¹ï¼è¯¥æ件å¿
é¡»è½å¤éè¿
http://www.baidu.com/robots.txt æå¼å¹¶çå°éé¢çå
容ã
æ ¼å¼ï¼
User-agent:
ç¨äºæè¿°æç´¢å¼æèèçååï¼å¨" Robots.txt "æ件ä¸ï¼å¦ææå¤æ¡User-agentè®°å½è¯´ææå¤ä¸ªæç´¢å¼æèèä¼åå°è¯¥åè®®çéå¶ï¼å¯¹è¯¥æ件æ¥è¯´ï¼è³å°è¦æä¸æ¡User-agentè®°å½ãå¦æ该项çå¼è®¾ä¸º*ï¼å该å议对任ä½æç´¢å¼æèèåææï¼å¨" Robots.txt "æ件ä¸ï¼"User-agent:*"è¿æ ·çè®°å½åªè½æä¸æ¡ã
Disallow:
ç¨äºæè¿°ä¸å¸æ被访é®å°çä¸ä¸ªURLï¼è¿ä¸ªURLå¯ä»¥æ¯ä¸æ¡å®æ´çè·¯å¾ï¼ä¹å¯ä»¥æ¯é¨åçï¼ä»»ä½ä»¥Disallowå¼å¤´çURLåä¸ä¼è¢«Robot访é®å°ã
举ä¾ï¼
ä¾ä¸ï¼"Disallow:/help"æ¯æ/help.html å/help/index.htmlé½ä¸å
许æç´¢å¼æèèæåã
ä¾äºï¼"Disallow:/help/"æ¯æå
许æç´¢å¼æèèæå/help.htmlï¼èä¸è½æå/help/index.htmlã
ä¾ä¸ï¼Disallowè®°å½ä¸ºç©ºè¯´æ该ç½ç«çææ页é¢é½å
许被æç´¢å¼ææåï¼å¨"/robots.txt"æ件ä¸ï¼è³å°è¦æä¸æ¡Disallowè®°å½ãå¦æ"/robots.txt"æ¯ä¸ä¸ªç©ºæ件ï¼å对äºææçæç´¢å¼æèèï¼è¯¥ç½ç«é½æ¯å¼æ¾çå¯ä»¥è¢«æåçã
#:Robots.txt åè®®ä¸ç注é符ã
综åä¾å ï¼
ä¾ä¸ï¼éè¿"/robots.txt"ç¦æ¢æææç´¢å¼æèèæå"/bin/cgi/"ç®å½ï¼ä»¥å "/tmp/"ç®å½å /foo.html æ件ï¼è®¾ç½®æ¹æ³å¦ä¸ï¼
User-agent: *
Disallow: /bin/cgi/
Disallow: /tmp/
Disallow: /foo.html
ä¾äºï¼éè¿"/robots.txt"åªå
许æ个æç´¢å¼ææåï¼èç¦æ¢å
¶ä»çæç´¢å¼ææåãå¦ï¼åªå
许å为"slurp"çæç´¢å¼æèèæåï¼èæç»å
¶ä»çæç´¢å¼æèèæå "/cgi/" ç®å½ä¸çå
容ï¼è®¾ç½®æ¹æ³å¦ä¸ï¼
User-agent: *
Disallow: /cgi/
User-agent: slurp
Disallow:
ä¾ä¸ï¼ç¦æ¢ä»»ä½æç´¢å¼ææåæçç½ç«ï¼è®¾ç½®æ¹æ³å¦ä¸ï¼
User-agent: *
Disallow: /
ä¾åï¼åªç¦æ¢æ个æç´¢å¼ææåæçç½ç«å¦ï¼åªç¦æ¢å为âslurpâçæç´¢å¼æèèæåï¼è®¾ç½®æ¹æ³å¦ä¸ï¼
User-agent: slurp
Disallow: /