æ们å¨åç½ç«æ¶ï¼æäºç®å½æè
æ件ä¸å¸ææç´¢å¼æèèæç´¢å°ï¼æ¯å¦åå°ï¼éè¦é¡µé¢ææ件ãé£ä¹æ们è¿æ¶è¯¥æä¹åå¢ï¼æç®åçæ¹æ³æ¯ä¿®æ¹robots.txtæ件åç½é¡µä¸çrobots Metaæ ç¾ã
robots.txtæ件æ¯åä»ä¹çï¼robotsæ件å称robotsåè®®ï¼ä¹ç§°ä¸ºç¬è«åè®®ãç¬è«è§åãæºå¨äººåè®®çï¼robotsåè®®æ¯åè¯æç´¢å¼æåªäºé¡µé¢å¯ä»¥æåï¼åªäºé¡µé¢ä¸è½æåãrobotsåè®®ä¸æ¯å½ä»¤ï¼å®åªæ¯ä¸ä¸ªç½ç«å½é
äºèç½çéè¡çéå¾·è§èï¼ç¥åçå¼æèèé½ä¼éµå®è¿ä¸ªåè®®ï¼åªæä¸äºç¼ºä¹èä¸éå¾·çæç´¢å¼æä¼ç»å¼å®ãæ以修æ¹è¿ä¸ªåè®®ä¹ä¸æ¯ä¸è½ä¿é©çã
robots.txtæ件æ¯ä¸ä¸ªæ¾ç½®å¨ç½ç«æ ¹ç®å½ä¸çè®°äºæ¬æ件ï¼å½æç´¢èè访é®ç½ç«æ¶ï¼å®ä¼å
æ£æ¥è¯¥ç«ç¹æ ¹ç®å½ä¸æ¯å¦åå¨robots.txtï¼å¦æåå¨ï¼æç´¢æºå¨äººå°±ä¼æç
§è¯¥æ件ä¸çå
容æ¥ç¡®å®è®¿é®çèå´ï¼å¦æ该æ件ä¸åå¨ï¼ææçæç´¢èèå°è½å¤è®¿é®ç½ç«ä¸ææ没æ被å£ä»¤ä¿æ¤ç页é¢ãç¾åº¦å®æ¹å»ºè®®ï¼ä»
å½æ¨çç½ç«å
å«ä¸å¸æ被æç´¢å¼ææ¶å½çå
容æ¶ï¼æéè¦ä½¿ç¨robots.txtæ件ãå¦ææ¨å¸ææç´¢å¼ææ¶å½ç½ç«ä¸ææå
容ï¼è¯·å¿å»ºç«robots.txtæ件ã
robotsæ件å±è½çåæ³
ä¸é¢æ¯ä¸ä¸ªç½ç«çrobotsæ件ç代ç æ ·å¼ã
===================================================================================
User-agent: *
Disallow: /plus/ad_js.php
Disallow: /plus/advancedsearch.php
Disallow: /plus/car.php
Disallow: /plus/carbuyaction.php
Disallow: /plus/shops_buyaction.php
Disallow: /plus/erraddsave.php
Disallow: /plus/posttocar.php
Disallow: /plus/disdls.php
Disallow: /plus/feedback_js.php
Disallow: /plus/mytag_js.php
Disallow: /plus/rss.php
Disallow: /plus/search.php
Disallow: /plus/recommend.php
Disallow: /plus/stow.php
Disallow: /plus/count.php
Disallow: /include
Disallow: /templets
===================================================================================
æ们æ¥çä¸ä¸å®çææã
robotsæ件ä¸åºåæ¶å
å«2个åï¼âUser-agent:âåâDisallow:âï¼å
¶ä¸User-agent:代表å
许ã Disallow: 代表ç¦æ¢ãæ¯æ¡æ令ç¬ç«ä¸è¡ã并ä¸User-agentå¿
é¡»åºç°å¨ç¬¬ä¸è¡ï¼ææä¹çè¡ï¼æ³¨éé¤å¤ï¼ï¼é¦å
声æç¨æ·ä»£çã
User-agent:
User-agent:æå®å
许åªäºèèæåï¼å¦æç»åºåæ°ï¼ååªææå®çèèè½å¤æåï¼å¦å¼ä¸ºéé
符â*âå·ï¼ä»£è¡¨å
许ææèèæåãå¦ï¼
User-agent: Googlebot
åªå
许Googleçèèæåã
User-agent: baiduspider
åªå
许ç¾åº¦çèèæåã
User-agent: *
è¿ä¸ªæ¯æå
许ææèèæåï¼*为éé
符ï¼ä»£è¡¨ææã
Disallow:
è¿ä¸ªæ¯è¡¨ç¤ºç¦æ¢èèæåçï¼ä»¥ä¸é¢çrobotsæ件为ä¾ã
Disallow: /plus/ad_js.php
è¿ä¸ªæ¯æç¦æ¢å¼æèèæåç½ç«ä¸plusç®å½ä¸çad_js.phpæ件
Disallow: /include
è¿ä¸ªæ¯æç¦æ¢å¼æèèæåç½ç«ä¸includeæ´ä¸ªç®å½ï¼å®åDisallow: /include/ä¸æ¯çåçã
Disallow: /include/
è¿ä¸ªæ¯æç¦æ¢å¼æèèæåç½ç«ä¸includeç®å½ä¸é¢çç®å½ï¼ä½includeç®å½ä¸çæ件èèä»ç¶å¯ä»¥æåã
Disallow: /cgi-bin/*.htm
ç¦æ¢è®¿é®/cgi-bin/ç®å½ä¸çææ以".htm"为åç¼çURL(å
å«åç®å½)ã
Disallow: /*?*
ç¦æ¢è®¿é®ç½ç«ä¸ææå
å«é®å· (?) çç½å
Disallow: /.jpg$
ç¦æ¢æåç½é¡µææç.jpgæ ¼å¼çå¾ç
Allow:
è¿æ¯å
许çææã
Allow: /include/ã
è¿éæ¯å
许ç¬å¯»includeç®å½ä¸é¢çç®å½
Allow: /include
è¿éæ¯å
许ç¬å¯»includeæ´ä¸ªç®å½
Allow: .htm$
ä»
å
许访é®ä»¥".htm"为åç¼çURLã
Allow: .gif$
å
许æåç½é¡µågifæ ¼å¼å¾ç
Sitemap:
ç½ç«å°å¾ åè¯ç¬è«è¿ä¸ªé¡µé¢æ¯ç½ç«å°å¾
robotæ件ç常è§ç¨æ³
ä¾1. ç¦æ¢æææç´¢å¼æ访é®æ´ä¸ªç½ç«ã
User-agent: *
Disallow: /
ä¾2.ç¦æ¢ç¾åº¦èè访é®æ´ä¸ªç½ç«ã
User-agent: baiduspider
Disallow: /
ä¾3.å
许æææç´¢å¼æ访é®ï¼ä¹å¯ä»¥å»ºç«ä¸ä¸ªç©ºrobots.txtæ件æè
ä¸å»ºç«robotsæ件ï¼
User-agent: *
Allow:ã/
User-agent:åç*å
·æç¹æ®çå«ä¹ï¼ä»£è¡¨âany robotâï¼æ以å¨è¯¥æ件ä¸ä¸è½æâDisallow: /tmp/*â or âDisallow:*.gifâè¿æ ·çè®°å½åºç°ã
robots Metaæ ç¾å±è½æ³
robots.txtæ¯æ¾å¨ç½ç«ä¸ï¼æ件级çç½ç»èèææï¼èrobots Metaæ ç¾æ¯æ¾å¨ç½é¡µä¸ï¼ä¸è¬ç¨äºé¨åç½é¡µéè¦åç¬è®¾ç½®çæ
åµä¸ã两è
çåè½æ¯ä¸æ ·çã
Meta robotsæ ç¾å¿
é¡»æ¾å¨<head>å</head>ä¹é´ï¼æ ¼å¼ï¼
<meta name=ârobotsâ content=âindex,followâ />
contentä¸çå¼å³å®å
许æåçç±»åï¼å¿
é¡»åæ¶å
å«ä¸¤ä¸ªå¼ï¼æ¯å¦å
许索å¼ï¼indexï¼åæ¯å¦è·è¸ªé¾æ¥ï¼followï¼ä¹å¯ä»¥ç解为æ¯å¦å
许沿çç½é¡µä¸çè¶
级é¾æ¥ç»§ç»æåï¼ãå
±æ4个åæ°å¯éï¼ç»æ4个ç»åï¼
index,followï¼å
许æåæ¬é¡µï¼å
许è·è¸ªé¾æ¥ã
index,nofollowï¼å
许æåæ¬é¡µï¼ä½ç¦æ¢è·è¸ªé¾æ¥ã
noindex,followï¼ç¦æ¢æåæ¬é¡µï¼ä½å
许è·è¸ªé¾æ¥ã
noindex,nofllowï¼ç¦æ¢æåæ¬é¡µï¼åæ¶ç¦æ¢è·è¸ªæ¬é¡µä¸çé¾æ¥ã
以ä¸1å4è¿æå¦ä¸ç§åæ³ï¼
index,followå¯ä»¥åæallï¼å¦ï¼
<meta name=ârobotsâ content=âallâ />
noindex,nofollowå¯ä»¥åænoneï¼å¦ï¼
<meta name=ârobotsâ content=ânoneâ />
éè¦æ³¨æçæ¯ï¼robots Metaæ ç¾å¾å¤æç´¢å¼ææ¯ä¸æ¯æçï¼åªæå°æ°æç´¢å¼æè½å¤è¯å«å¹¶æç»å®çå¼æåãæ以ï¼å°½å¯è½ç使ç¨robots.txtæ件æ¥éå¶æåã
温馨提示:答案为网友推荐,仅供参考