Robots.txt file jise Hum (web crawler, ya spider ) ke naam se bhi jante hai ye ek aisa programe ko jo kisi bhi blog ya website ko track karta hai.
Google search engine bhi apne search result dikhane ke liye robots.txt file ka use karta hai
Robots.txt Kya Hai?
Robots.txt file example hai :-
Example 1
User-agent: Googlebot
Disallow: /nogooglebot/
Example 2
User-agent: *
Allow: /
Sitemap: http://www.example.com/sitemap.xml
Sabhi blogger platform ke liye ek default robots.txt file hota hai jo hai :-
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
Jise 3 part pe dived kiya gaya hai jise humne explain kiya hai apne users ke liye
User-agent: Mediapartners-Google
Yeh code hum Google Adsense Robots ke liye use karte hai jisse aapke blog ko ek better ads Google Adsense provide karta hai.
User-agent: *
Yeha jo aap ek start * Mark dekh rahe hai ye aapke blog pe jo labels links hai unhe Google search result pe index hone se rokta hai
Disallow: /search
Iska use hum isliye karte hai jab google apne search result pe hamare blog se koi content/post dikhayeha to uss result pe search Label ko na dikhaye
http://www.devanagaritech.com /label/Robots.txt
agar aap ise removed kar de to Google aisa Result dikhayega
http://www.devanagaritech.com/search/label/Robots.txt
Koi Post Hai jo aap Google search pe nahi dikhana chahte hai to iss robots.txt file code ko use kar sakte hai :-
Disallow: /yyyy/mm/post-url.html
Jaha yyyy year ko darshata hai aur mm month for example
Disallow: /2013/03/post-url.html
robot.txt file blogger pe Kaise Add kare:-
Go to your blogger blog.
Blogger Dashboard >> Settings >> Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes
Ab apna robots.txt file code ko box pe paste kar de and save changes button pe click kar de
Ho Gaya robots.txt file add aapne blogger Blogspot blog pe.
Blogger blog pe robots.txt file Kaise check kare?
Apne url pe last pe bas iss code ko use kar aap robots.txt file check kar sakte hai
/robots.txt
For example http://devanagaritech.com/robots.txt
Ye hamare DevanagariTech ka robots.txt file hai.
robots.txt file Generate Kaise Kare?
Aap Google ka Robots Testing Tool ko use kar apne blog pe robots.txt file tester kar sakte hai.