robot.txt help
Dec 13, 2008 | 8:48 am
#1
Original Poster
Join Date: Nov 2006
Location: Detroit; Formerly Dubai
Posts: 3,676
robot.txt help
I'm throwing together a simple website for my civic association and at the moment don't want it to be indexed by the search engines. Does anyone off their head know how to write a robot.txt file that simply excludes everything in the domain?
Any sorry I know the post is a tad off topic, but hopefully my contributions to the forum outweigh this minor diversion.
Any sorry I know the post is a tad off topic, but hopefully my contributions to the forum outweigh this minor diversion.
Dec 13, 2008 | 9:15 am
#2
A FlyerTalk Posting Legend
Join Date: Apr 2001
Location: PSM
Posts: 69,232
Google is pretty useful. 
http://www.javascriptkit.com/howto/robots.shtml
http://www.javascriptkit.com/howto/robots.shtml
1) Here's a basic "robots.txt":
User-agent: *
Disallow: /
With the above declared, all robots (indicated by "*") are instructed to not index any of your pages (indicated by "/"). Most likely not what you want, but you get the idea.
And remember that "hiding" the content with robots.txt doesn't really hide anything. The content is still on the internet and anyone who wants to can find it.
User-agent: *
Disallow: /
With the above declared, all robots (indicated by "*") are instructed to not index any of your pages (indicated by "/"). Most likely not what you want, but you get the idea.