Travel Technology - robot.txt help




View Full Version : robot.txt help


Dubai Stu
Dec 13, 08, 9:48 am
I'm throwing together a simple website for my civic association and at the moment don't want it to be indexed by the search engines. Does anyone off their head know how to write a robot.txt file that simply excludes everything in the domain?

Any sorry I know the post is a tad off topic, but hopefully my contributions to the forum outweigh this minor diversion.


sbm12
Dec 13, 08, 10:15 am
Google (http://www.google.com/search?q=robots.txt+exclude+all&rls=com.microsoft:*&ie=UTF-8&oe=UTF-8&startIndex=&startPage=1) is pretty useful. ;)

http://www.javascriptkit.com/howto/robots.shtml
1) Here's a basic "robots.txt":

User-agent: *
Disallow: /

With the above declared, all robots (indicated by "*") are instructed to not index any of your pages (indicated by "/"). Most likely not what you want, but you get the idea.

And remember that "hiding" the content with robots.txt doesn't really hide anything (http://www.flyertalk.com/forum/travel-technology/893896-robots-txt-usage-personal-site.html). The content is still on the internet and anyone who wants to can find it.

Dubai Stu
Dec 13, 08, 10:28 am
Thanks for the tip. Anyone who really wants to find it will find an early draft of a boring website with typos, comma splices, and a some non-implemented features.




SEO by vBSEO 3.2.0