FlyerTalk Forums

FlyerTalk Forums (https://www.flyertalk.com/forum/index.php)
-   Travel Technology (https://www.flyertalk.com/forum/travel-technology-169/)
-   -   robot.txt help (https://www.flyertalk.com/forum/travel-technology/898789-robot-txt-help.html)

Dubai Stu Dec 13, 2008 8:48 am

robot.txt help
 
I'm throwing together a simple website for my civic association and at the moment don't want it to be indexed by the search engines. Does anyone off their head know how to write a robot.txt file that simply excludes everything in the domain?

Any sorry I know the post is a tad off topic, but hopefully my contributions to the forum outweigh this minor diversion.

sbm12 Dec 13, 2008 9:15 am

Google is pretty useful. ;)

http://www.javascriptkit.com/howto/robots.shtml
1) Here's a basic "robots.txt":

User-agent: *
Disallow: /

With the above declared, all robots (indicated by "*") are instructed to not index any of your pages (indicated by "/"). Most likely not what you want, but you get the idea.
And remember that "hiding" the content with robots.txt doesn't really hide anything. The content is still on the internet and anyone who wants to can find it.

Dubai Stu Dec 13, 2008 9:28 am

Thanks for the tip. Anyone who really wants to find it will find an early draft of a boring website with typos, comma splices, and a some non-implemented features.


All times are GMT -6. The time now is 8:20 am.


This site is owned, operated, and maintained by MH Sub I, LLC dba Internet Brands. Copyright © 2026 MH Sub I, LLC dba Internet Brands. All rights reserved. Designated trademarks are the property of their respective owners.