tipsandcoupons
Coupon Code Guru
|
Posted: Sat Jan 28, 2006 10:49 pm
Post subject: A robots.txt file for your website - more important
|
|
  Save This Page  |
|
What's the most important thing for your website (aside from not using Flash)? A robots.txt file - this is the file that tells the search engine spiders (the tools the search engines use to read your pages into their index) what they can and can't see. What's the simplest robots.txt file you can have?
User-agent: *
Disallow:
This tells the search engine spider that any of them can see anything on your site. For most websites that's all you need - if you have an admin section or some other pages you don't want the world to be able to see, this is the place to put it.
To lazy, or unsure if you can create your own file? Use the robots.txt creator tool at http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator/. |
|
