| Home | Free Articles for Your Site | Submit an Article | Advertise | Link to Us | Search | Contact Us | |
Custom Search
|
Article Surfing ArchiveThe Proper Way To Use The robot.txt File - Articles SurfingWhen optimizing your web site most webmasters don't consider using the robot.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder. Here is a list of variables that you can include in a robot.txt file and there meaning:
Here are some examples of a robot.txt file User-agent: *Disallow: The above would let all spiders index all content. Here another User-agent: *Disallow: /cgi-bin/ The above would block all spiders from indexing the cgi-bin directory. User-agent: googlebotDisallow: User-agent: *Disallow: /admin.phpDisallow: /cgi-bin/Disallow: /admin/Disallow: /stats/ In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.
RELATED SITES
Copyright © 1995 - Photius Coutsoukis (All Rights Reserved). |
ARTICLE CATEGORIES
Aging Arts and Crafts Auto and Trucks Automotive Business Business and Finance Cancer Survival Career Classifieds Computers and Internet Computers and Technology Cooking Culture Education Education #2 Entertainment Etiquette Family Finances Food and Drink Food and Drink B Gadgets and Gizmos Gardening Health Hobbies Home Improvement Home Management Humor Internet Jobs Kids and Teens Learning Languages Leadership Legal Legal B Marketing Marketing B Medical Business Medicines and Remedies Music and Movies Online Business Opinions Parenting Parenting B Pets Pets and Animals Poetry Politics Politics and Government Real Estate Recreation Recreation and Sports Science Self Help Self Improvement Short Stories Site Promotion Society Sports Travel and Leisure Travel Part B Web Development Wellness, Fitness and Diet World Affairs Writing Writing B |