The robots.txt file is a file that sits at the root level of your web site and asks spiders and bots to behave themselves when they're on your site. Think of it as an electronic NO TRESPASSING sign that tells search engines not to crawl a certain diretory or page of your site. The robots.txt file is essential for your site to be crawled by the search bots. Without it, Google will not crawl your site and you can kiss a high rating in the search engines goodbye. On Friday, we’ll talk about which changes that you need to make to your robots.txt file, but here is a reference on how to make the changes.

1. Check to see if your robots.txt file is there and available to visiting search bots. Open your browser and visit the following link: http://www.yourDrupalsite.com/robots.txt

2. Using your FTP program or command line editor, navigate to the top level of your Drupal web site and locate the robots.txt file.

3. Make a backup of the file.

4. Open the robots.txt file for editing. If necessary, download the file and open it in a local text editor tool.

5. Most directives in the robots.txt file are based on the line User-agent:. If you are going to give different instructions to different engines, be sure to place them above the User-agent: *, as some search engines will only read the directives for * if you place their specific instructions following that section.

6. Add the lines you want. There are several changes which will help you with your SEO.

7. Save your robots.txt file, uploading it if necessary, replacing the existing file (you backed it up, didn’t you?).

8. Point your browser to http://www.yourDrupalsite.com/robots.txt and double­ check that your changes are in effect. You may need to do a refresh on your browser to see the changes.

Check back on Friday to see some of the changes you need to make to your robots.txt file to make it the most effective.