This is terribly straightforward to try to to. you'll open pad of paper or any text editor you wish and build the file robots.txt is text format solely. Below is that the basic sample code to purpose to your sitemap and permit categorisation and creep of all of your pages. simply save this file and transfer it to your high internet directory. User-agent: * Disallow: sitemap: Your URL to Your SiteMap http://www.robotstxt.org/wc/robots.html - More Advanced Robots.txt Usage.