What Is The Robots.txt File?

Discussion in 'Internet Marketing' started by sayem001, Dec 30, 2013.

Thread Status:
Not open for further replies.
  1. sayem001

    sayem001
    uix_expand uix_collapse
    Member

    Joined:
    Dec 29, 2013
    Messages:
    51
    Likes Received:
    1
    The robots.txt file is an element of the Robots Exclusion Protocol. you'll be able to use this file to permit or require search engines from creep and compartmentalization sure areas of your web site. this can be nice to scale back information measure usage by crawlers. you'll be able to conjointly purpose to your XML sitemap file; this can be helpful since it permits search engines to search out your XML sitemap.
    http://en.wikipedia.org/wiki/Robots.txt - More Detailed Information
     
  2. epicresearch

    epicresearch
    uix_expand uix_collapse
    Banned

    Joined:
    Nov 30, 2013
    Messages:
    80
    Likes Received:
    2
    A simple text file that stops Google (and other search engines that recognize the file and its commands) from crawling the site, selected pages in the site, or selected file types in the site.
     
  3. spyshekhar

    spyshekhar
    uix_expand uix_collapse
    Member

    Joined:
    Feb 13, 2014
    Messages:
    116
    Likes Received:
    5
    robot.txt file is used to tell the search engine that which file you want to crawled and which one you want not crawled by the search engine.
     
  4. jaysh4922

    jaysh4922
    uix_expand uix_collapse
    Member

    Joined:
    Dec 3, 2012
    Messages:
    143
    Likes Received:
    6
    Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.
     
  5. spyindiasachin

    spyindiasachin
    uix_expand uix_collapse
    New Member

    Joined:
    Mar 13, 2014
    Messages:
    9
    Likes Received:
    0
    Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.
     
  6. Zwillson

    Zwillson
    uix_expand uix_collapse
    Member

    Joined:
    Feb 5, 2014
    Messages:
    110
    Likes Received:
    4
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
     
  7. dancarto

    dancarto
    uix_expand uix_collapse
    New Member

    Joined:
    Dec 16, 2013
    Messages:
    14
    Likes Received:
    0
    It can be use to tell the search engine, which file you want to crawled and which one won't...
     
  8. Joseph.Shivell

    Joseph.Shivell
    uix_expand uix_collapse
    Community Leader
    Community Leader

    Joined:
    Oct 8, 2009
    Messages:
    2,263
    Likes Received:
    697
    This thread is now closed.If you have something new to add, let me know and I'll re-open it.
     
Thread Status:
Not open for further replies.

Share This Page