Crawler related questions!

Discussion in 'Search Engine Optimization' started by TimB, Mar 1, 2013.

  1. TimB

    TimB
    uix_expand uix_collapse
    Member

    Joined:
    Aug 15, 2012
    Messages:
    161
    Likes Received:
    19
    Hey All, :)

    I have a few questions kindly answer them;

    I wanted to know that can we effect the crawler's speed and the number of crawls to our website? Is there a crawling pattern of the search engine or its random?

    Thank-you for the help in advance!
     
  2. daytrader

    daytrader
    uix_expand uix_collapse
    Active Member

    Joined:
    Jul 29, 2009
    Messages:
    1,678
    Likes Received:
    216
    We don't have control over when and their speed but you can set it up in robot.txt so that your site can be crawled any moment.
     
  3. Bagchi

    Bagchi
    uix_expand uix_collapse
    Member

    Joined:
    Sep 10, 2013
    Messages:
    86
    Likes Received:
    12
    I suppose, robots.txt, if used as "Disallow: ", it works as a catalyst in increasing the speed of crawlers, in case their speed is decreased due to special characters like ? or ; or , . Pipes "|" and hyphen "-" are good characters to use and don't affect the speed of crawlers. :)
     
  4. sportychints

    sportychints
    uix_expand uix_collapse
    New Member

    Joined:
    Aug 14, 2013
    Messages:
    21
    Likes Received:
    0
    Frankly, We can't control the crawler ...It's totally depends on it's mood...As per my opinion it is a useless function of Google Webmaster Tools...
     

Share This Page