How to stop Search Engines from crawling your Website?

Discussion in 'Search Engine Optimization' started by rkdolla, Dec 20, 2013.

Thread Status:
Not open for further replies.
  1. rkdolla

    rkdolla
    uix_expand uix_collapse
    Member

    Joined:
    Dec 19, 2013
    Messages:
    50
    Likes Received:
    1
    Please share your answers..
     
  2. RoseAb

    RoseAb
    uix_expand uix_collapse
    Member

    Joined:
    Dec 7, 2013
    Messages:
    87
    Likes Received:
    10
    You really don't want to stop search engines entirely. I think you want to keep them out of some pages. You use robots.txt file to do that.
    Also you can use Sitemap.xml to tell them what you do want indexed.
     
    • Like Like x 2
  3. Sean_DeSilva

    Sean_DeSilva
    uix_expand uix_collapse
    Member

    Joined:
    Dec 19, 2013
    Messages:
    59
    Likes Received:
    3
    you can also use the .htaccess file to block undesired web crawlers.
     
    • Like Like x 1
  4. Iconiplex

    Iconiplex
    uix_expand uix_collapse
    New Member

    Joined:
    Dec 15, 2013
    Messages:
    10
    Likes Received:
    3
    You don't know his needs and whether this is necessary for his implementation or not.

    http://www.robotstxt.org/robotstxt.html
     
  5. spyindiaankur

    spyindiaankur
    uix_expand uix_collapse
    Member

    Joined:
    Jul 13, 2013
    Messages:
    134
    Likes Received:
    9
    You can put rel="No follow, no index" attribute in meta tag.
    You can use robots.txt file
    or you can do it through search engine webmaster tools.
     
  6. hamishcode

    hamishcode
    uix_expand uix_collapse
    Member

    Joined:
    Oct 8, 2013
    Messages:
    99
    Likes Received:
    3
    Either You Can Put put rel="No follow, no index" In Meta Or You Can Put .htaccess File to Block Crawler or Even you Can Put Robot.txt File In That.
     
  7. joy.linford

    joy.linford
    uix_expand uix_collapse
    Member

    Joined:
    Dec 9, 2013
    Messages:
    89
    Likes Received:
    3
    The Pages which you do not want search engine to crawl then you need to use ROBOT.TXT file.
     
  8. rudradhar

    rudradhar
    uix_expand uix_collapse
    Member

    Joined:
    Jul 30, 2012
    Messages:
    268
    Likes Received:
    29
    As per I know we can't do stop crawling through webmaster tools. If there is anyway in the Google webmasters tool then its in URL Parameter section which is useable only for dynamic urls.
     
  9. marriott

    marriott
    uix_expand uix_collapse
    Member

    Joined:
    Sep 20, 2013
    Messages:
    400
    Likes Received:
    11
    If you don't want to crawl your website from search engines, you can put your site pages in the robot. text file.
     
  10. spyshekhar

    spyshekhar
    uix_expand uix_collapse
    Member

    Joined:
    Feb 13, 2014
    Messages:
    116
    Likes Received:
    5
    if you want that search engine not crawled your webpages so you can use robot.txt.
     
  11. prakash

    prakash
    uix_expand uix_collapse
    Member

    Joined:
    Jan 25, 2013
    Messages:
    72
    Likes Received:
    9
    best and fastest way is to put <a href="..." rel="nofollow, noindex"> tag.. b coz of that crowler won't crowl ur page
     
  12. Joseph.Shivell

    Joseph.Shivell
    uix_expand uix_collapse
    Community Leader
    Community Leader

    Joined:
    Oct 8, 2009
    Messages:
    2,267
    Likes Received:
    697
    This thread is now closed. If you have anything new to add, let me know and I'll re-open it.
     
Thread Status:
Not open for further replies.

Share This Page