The Bot List

We block the following robots and spiders.

HomeThe Bot List
  • 200PleaseBot
  • 360Spider
  • CrawlDaddy
  • AddThis bot
  • Microsoft Bing Ads bot
  • ADmantX Platform Semantic bot
  • Google Adwords bot
  • AdvBot
  • Ahrefs backlinks research tool
  • Alexa Crawler
  • apache-httpclient Java http library
  • ApacheBench (ab)
  • apis-google APIs-Google
  • appengine-google Google App Engine
  • applebot Apple Bot
  • archive.org_bot Internet Archive
  • ask jeeves Ask Jeeves
  • asynchttpclient Java http and WebSocke
  • awe.sm Awe.sm URL expander
  • Baidu bot
  • bdcbot Big Data Corp
  • bingbot Microsoft Bing
  • bingpreview Microsoft Bing preview
  • bitlybot bit.ly bot
  • blekkobot Blekkobot
  • blexbot BLEXBot (webmeup)
  • bot@linkfluence.net Linkfluence bot
  • BufferBot
  • buibui-checkbot
  • butterfly Topsy Labs
  • buzztalk
  • CatchBot (catchbot.com)
  • check_http Nagios monitor
  • Cliqzbot
  • cmradar/0.1 CMRadar/0.1
  • coldfusion ColdFusion http library
  • commoncrawl CCBot
  • comodo-webinspector-crawler
  • Crowsnest
  • curabot cura.yt
  • curl curl unix CLI http client
  • dap/nethttp DAP/NetHTTP
  • datagnionbot datagnion.com/bot.html
  • daumoa Korean portal and search engine
  • developers.google.com/+/web/snippet
  • Diffbot
  • digitalpersona fingerprint software HP
  • Domain Re-Animator Bot
  • DomainsBot
  • DomainTuno
  • Dot Bot
  • Duck Duck Go
  • AWS ELB HealthChecker
  • Embedly
  • EOAAgent
  • eventmachine httpclient Ruby http library
  • EveryoneSocial bot
  • Evri bot
  • Exalead’s bot
  • ExaleadCloudView
  • Facebook Bot
  • Facebot Bot
  • feedburner RSS bot
  • Google Feedfetcher
  • Findxbot
  • FlipboardProxy
  • FriendFeed
  • Genieo Web filter bot
  • getprismatic.com
  • Gigabot spider
  • gimme60bot
  • gimmeusabot
  • Go http library
  • Google Page Speed Insights
  • Google Instant Previews crawler
  • Google-StructuredDataTestingTool
  • Googlebot
  • GoogleStackdriverMonitoring-UptimeChecks
  • GrapeshotCrawler
  • Gravity Bot
  • Hatena::Bookmark
  • heritrix
  • HTMLParser
  • HTTP_Request2
  • HTTPClient
  • HubSpot bot
  • Internet Archive (WayBackMachine)
  • iCoreService
  • idmarch.org/bot.html
  • inagist URL resolver
  • Insieve Bot
  • Insitesbot
  • Instapaper
  • IstellaBot
  • Jack bot
  • Jakarta Commons HttpClient
  • Java Generic http library
  • Jetslide
  • js-kit URL resolver
  • kemvibot
  • Kimengi Bot
  • knows.is bot
  • Kojitsubot
  • KomodiaBot
  • kraken
  • Laconica
  • libwww-perl Perl client-server
  • Lijit crawler
  • Linkdex Bot
  • LinkedIn Bot
  • LinksCrawler Bot
  • Linode Longview
  • Lipperhey
  • Livelapbot
  • Load Time Bot
  • longurl URL expander service
  • ltx71.com bot
  • Lumibot
  • lwp-trivial Another Perl library
  • magpie-crawler magpie-crawler
  • Mail.ru Bot
  • meanpathbot
  • mediapartners-google Google Adsense bot
  • megaindex.ru MegaIndex
  • memorybot
  • MetaURI
  • mfe_expand Mcafee spider
  • MIR web crawler
  • mj12bot Majestic-12 spider
  • mojeekbot Mojeek UK search crawler
  • MrChrome
  • MS Search 6.0 Robot
  • msnbot-media Microsoft media bot
  • msnbot Microsoft bot
  • NerdyBot
  • Netcraft bot
  • netEstate NE Crawler
  • netvibes Personalized dashboard bot
  • netzcheckbot
  • NewRelic monitor
  • NewRelicPinger
  • newsme
  • niki-bot
  • ning NING – Yet Another Twitter Swarmer
  • nutch Apache search spider
  • OpenHoseBot
  • OrangeBot
  • pagesinventory.com
  • panopta Monitoring service
  • PaperLi
  • peerindex
  • PercolateCrawler
  • PerfectMarket bot
  • PhantomJS
  • Pingdom monitoring
  • Pinterest bot
  • plukkie bot
  • PrivacyAwareBot
  • Proximic Spider
  • psbot-page Picsearch
  • publiclibraryarchive.org
  • pycurl Python http library
  • python-httplib2 Python-httplib2
  • python-requests Python http library
  • python-urllib Python http library
  • queryseeker QuerySeekerSpider
  • quicklook QuickLook
  • re-animator Domain Re-Animator Bot
  • readability Readability
  • rebelmouse RebelMouse
  • redditbot Reddit Bot
  • relateiq RelateIQ
  • riddler Riddler Bot
  • rogerbot SeoMoz spider
  • rssmicro RSS/Atom Feed Robot
  • Ruby bot
  • Scrapy bot
  • Screaming Frog SEO Spider
  • SearchmetricsBot
  • semrushbot SEO analysis bot
  • SEOKicks
  • SeznamBot
  • ShopWiki
  • shortlinktranslate Link shortener
  • showyoubot Showyou iOS app spider
  • Siege Joe Dog
  • SISTRIX
  • siteuptime Site monitoring services
  • Slackbot-LinkExpanding
  • Slack Bot
  • slurp Yahoo spider
  • smtbot SimilarTech
  • SocialRankIOBot
  • sogou Chinese search engine
  • spbot OpenLinkProfiler
  • spider generic web spider
  • Spinn3r aggregator
  • SputnikBot
  • Squider
  • StatusCake
  • Stripe bot
  • test certificate info C http library
  • TinEye Bot
  • Traackr Bot
  • Trendiction Search
  • TurnitinBot
  • tweetedtimes The Tweeted Times
  • tweetmemebot TweetMeMe Crawler
  • twikle Social web search bot
  • TwitJobSearch
  • Twitmunin
  • twitterbot Twitter URL expander
  • Twurly
  • Typhoeus
  • umbot uberMetrics
  • unwindfetch Gnip
  • uptimerobot Uptime Robot
  • Vagabondo
  • vb project Visual Basic
  • Vigil
  • VKontake Sharer
  • VoilaBot
  • vrcrawler Venture Radar
  • Wasalive Bots
  • WatchSumo bot
  • Ware Bay Best Buys bot
  • Webscout bot
  • WeSEE bot
  • wget unix CLI http client
  • wordpress WordPress spider
  • WormlyBot
  • Wotbox
  • Xenu Link Sleuth
  • Xing bot
  • XoviBot
  • YaCy bot
  • Yahoo Ad monitoring
  • Yandex bot
  • Naver Corp
  • YOURLS
  • zelist.ro feed parser
  • zibb ZIBB spider
  • Zite bot
  • Zyborg bot

Do you want to report a bot or a spider? Contact us!