Web Spiders, for webmasters Options

[NickTheGreek]

This is my - altered - web spider list :

googlebot=Google.com
slurp@inktomi=Hot Bot
ask jeeves=Ask Jeeves
lycos=Lycos.com
whatuseek=What You Seek
ia_archiver=Archive.org
msnbot=MSN.com
MSN.com=msnbot.msn.com

AbachoBOT=abacho.com
AESOP_com_SpiderMan=aesop.com
ah-ha.com crawler=ah-ha.com
ia_archiver=alexa.com
Scooter=altavista.com
Mercator=altavista.com
Scooter2_Mercator_3-1.0=altavista.com
roach.smo.av.com-1.0=altavista.com
AltaVista-Intranet=altavista.co.uk
FAST-WebCrawler=alltheweb.com
Wget=Unknown
Acoon Robot=acoon.de
Antibot=antisearch.net
Atomz=atomz.com
AxmoRobot=axmo.com
Buscaplus Robi=buscaplus.com
CanSeek=canseek.ca
ChristCRAWLER=christcrawler.com
Crawler=crawler.de
DaAdLe.com ROBOT=daadle.com
DeepIndex=en.deepindex.com
DittoSpyder=ditto.com
Jack=domanova.co.uk
Speedy Spider=entireweb.com
ArchitextSpider=excite.com
ArchitectSpider=Excite
Arachnoidea=euroseek.net
EZResult=ezresults.com
Fast PartnerSite Crawler=fastsearch.net
FAST Data Search Crawler=fastsearch.net
KIT-Fireball=fireball.de
FyberSearch=fybersearch.com
GalaxyBot=galaxy.com
Geckobot=geckobot.com
GenCrawler=gendoor.com
GeonaBot=geona.com
moget=goo.ne.jp
Aranha=girafa.com
Slurp.so=inktomi
Slurp=Inktomi Slurp
Toutatis 2.5-2=hoppa.com
Hubater=hubat.com
IlTrovatore-Setaccio=iltrovatore.it
IncyWincy=incywincy.com
UltraSeek=infoseek.com
InfoSeek Sidewinder=infoseek.com
Mole2=intags.de
MP3Bot=mp3bot.de
C-PBWF-ip3000.com-crawler=ip3000.com
ip3000.com-crawler=ip3000.com
kuloko-bot=kuloko.com
LNSpiderguy=lexis-nexis.com
lookbot=look.com
MantraAgent=looksmart.com
NetResearchServer=loopimprovements.com
Lycos_Spider_(T-Rex)=lycos.com
JoocerBot=joocer.com
HenryTheMiragoRobot=mirago.co.uk
mozDex=mozdex.com
Gulliver=northernlight.com
ObjectsSearch=objectssearch.com
PicoSearch=picosearch.com
PJspider=portaljuice.com
DIIbot=powerinter.net
nttdirectory_robot=navi.ocn.ne.jp
griffon=navi.ocn.ne.jp
Spider/maxbot.com=maxbot.com
various=Multiple Robots
gazz=Unknown
NationalDirectory-SuperSpider=nationaldirectory.com
dloader(NaverRobot)=naver.com
dumrobo(NaverRobot)=naver.com
Openfind piranha=openfind.com
Shark=openfind.com
Openbot=Chinese Robot
psbot=picsearch.org
CrawlerBoy=pinpoint.com
Pinpoint.com=pinpoint.com
AlkalineBOT=vestris.com
Fluffy the spider=searchhippo.com
Scrubby=scrubtheweb.com
asterias=singingfish.com
speedfind=speedfind.de
ramBot=speedfind.de
xtreme=speedfind.de
Kototoi=s.u-tokyo.ac.jp
Searchspider=searchspider.com
SightQuestBot=sightquest.com
Spider_Monkey=spidermonkey.ca
Surfnomore Spider v1.1=surfnomore.com
Robot@SuperSnooper.Com=supersnooper.com
teoma_agent1=teoma.com=teoma.com
Teradex_Mapper=mapper.teradex.com
ESISmartSpider=travel-finder.com
Spider TraficDublu=traficdublu.ro
Tutorial Crawler=tutorgig.com
UK Searcher Spider=uksearcher.co.uk
Vivante Link Checker=vivante.com
appie=walhello.com
Nazilla=websmostlinked.com
www.WebWombat.com.au=webwombat.com.au
marvin/infoseek=webseek.de
MuscatFerret=webtop.com
WhizBang! Lab=whizbanglabs.com
ZyBorg=wisenut.com
WIRE WebRefiner:=wire.co.uk
WSCbot=worldsearchcenter.com
Yandex=yandex.com
Yellopet-Spider=yellowpet.com
libwww-perl=client sites
Iron33=verno.ueda.info.waseda.ac.jp

Attached File(s)

 ROBOT_LIST.txt ( 3.14K ) Number of downloads: 12

 

[Ye®o©]

Okay try mine.

CODE

AbachoBOT (Mozilla compatible)=Crawler.de
AbachoBOT=Crawler.de
abcdatos_botlink=Abcdatos.com
abot/0.1 (abot; http://www.abot.com; abot@abot.com)=Abot.com
abot/0.1=Abot.com
About/0.1libwww-perl/5.47=About.com
accoona=Accoona.com
AcoiRobot=Acoi Picture Robot
Acoon Robot v1.50.001=Acoon.de
Acoon Robot v1.52=Acoon.de
Acoon Robot=Accon.de
ActiveWorlds/3.xx=ActiveWorlds.com
admin@crawler.de=Crawler.de
admin@maxbot.com=maxbot.com
ADSARobot=ADSARobot
AESOP_com_SpiderMan=Aesop.com
agadine/1.x.x=Agada.de
Agent-admin/=Daum.net
ah-ha.com crawler (crawler@ah-ha.com)=Ah-ha.com
ah-ha.com crawler=Ah-ha.com
AIBOT/2.1=21Seek.com
aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)=NameProtect.com
aipbot/1.0=NameProtect.com
aipbot=NameProtect.com
Aladin/3.324=Aladin.de
Aleksika Spider/1.0=Aleksika.de
AlkalineBOT/1.3=Vestris.com
AlkalineBOT/1.4=Vestris.com
AlkalineBOT=AlkalineBOT
Allesklar/0.1 libwww-perl/5.46=Allesklar.de
AltaVista-Intranet=Altavista.co.uk
AmfibiBOT=Amfibi.com
Amiga-AWeb/3.4.167SE=AWeb.com
amzn_assoc=Amazon.com
AnswerChase PROve x.0=AnswerChase.com
AnswerChase x.0=AnwserChase.com
AnswerChase=AnswerChase.com
Antibot=AntiSeach.net
antibot-V1.1.5/i586-linux-2.2=AntiSeach.net
AnzwersCrawl/2.0=Anzwers
A-Online Search=Jet2Web.com
Aport=Aport.ru
appie 1.1=Walhello.com
appie=Walhello.com
arachnoidea@euroseek.net=Euroseek.net
Arachnoidea=Euroseek.com
Aranha=Girafa.com
ArchitectSpider=Excite.com
archive.org_bot=Archive.org
archive_org=Archive.org
ask jeeves=Ask Jeeves
ask.24x.info=Ask.24x.info
ASPSeek/1.2.5=ASPSeek.org
ASPseek/1.2.9d=ASPSeek.com
ASPSeek/1.2.x=ASPSeek.org
ASPSeek/1.2.xa=ASPSeek.org
ASPseek/1.2.xx=ASPseek.org
ASPSeek/1.2.xxpre=ASPSeek.org
ASPSeek=ASPSeek.org
asterias/2.0=SingingFish.com
Asterias=SingingFish.com
AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)=Atlocal.com
AtlocalBot=Atlocal.com
Atomz/1.0=Atomz.com
Atomz=Atomz.com
Augurfind=Augurnet.ch
augurnfind V-1.x=Augurnet.ch
autohttp=Elsop.com
AV Fetch 1.0=Altavista.com
AVSearch=Altavista.com
AVSearch-3.0=Altavista.com
AxmoRobot=Axmo.com
Baiduspider+=Baidu.com
BaiDuSpider=Baidu.com
BanBots/1.2=Banbots.com
BarraHomeCrawler=BarraHome.com
bdcindexer_2.6.2=Business.com
BDFetch=Brandimensions.com
BDNcentral Crawler v2.3=Bdncentral.com
beautybot/1.0=Cosmoty.com
BigCliqueBOT/1.03-dev=BigClique.com
Bilbo/2.3b-UNIX=Biblo Bot
BlackWidow=BlackWidow Bot
Blaiz-Bee/1.0 (+http://www.blaiz.net)=Blaiz.net
Blaiz-Bee/1.0=Blaiz.net
BlitzBOT@tricus.net (Mozilla compatible)=Blitzsuche.rp-online.de
BlitzBOT@tricus.net=Blitzsuche.rp-online.de
BlogBot/1.x=Blogdex.net
BlogBot=Blogdex.net
Bloglines Title Fetch/1.0 (http://www.bloglines.com)=Bloglines.com
blogWatcher_Spider/0.1=Blogwatcher.pi.titech.ac.jp
BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)=IceRocket.com
BlogzIce/1.0 +http://www.icerocket.com/=IceRocket.com
BlogzIce/1.0=IceRocket.com
boitho.com-dc/0.xx=Boitho.com
boitho.com-robot/1.x=Boitho.com
BSDSeek/1.0=Inktomi.com
BullsEye=Intelliseek.com
bumblebee@relevare.com=Relevare.com
Buscaplus Robi/1.0=Buscaplus.com
Buscaplus Robi=Buscaplus.com
CanSeek/=Cansee.ca
ChristCRAWLER 2.0=Christcrawler.com
ChristCRAWLER=Christcrawler.com
CipinetBot=Cipinet.com
citenikbot/=Cenik.co.uk
ClariaBot/1.0=Searchscout.com
Claymont.com=Claymont.com
CLIPS-index=Cips-index.imag.fr
Clushbot/3.3-BinaryFury (+http://www.clush.com/bot.html)=Clush.com
Clushbot/3.3-BinaryFury=Clush.com
Combine/x.0=Combine Bot
Computer_and_Automation_Research_Institute_Crawler=Research Bot
contact/jylee@kies.co.kr=Kies.co.kr
Convera Internet Spider V6.x=Convera.com
cosmos/0.8_(robot@xyleme.com)=Xyleme.com
cosmos/0.9_(robot@xyleme.com)=Xyleme.com
cosmos=Xyleme.com
C-PBWF-ip3000.com-crawler=IP3000.com
CrawlConvera0.1=Converna.net
Crawler (cometsearch@cometsystems.com)=Findwhat.com
Crawler admin@crawler.de=Crawler.de
Crawler V 0.2.x admin@crawler.de=Crawler.de
crawler@alexa.com=Alexa.com
crawler@brainbot.com=Brainbot.com
crawler@fast.no=Alltheweb.com
Crawler=Crawler.de
CrawlerBoy Pinpoint.com=Pinpoint.com
CrawlerBoy=Pinpoint.com
CrocCrawler= CrocCrawler.com
Custom Spider www.bisnisseek.com /1.0=Bisnisseek.com
DaAdLe.com ROBOT/=Daadle.com
DataparkSearch/4.xx=DataparkSearch.com
DaviesBot/1.7=Wholeweb.net
DaviesBot=Wholeweb.net
dbDig=Prairielandconsulting.com
DBrowse 1.4b=DBrowse Bot
DBrowse 1.4d=DBrowse Bot
dCSbot/1.1=Openmarket.com
DeepIndex=DeepIndex.com
DeepIndexer.ca=DeepIndex.com
deepweb=Deepweb.com
Demo Bot DOT 16b= Btopenworld.com
DiaGem/=Skyrocket.gr.jp
DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)=Skyrocket.gr.jp
DiaGem/1.1=Skyrocket.gr.jp
Diamond/1.0=Searchscout.com
Digger/1.0 JDK/1.3.0rc3=Diggit.com
Digimarc WebReader=Digimarc.com
DigOut4U=Arisem.com
DIIbot/1.2=Digital-Integrity.com
DIIbot=Digital-Integrity.com
DittoSpyder=Ditto.com
dloader(NaverRobot)/=Naver.com
dtSearchSpider=Dtsearch.com
Dumbot(version 0.1 beta - dumbfind.com)=Dumbfind.com
Dumbot(version 0.1 beta)=Dumbfind.com
dumrobo(NaverRobot)/=Naver.com
EARTHCOM.info/1.4beta=Earthcom.info
EasyDL/3.02=Keywen.com
EasyDL/3.03=Keywen.com
EasyDL/3.04=Keywen.com
EasyDL/3.xx http://keywen.com/Encyclopedia/Bot=Keywen.com
EasyDL/3.xx=Keywen.com
EchO!/2.0=Voila.com
egothor/3.0a (+http://www.xdefine.org/robot.html)=Xdefine.com
egothor/3.0a=Xdefine.com
EgotoBot/4.8=Egoto.com
Enterprise_Search/1.0.xxx=Innerprise.net
Enterprise_Search/1.0=Innerprise.net
erik@malfunction.org=Malfunction.org
EroCrawler=Erocrawler.com
ES.NET_Crawler/2.0=Innerprise.net
eseek-larbin_2.6.2=Exactseek.com
ESISmartSpider=Travel-Finder.com
ExactSeek Crawler/0.1=Exactseek.com
exactseek-crawler-2.63=Exactseek.com
Exalead NG/MimeLive Client=Exabot.com
Excalibur Internet Spider V6.5.4=Excalib.com
ExperimentalHenrytheMiragoRobot=Mirago.co.uk
EyeCatcher (Download-tipp.de)/1.0=Download-tipp.de
EyeCatcher=Download-tipp.de
EZResult=Ezresults.com
Fast Crawler Gold Edition=Alltheweb.com
FAST Data Search Crawler=Alltheweb.com
FAST Enterprise Crawler 6 (Experimental)=Alltheweb.com
FAST Enterprise Crawler 6 used by FAST (FAST)=Alltheweb.com
FAST Enterprise Crawler 6 used by FAST=Alltheweb.com
FAST Enterprise Crawler 6=Alltheweb.com
FAST Enterprise Crawler/6 (www.fastsearch.com)=Alltheweb.com
FAST Enterprise Crawler/6=Alltheweb.com
FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)=Alltheweb.com
FAST FirstPage retriever=Alltheweb.com
Fast PartnerSite Crawler=Alltheweb.com
FastCrawler 3.0.1 (crawler@1klik.dk)=Alltheweb.com
FastCrawler 3.0.1=Alltheweb.com
FAST-WebCrawler/2.2.10=Altheweb.com
FAST-WebCrawler/2.2.6=Altheweb.com
FAST-WebCrawler/2.2.7=Altheweb.com
FAST-WebCrawler/2.2.8=Altheweb.com
FAST-WebCrawler/3.2 test=Altheweb.com
FAST-WebCrawler/3.2=Altheweb.com
FAST-WebCrawler/3.3=Altheweb.com
FAST-WebCrawler/3.4/Nirvana=Altheweb.com
FAST-WebCrawler/3.4/PartnerSite=Altheweb.com
FAST-WebCrawler/3.5=Altheweb.com
FAST-WebCrawler/3.6/FirstPage=Altheweb.com
FAST-WebCrawler/3.6=Altheweb.com
FAST-WebCrawler/3.7/FirstPage=Altheweb.com
FAST-WebCrawler/3.7=Altheweb.com
FAST-WebCrawler/3.8/Fresh=Altheweb.com
FAST-WebCrawler/3.8=Altheweb.com
FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)=Altheweb.com
FAST-WebCrawler/3.x Multimedia=Altheweb.com
FAST-WebCrawler=Alltheweb.com
Faxobot/1.0=FaXo.com
Felix - Mixcat Crawler=Mixcat.com
Felix=Mixcat.com
FFC Trap Door Spider=Frequentfinders.com
fido/1.0 Harvest/1.4.pl2=Planet Search
FinaleRobot=expressus.com
FindAnISP.com_ISP_Finder_v99a=FindAnISP.com
FineBot=Finesearch.com
Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)=Fireball.de
Firefly/1.0=Fireball.de
FlickBot 2.0 RPT-HTTPClient/0.3-3=DivX.com
Fluffy the spider=SearchHippo.com
Flunky=Metacarta.com
Francis/1.0 (francis@neomo.de http://www.neomo.de/)=Neomo.de
Francis/1.0=Neomo.de
FreeFind.com-SiteSearchEngine/1.0=Freefind.com
FyberSearch=Fybersearch.com
FyberSpider=Fybersearch.com
Gagglebot=Gagglebot
GAIS Robot/1.0B2=Seed.net.tw
Gaisbot/3.0+=Seed.net.tw
Gaisbot/3.0=Seed.net.tw
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)=Galaxy.com
GalaxyBot/1.0=Galaxy.com
GalaxyBot=Galaxy.com
GammaSpider/1.0=Gammasite.com
gazz/1.0=Infobee.ne.jp
gazz@nttrd.com=Infobee.ne.jp
Geckobot=Geckobot.com
geckobot=Geckobot.com
GenCrawler=Gendoor.com
generic_crawler/01.0217/=Cmu.edu
GentleSpider=Research.att.com
GeonaBot=Geona.com
GigaBaz=Brainbot.com
GigaBazVStheWeb=Brainbot.com
GigaBlast=Gigablast.com
Gigabot/1.0=Gigablast.com
Gigabot=Gigablast.com
GigabotSiteSearch/2.0 (sitesearch.gigablast.com)=Gigablast.com
GigabotSiteSearch/2.0=Gigablast.com
Giskard=oralco.com
GNODSPIDER=Gnod.net
Goblin/0.9 (http://www.goguides.org/)=Goguides.org
Goblin/0.9.x (http://www.goguides.org/goblin-info.html)=Goguides.org
Goblin/0.9.x=Goguides.org
Goblin/0.9=Goguides.org
GoForIt.com=Goforit.com
Goofer/0.2=Wanadoo.fr
Google=GoogleBot
Googlebot/1.0=GoogleBot
Googlebot/2.1= GoogleBot
Googlebot/Test=GoogleBot
googlebot@googlebot.com=GoogleBot
Googlebot=GoogleBot
Googlebot-Image/1.0=Google.com Image Bot
GrigorBot 0.8=Grigor.biz
grub crawler(http://www.grub.org)=Grub.com
grub-client=Grub.org
gsa-crawler (Enterprise; GID-01422; jplastiras@google.com)=Google.com
gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)=Google.com
gsa-crawler=Google.com
Gulliver/1.2=Northernlight.com
Gulliver/1.3=Northernlight.com
Gulliver=Northernlight.com
Gulper Web Bot 0.2.4=Gulper Bot
Gulper Web Bot=Gulper Bot
Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)=a.hatena.ne.jp
Hatena Antenna/0.4=a.hatena.ne.jp
HeinrichderMiragoRobot=Mirago.de
HenrytheMiragoRobot=Mirago.co.uk
HenryTheMiragoRobot=Mirago.co.uk
Hippias/0.9 Beta= Hippias Bot
holmes/x.x=Morfeo.centrum.cz
HomePageSearch(hpsearch.uni-trier.de)=Hpsearch.uni-trier.de
Homerbot: www.homerweb.com=Homerweb.com
Homerbot=Homerweb.com
htdig/3.1.6 (http://computerorgs.com)=Computerorgs.com
htdig/3.1.x (root@localhost)=Htdig.org
htdig/3.1.x=Htdig.org
http://www.abcdatos.com/botlink/=Abcdatos.com
Hubater=Hubat.com
ia_archiver/1.6=Archive.org
ia_archiver=Archive.org
IAArchiver-1.0=Archive.org
icsbot-0.1=Icseoul.org
ideare - SignSite/1.x=Ideare.com
Ideare - SignSite=Ideare.com
IlTrovatore-Setaccio (+http://www.iltrovatore.it)=Iltrovatore.it
Iltrovatore-Setaccio/0.3-dev=Iltrovatore.it
IlTrovatore-Setaccio/1.2=Iltrovatore.it
iltrovatore-setaccio/1.2-dev=Iltrovatore.it
IlTrovatore-Setaccio=Iltrovatore.it
IncyWincy=Incywincy.com
IndexTheWeb.com Crawler7=IndexTheWeb.com
Inet library=Inetlibrary.com
info@searchhippo.com=Searchhippo.com
InfoFly/1.0 (http://www.versions-project.org/)=Versions-project.org
InfoFly/1.0= Versions-project.org
InfoSeek Sidewinder/0.9=Infoseek.com
InfoSeek Sidewinder/1.0A= Infoseek.com
InfoSeek Sidewinder/1.1A= Infoseek.com
Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)= Infoseek.com
Infoseek SideWinder/1.45= Infoseek.com
Infoseek SideWinder/2.0B (Linux 2.4 i686)= Infoseek.com
Infoseek SideWinder/2.0B= Infoseek.com
InfoSeek Sidewinder=infoseek.com
Inktomi Search=Yahoo.com
Inktomi=Yahoo.com
InnerpriseBot/1.0 (http://www.innerprise.com/)=Innerprise.net
InnerpriseBot/1.0=Innerprise.net
InsumaScout=InsumaScout
Internet Ninja 6.0=Dti.ne.jp
InternetSeer.com= InternetSeer.com
InternetSeer=InternetSeer
InternetSeer.com=AOL Search
ip3000.com=ip3000.com
ip3000.com-crawler=ip3000.com
IPiumBot laurion(dot)com=Laurion.com
ipiumBot=Laurion.com
Iron33=Verno.ueda.info.waseda.ac.jp
Jabot/6.x (http://odin.ingrid.org/)=Ingrid.org
Jabot/6.x=Ingrid.org
Jabot/7.x.x (http://odin.ingrid.org/)=Ingrid.org
Jabot/7.x.x=Ingrid.org
Jack=Domanova.co.uk
jan.gelin@av.com=Altavista.co.uk
jeeves=Ask Jeeves
Jetbot/1.0=Jeteye.com
JoocerBot=Joocer.com
Jyxobot/x=Jyxo.cz
Katriona=Katriona
KE_1.0/2.0 libwww/5.2.8=Voila.fr
Kenjin Spider=Kenjin.ne.jp
KIT_Fireball/2.0=Fireball.de
KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)=Fireball.de
KIT-Fireball/2.0=Fireball.de
KIT-Fireball=Fireball.de
Knowledge.com/0.x=Knowledge.com
Kototoi/0.1=s.u-tokyo.ac.jp
kuloko-bot/0.2=Kuloko.com
kuloko-bot/0.x=Kuloko.com
Kulokobot=Kuloko.com
kulturarw3/0.1=National Library of Sweden
LEIA/3.01pr (LEIAcrawler; [SNIP])=Gseek.com
LEIA=Gseek.com
LexiBot=Lexibot.com
LNSpiderguy=Lexis-nexis.com
Look.com=Look.com
lwp-trivial/1.27=Search4free.com
lwp-trivial/1.34=Search4free.com
Lycos_Spider_(T-Rex)/3.0=Lycos.com
Lycos_Spider_(T-Rex)=Lycos.com
lycos=Lycos.com
Mackster( http://www.ukwizz.com )=Ukwizz.com
Mackster=Ukwizz.com
Mag-Net=Wind.it
mailto:webcraft@bea.com=Beasys.com
mammoth/1.0 (+http://www.sli-systems.com/)=Sli-Systems.com
mammoth/1.0=Sli-Systems.com
MantraAgent=Looksmart.com
mapper@teradex.com=Mapper.teradex.com
Mariner/5.1b [de] (Win95; I;Kolibri gncwebbot)=Kolibri.de
Mariner/5.1b= Kolibri.de
Martini=Looksmart.com
Marvin v0.3=Marvin Bot
marvin/infoseek=Webseek.de
marvin-team@webseek.de=Webseek.de
Mata Hari/2.00=Lexibot.com
MediaCrawler-1.0 (Experimental)=Mediacrawler.de
MediaCrawler-1.0=Mediacrawler.de
Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)=Google.com Adsense
Mediapartners-Google/2.1=Google.com Adsense
MediaSearch/0.1=Www.fi
MegaSheep v1.0 (www.searchuk.com internet sheep)=SearchUK.com
MegaSheep v1.0=SearchUK.com
Mercator=Altavista.com
Mercator=Altavista.com
Mercator-1.x=Altavista.com
Mercator-2.0=Altavista.com
Mercator-Scrub-1.1=Altavista.com
metabot=Metabot
Metaspinner/0.01=Meta-Spinner.de
MFC_Tear_Sample=Microsoft.com
MicrosoftPrototypeCrawler=Microsoft.com
Minibot=Minibot
MiracleAlphaTest=Nec.co.jp
MnogoSearch/3.2.xx=mNoGoSearch.org
moget/2.0=Goo.ne.jp
moget/2.1 (moget@goo.ne.jp)=Goo.ne.jp
moget/2.1=Goo.ne.jp
moget@goo.ne.jp=Goo.ne.jp
Mole2/1.0=Intags.de
Morris - Mixcat Crawler (+http://mixcat.com)=Mixcat.com
Morris - Mixcat Crawler=Mixcat.com
Morris=Mixcat.com
Mouse-House/7.4=Spidermonkey.ca
mozDex/=Mozdex.com
mozDex/0.04-dev=Mozdex.com
Mozilla (Mozilla@somewhere.com)=Somewhere.com
Mozilla/2.0 (compatible; Ask Jeeves)=Ask Jeeves
Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)=DirectHit.com
Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)=Euroferret.com
Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)=Euroferret.com
Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)=Euroferret.com
Mozilla/3.01 (Compatible; Links2Go Similarity Engine)=Links2go.com
Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)=AllTheWeb.com
Mozilla/4.0 (compatible; grub-client-2.x)=Grub.org
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5=Echo.com
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)=MSN.com
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)=EntireWeb.com
Mozilla/4.0 (compatible; www.galaxy.com)=Galaxy.com
Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)=Ask Jeeves
Mozilla/4.0(compatible; Zealbot 1.0)=LookSmart.com
mozilla/5.0 (compatible; genevabot +http://www.healthdash.com)=HealthDash.com
Mozilla/5.0 (wgao@genieknows.com)=GenieKnows.com
Mozilla/5.0 wgao@genieknows.com=GenieKnows.com
MP3Bot=mp3bot.de
MSN=MSN Bot
MSNBOT/0.1=MSN Bot
msnbot=MSN Bot
MultiText/0.1=MultiText
MultiText=MultiText
MuscatFerret=Webtop.com
NationalDirectoryAddURL/1.0=NationalDirectory.com
NationalDirectory-SuperSpider=Nationaldirectory.com
NationalDirectory-WebSpider/1.3=NationalDirectory.com
Nazilla=Websmostlinked.com
NEC Research Agent=NEC Research Agent
NetCruiser=NetCruiser
NetResearchServer(http://www.look.com)=Look.com
NetResearchServer=Look.com
NetZippy=Netzippy.com
NextopiaBOT=Nextopia.com
NG/1.0=Exabot.com
Noago Spider=Noago.com
NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com)=NameProtect.com
NP/0.1=NameProtect.com
NPBot=NameProtect.com
NPBot-1/2.0=NameProtect.com
nttdirectory_robot=Navi.ocn.ne.jp
Nutch=Nutch.org
NutchCVS/0.0x-dev=Nutch.org
NutchOrg/0.0x-dev=Nutch.org
NutchOrg=Nutch.org
NZBot=NavigationZone.com
ObjectsSearch/0.01=ObjectsSearch.com
ObjectsSearch/0.0x=ObjectsSearch.com
oBot ((compatible;Win32))=Cobion.com
oBot=Cobion.com
OliverPerry=Claymont.com
OntoSpider/1.0 libwww-perl/5.65=Ontospider.i-n.info
OntoSpider/1.0=Ontospider.i-n.info
OntoSpider=Ontospider.i-n.info
Openbot/=Openfind.com.tw
Openbot/3.0+=Openfind.com.tw
Openbot/3.0=Openfind.com.tw
Opencola=Opencola.com
Openfind piranha=Openfind.com
Openfind Robot/1.1A2=Openfind.com.tw
OpenTextSiteCrawler/2.9.2=Opentext.net
Oracle Ultra Search=Oracle.com
Oxxbot1=Oxxfordinfo.com
Pansophica=Pansophica
ParaSite/1.0b (http://www.ianett.com/parasite/)=Ianett.com
ParaSite/1.0b=Ianett.com
pd02_1.0.0 pd02_1.0.0@dzimi@post.sk=Post.sk
peerbot=Peerbot.com
Phoaks=Phoaks
phortse@hanmail.net=Daum.net
PICgrabber=PICgrabber
PicoSearch/=PicoSearch.com
PicoSearch/1.0=PicoSearch.com
PictureOfInternet=PictureOfInternet
PintaSpider=PintaSpider
pipeLiner/0.10=Pipeline-Search.com
pipeLiner=Pipeline-Search.com
Pita=Pita Bot
Pizilla++ ver 2.45=He.net
PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)=Portaljuice.com
PJspider/3.0=Portaljuice.com
PJspider=Portaljuice.com
polybot 1.0 (http://cis.poly.edu/polybot/)= PolyBot
polybot 1.0=PolyBot
PolyBot=PolyBot
Pompos/1.x http://dir.com/pompos.html=Dir.com
Pompos/1.x pompos@iliad.fr=Iliad.fr
Pompos/1.x=Free.fr
Popdexter/1.0=Popdex.com
PortalBSpider/2.0 (spider@portalb.com)=Portalb.com
PortalBSpider/2.0=Portalb.com
Project XP5 [2.03.07-111203]= XP5 Bot
Project XP5=XP5 Bot
PROve AnswerBot 4.0=Answerchase.com
psbot/0.1 (+http://www.picsearch.com/bot.html)=Picsearch.com
psbot/0.1=Picsearch.com
psbot=Picsearch.com
Qango.com Web Directory (http://www.qango.com/)=Qango.com
Qango.com Web Directory= Qango.com
QPCreep Test Rig ( We are not indexing, just testing ) =Quepasa.com
QPCreep Test Rig=Quepasa.com
QuepasaCreep ( crawler@quepasacorp.com )=Quepasa.com
QuepasaCreep v0.9.1x=Quepasa.com
QuepasaCreep=Quepasa.com
QueryN Metasearch=Queryn.com
rabaz (rabaz at gigabaz dot com)=Brainbot.com
Rabaz=Brainbot.com
RaBot=Daum.net
ramBot xtreme x.x=Intersearch.de
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)=Sitesearch.ca
Reaper [2.03.10-031204]=Sitesearch.ca
Reaper/2.0x (+http://www.sitesearch.ca/reaper)=Sitesearch.ca
Reaper/2.0x=Sitesearch.ca
Reaper=Sitesearch.ca
roach.smo.av.com-1.0=Altavista.com
roach.smo.av.com-1.0=Altavista.com
RoboCrawl (www.canadiancontent.net)=CanadianContent.net
RoboCrawl=CanadianContent.net
Robot/www.pj-search.com=PJ-Search.com
Robot@SuperSnooper.Com=SuperSnooper.com
robot@xyleme.com=Xyleme.com
robot-master@expressus.com=Expressus.com
robot-response@openfind.com.tw=Openfind.com.tw
Robozilla/1.0=Dmoz.org
Rotondo/3.1 libwww/5.3.1=Qualigo.de
ru-robot=Rutgers.edu
Scooter/1.0 scooter@pa.dec.com=Altavista.com
Scooter/1.0=Altavista.com
Scooter/1.1 (custom)=Altavista.com
Scooter/2.0 G.R.A.B. V1.1.0 =Altavista.com
Scooter/2.0 G.R.A.B. X2.0=Altavista.com
Scooter/3.3.QA.pczukor=Altavista.com
Scooter/3.3.vscooter=Altavista.com
Scooter/3.3_SF=Altavista.com
Scooter/3.3=Altavista.com
Scooter_bh0-3.0.3=Altavista.com
Scooter_trk3-3.0.3=Altavista.com
Scooter=Altavista.com
Scooter2_Mercator_3-1.0=Altavista.com
Scooter2_Mercator_x-x.0=Altavista.com
Scooter-3.0.EU=Altavista.com
Scooter-3.0.FS=Altavista.com
Scooter-3.0.HD=Altavista.com
Scooter-3.0.VNS=Altavista.com
Scooter-3.0QI=Altavista.com
Scooter-3.2.BT=Altavista.com
Scooter-3.2.DIL=Altavista.com
Scooter-3.2.EX=Altavista.com
Scooter-3.2.JT=Altavista.com
Scooter-3.2.NIV=Altavista.com
Scooter-3.2.SF0=Altavista.com
Scooter-3.2.snippet=Altavista.com
Scooter-3.2=Altavista.com
Scooter-3.3dev=Altavista.com
Scooter-ARS-1.1=Altavista.com
Scooter-ARS-1.1-ih=Altavista.com
scooter-venus-3.0.vns=Altavista.com
Scooter-W3.1.2=Altavista.com
Scooter-W3-1.0=Altavista.com
Scrubby/=Scrubtheweb.com
search.at V1.2=Search.at
search.ch V1.4.2=Search.ch
search.ch V1.4=Search.ch
SearchExpress Spider0.99=SearchExpress.com
Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)=Searchit-now.co.uk
Searchit-Now Robot/2.2=Searchit-now.co.uk
SearchSpider.com/1.1=Searchspider.com
Searchspider/=Searchspider.com
Searchspider/1.2=Searchspider.com
sebastien.ailleret@inria.fr=Inria.fr
Seeker.lookseek.com=Lookseek.com
SeznamBot/1.0 (+http://fulltext.seznam.cz/)=Seznam.cz
SeznamBot/1.0=Seznam.cz
Shark=Openfind.com
sherlock_spider=Sherlock.com.cn
SightQuestBot/=Sightquest.com
SiteSpider +(http://www.SiteSpider.com/)=Sitespider.com
SiteSpider=Sitespider.com
SiteXpert=Xtreeme.com
Skampy/0.9.x= Skaffe.com
Skimpy/0.x=Skaffe.com
Slarp/0.1=Only.com
Slider_Search_v1-de=Slider.com
Slurp.so/1.0= Yahoo.com
Slurp/2.0j=Yahoo.com
Slurp/2.0-KiteHourly=Yahoo.com
Slurp/2.0-OwlWeekly=Yahoo.com
Slurp/3.0-AU=Yahoo.com
slurp@inktomi.com=Yahoo.com
slurp@inktomi= Yahoo.com
Slurp=Yahoo.com
SlySearch/1.0=Turnitin.com
SlySearch/1.x=Turnitin.com
SlySearch=Turnitin.com
speedfind ramBot xtreme 8.1=Speedfind.de
speedfind ramBot xtreme=Speedfind.de
Speedy Spider (Beta/x.x; speedy@entireweb.com)=EntireWeb.com
Speedy Spider=EntireWeb.com
Speedy_Spider (http://www.entireweb.com)=EntireWeb.com
Speedy_Spider=EntireWeb.com
Spida/0.1=Only.com
Spider TraficDublu=Traficdublu.ro
Spider/maxbot.com admin@maxbot.com=Maxbot.com
Spider/maxbot.com=Maxbot.com
spider@aeneid.com=Yahoo.com
Spider_Monkey/=Spidermonkey.ca
spider_monkey=Spidermonkey.ca
SpiderMonkey/7.x=Spidermonkey.ca
Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)=Aol.com
Sqworm=Aol.com
Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)=Steeler Bot
Steeler/1.x=Steeler Bot
Steeler=Steeler Bot
Suchbot=Suchbot.de
Suchknecht.at-Robot=Suchknecht.at
super-robot@super.navi.ocn.ne.jp=Navi.ocn.ne.jp
support@canseek.ca=Canseek.ca
Surfnomore Spider v1.1=Surfnomore.com
SwissSearch V1.2=SwissSearch
SynoBot=Synomia.fr
TaWWWantula=TaWWWantula
teoma_admin@hawkholdings.com=Teoma.com
teoma_agent1=Teoma.com
TeraCrawl=TeraCrawl
Teradex_Mapper=Mapper.teradex.com
The Informant=The Informant
TheSuBot/0.1 (www.thesubot.de)=TheSuBot.de
TheSuBot/0.1=TheSuBot.de
thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)=ThumbShots.de
thumbshots-de-Bot=ThumbShots.de
Toutatis 2.5-2=Hoppa.com
Toutatis 3.3 (hoppa.com)=Hoppa.com
Toutatis 4-19.6 (hoppa.com)=Hoppa.com
TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)=Turnitin.com
TurnitinBot/x.x=Turnitin.com
TurnitinBot=Turnitin.com
Tutorial Crawler=Tutorgig.com
Tv<nn>_Merc_resh_26_1_D-1.0=Altavista.com
Tv<nn>_Merc_resh_26_1_D-1.0=Altavista.com
TygoBot=Tygo.com
UCmore=Ucmore.com
UdmSearch=Mnogosearch.org
UK Searcher Spider=Uksearcher.co.uk
Ultraseek=Infoseek.com
unlostBot=Unlost.com
updated/0.1beta=Updated.com
URLBlaze=Urlblaze.net
UrlScope=UrlScope
Vagabondo/1.x MT (webagent@wise-guys.nl)=Wise-guys.nl
Vagabondo/2.0 MT (webagent at wise-guys dot nl)=Wise-guys.nl
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)=Wise-guys.nl
Vagabondo/2.0 MT=Wise-guys.nl
Vagabondo=Wise-guys.nl
vspider=Verity.com
webbandit/4.xx.0=WebBandit
Webbandit=WebBandit
Webclipping.com=WebClipping.com
webcollage/1.xx=WebCollage.com
webcollage=WebCollage.com
WebCompass=WebCompass
WebCrawler/3.0 Robot libwww/5.0a=WebCrawler
WebCrawler-AddURL/2.0=WebCrawler
WebFindBot(http://www.web-find.com)=Web-find.com
WebFindBot=Web-find.com
WebGather=WebGather
WebGenie=WebGenie
webmaster@intags.de=Intags.de
whatUseek_winona/3.0=Whatuseek.com
whatuseek=Whatuseek.com
WhizBang! Lab=Whizbanglabs.com
WIRE WebRefiner=Wire.co.uk
wisenutbot=Looksmart.com
WSCbot=WorldSearchCenter.com
www.inktomisearch.com=Yahoo.com
www.WebWombat.com.au=Webwombat.com.au
YahooSeeker/CafeKelsa=Yahoo! Slurp
Yahoo=Yahoo! Slurp
Yahoo! Slurp=Yahoo! Slurp
Yandex=Yandex.com
Yellopet-Spider=YellowPet.com
YottaShopping_Bot/4.12=YottaShopping.com
Zao/0.1 (http://www.kototoi.org/zao/)=Zao Bot
Zao/0.1=Zao Bot
Zao/0.2=Zao Bot
Zao-Crawler=Zao Bot
zerxbot/Version 0.6 libwww-perl/5.79=Zerx.com
Zippy v2.0 - Zippyfinder.com=Zippyfinder.com
ZyBorg=Wisenut.com
chitika=WinBetas Sponsor
OmniExplorer_Bot/3.96=World Indexer
BecomeBot/2.3=Become.com Search

[NickTheGreek]

i found some more user agents today... i was looking for some information regarding the Septera bot which has been lurking the site for 6 months now... it took me some time to identify it properly !

anyway, here is some more info :

CODE

msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Googlebot/2.1 (+http://www.google.com/bot.html)
MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
LinkWalker
ASPseek/1.2.11pre
ia_archiver
msnbot/0.9 (+http://search.msn.com/msnbot.htm)
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo
Gigabot/2.0/gigablast.com/spider.html
Exabot/2.0
Opera/8.52 (Windows NT 5.1; U; pl)
Septera
LWP::Simple/5.48
MIIxpc/4.2
MnogoSearch/3.2.xx
NationalDirectoryAddURL/1.0
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.htm
Speedy_Spider (http://www.entireweb.com)
InfoSeek Sidewinder/1.0A
JoBo/1.x (http://www.matuschek.net/jobo.html)
LARBIN-EXPERIMENTAL (efp@gmx.net)
Mozi!
Rational SiteCheck (Windows NT)
Robozilla/1.0
Shareaza v1.x.x.xx
SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.htm
Suchknecht.at-Robot
SurveyBot/2.2
larbin_2.6.2 (listonATccDOTgatechDOTedu)
libwww-perl/5.64
Aplix_SEGASATURN_browser/1.x
BSDSeek/1.0
Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net
Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Enterprise_Search/1.0.xxx
ExactSeek Crawler/0.1
FAST Enterprise Crawler 6 used by FAST (FAST)
FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/fa
FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;htt
InfoSeek Sidewinder/1.1A
Infoseek SideWinder/2.0B (Linux 2.4 i686)
Kontiki Client x.xx
LWP::Simple/5.22
Links 2.0 (http://gossamer-threads.com/scripts/links/)
LinksManager.com (http://linksmanager.com/linkchecker.html)
Lotus-Notes/4.5 ( Windows-NT )
Mariner/5.1b [de] (Win95; I;Kolibri gncwebbot)
Marvin v0.3
MegaSheep v1.0 (www.searchuk.com internet sheep)
MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecr
Missigua Locator 1.9
NationalDirectory-WebSpider/1.3
NetResearchServer/4.0(loopimprovements.com/robot.html)
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html;
OpenTextSiteCrawler/2.9.2
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.
Opera/5.11 (Windows ME; U) [ru]
Opera/6.x (Linux 2.4.8-26mdk i686; U) ?[en]
Opera/6.x (Windows NT 4.0; U) [de]
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; ht
PEval 1.4b
PSurf15a 51
PSurf15a VA
PlantyNet_WebRobot_V1.9 dhkang@plantynet.com
Popdexter/1.0
Program Shareware 1.0.2
Progressive Download HTTP check
Rainbot1.1
Reaper/2.0x (+http://www.sitesearch.ca/reaper)
SQ Webscanner
Scooter-3.2
Scooter-ARS-1.1
Scooter/1.0
Scooter/3.3
Scooter2_Mercator_x-x.0
SearchGuild_DMOZ_Experiment (chris@searchguild.com)
SearchSpider.com/1.1
Simpy 1.0; http://www.simpy.com/
Slider_Search_v1-de
Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.ht
Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.c
Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.htm
SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.c
Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Submission Spider at surfsafely.com
TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.htm
billbot wjj@cs.cmu.edu
crawler@alexa.com
iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovato
lwp-trivial/1.34
lwp-trivial/1.41
search.ch V1.4
semanticdiscovery/0.x
speedfind ramBot xtreme 8.1
Aplix HTTP/1.0.1
Arachnoidea (arachnoidea@euroseek.com)
ClariaBot/1.0
DA 4.0 (www.downloadaccelerator.com)
DA 5.0
DBrowse 1.4d
DataparkSearch/4.xx (http://www.dataparksearch.org/)
DoCoMo/1.0/Nxxxi/c10
DoCoMo/1.0/Nxxxi/c10/TB
Download Demon/3.x.x.x
EasyDL/3.xx http://keywen.com/Encyclopedia/Bot
Enterprise_Search/1.0
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/e
ExperimentalHenrytheMiragoRobot
FAST Enterprise Crawler 6 (Experimental)
FAST Enterprise Crawler/6 (www.fastsearch.com)
FAST-WebCrawler/3.2 test
FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.
FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot n
Fast Crawler Gold Edition
Fast PartnerSite Crawler
FastCrawler 3.0.1 (crawler@1klik.dk)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Go!Zilla/4.x.x.xx
Googlebot-Image/1.0
Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
Gulliver/1.3
Html Link Validator (www.lithopssoft.com)
IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/fa
IncyWincy/2.1(loopimprovements.com/robot.html)
IndexTheWeb.com Crawler7
InfoSeek Sidewinder/0.9
InstallShield DigitalWizard
JCheckLinks/0.1 RPT-HTTPClient/0.3-1
Java/1.4.1_01
Java1.3.0rc1
Java1.3.1_15
Java1.3.x
JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html)
KIT-Fireball/2.0
LECodeChecker/3.0 libgetdoc/1.0
LEIA/3.01pr (LEIAcrawler; [SNIP])
LWP::Simple/5.51
LexiBot/1.00
LightningDownload/1.x.x
LightningDownload/1.x.x [Accelerated x]
Lincoln State Web Browser
Link Valet Online 1.x
LinkLint-checkonly/2.x.x
LinkScan/11.0beta2 Unix
LinkScan/x.x Unix
Links (2.xpre7; Linux 2.4.18 i586; x)
Links SQL (http://gossamer-threads.com/scripts/links-sql/)
ListBidBot (freelance job spider http://listbid.com)
Look.com
MSNBOT/0.1x (http://search.msn.com/msnbot.htm)
MSNPTC/1.0
Mackster( http://www.ukwizz.com )
MantraAgent
Mata Hari/2.00
MediaCrawler-1.0 (Experimental)
Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)
Mercator-2.0
Mercator-Scrub-1.1
Missauga Locate 1.0.0
Mister PiX version.dll
Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com
NETCOMplete/x.xx
NG/1.0
NITLE Blog Spider/0.01
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
NetMechanic Vx.0
NetResearchServer/2.x(loopimprovements.com/robot.html)
Netprospector JavaCrawler
NextopiaBOT (+http://www.nextopia.com) distributed crawler cl
Nitro Downloader 1.x (www.klsofttools.com)
Nocilla/1.0
Offline Explorer 1.*
OntoSpider
Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openf
Openfind Robot/1.1A2
Opera/5.12 (Windows 98; U) [en]
Opera/8.50 (Windows NT 5.2; U; en)
Opera/8.51 (Windows NT 5.1; U; cs)
PBrowse 1.4b
PHP/4.0.6
ParaSite/1.0b (http://www.ianett.com/parasite/)
PeopleChat/Search_Engine
PicoSearch/1.0
PingALink Monitoring Services 1.0
PingALink Monitoring Services 1.0 (http://www.pingalink.com)
Pita
PlagiarBot/1.0
Pockey-GetHTML/x.xx
Pockey7.x.x(WIN32GUI)
Pompos/1.x http://dir.com/pompos.html
Pompos/1.x pompos@iliad.fr
PortalBSpider/2.0 (spider@portalb.com)
Privoxy/3.0 (Anonymous)
ProWebGuide Link Checker (http://www.prowebguide.com)
Production Bot 2016B
Project XP5 [2.03.07-111203]
QuepasaCreep v0.9.1x
QueryN Metasearch
RRC (crawler_admin@bigfoot.com)
RSurf15a 51
RaBot/1.0 Agent-admin/phortse@hanmail.net
RealDownload/4.0.0.4x
SSurf15a 11
SafariBookmarkChecker (+http://www.coriolis.ch/)
Science Traveller International 1X/1.0
Scooter-3.0.EU
Scooter-3.0.HD
Scooter-3.0.VNS
Scooter-3.2.BT
Scooter-3.2.DIL
Scooter-3.2.EX
Scooter/1.0 scooter@pa.dec.com
Scooter/3.3.QA.pczukor
Scooter/3.3_SF
SearchTone2.0 - IDEARE
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetche
Seeker.lookseek.com
Sensis.com.au Web Crawler (search_comments\\at\\sensis\\dot\\
SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.co
SiteBar/x.x
SiteWinder
Skimpy/0.x (http://www.skaffe.com/skampy-info.html)
Slarp/0.1
SlimBrowser
SlySearch (slysearch@slysearch.com)
SnykeBot/0.6 (http://www.snyke.com)
Spider/maxbot.com admin@maxbot.com
Spinne/2.0 med_AH
Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux
StackRambler/x.x
Stamina/1.4
Star Downloader
SurferF3 1/0
TeamSoft WinInet Component
Teoma MP
TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.ht
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot
blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWat
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
contype
exactseek-crawler-2.63 (crawler@exactseek.com)
gigabaz/3.14 (baz@gigabaz.com; http://gigabaz.com/gigabaz/)
googlebot (larbin2.6.0@unspecified.mail)
larbin_2.1.1 larbin2.1.1@somewhere.com
larbin_2.2.0 (crawl@compete.com)
larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp)
larbin_2.6.0 (larbin2.6.0@unspecified.mail)
larbin_2.6.2 (hamasaki@grad.nii.ac.jp)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (pimenas@systems.tuc.gr)
larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)
lcabotAccept: */*
libwww-perl/5.52 FP/2.1
libwww-perl/5.63
mammoth/1.0 (+http://www.sli-systems.com/)
msnbot/0.1x (+http://search.msn.com/msnbot.htm)
multiBlocker browser
myDaemon
nabot_1.0
newsearchengine (ThisUser@unspecified.mail)
parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1
pd02_1.0.0 pd02_1.0.0@dzimi@post.sk
potbot 1.0
ramBot xtreme x.x
search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)
spider.batsch.com
suchbot
teomaagent1 [crawler-admin@teoma.com]

[rocksfan]

Just put this robot list in here "Spider Bot User-Agent "

right?