Web Spiders, for webmasters Options

[NickTheGreek]

This is my - altered - web spider list :

googlebot=Google.com
slurp@inktomi=Hot Bot
ask jeeves=Ask Jeeves
lycos=Lycos.com
whatuseek=What You Seek
ia_archiver=Archive.org
msnbot=MSN.com
MSN.com=msnbot.msn.com

AbachoBOT=abacho.com
AESOP_com_SpiderMan=aesop.com
ah-ha.com crawler=ah-ha.com
ia_archiver=alexa.com
Scooter=altavista.com
Mercator=altavista.com
Scooter2_Mercator_3-1.0=altavista.com
roach.smo.av.com-1.0=altavista.com
AltaVista-Intranet=altavista.co.uk
FAST-WebCrawler=alltheweb.com
Wget=Unknown
Acoon Robot=acoon.de
Antibot=antisearch.net
Atomz=atomz.com
AxmoRobot=axmo.com
Buscaplus Robi=buscaplus.com
CanSeek=canseek.ca
ChristCRAWLER=christcrawler.com
Crawler=crawler.de
DaAdLe.com ROBOT=daadle.com
DeepIndex=en.deepindex.com
DittoSpyder=ditto.com
Jack=domanova.co.uk
Speedy Spider=entireweb.com
ArchitextSpider=excite.com
ArchitectSpider=Excite
Arachnoidea=euroseek.net
EZResult=ezresults.com
Fast PartnerSite Crawler=fastsearch.net
FAST Data Search Crawler=fastsearch.net
KIT-Fireball=fireball.de
FyberSearch=fybersearch.com
GalaxyBot=galaxy.com
Geckobot=geckobot.com
GenCrawler=gendoor.com
GeonaBot=geona.com
moget=goo.ne.jp
Aranha=girafa.com
Slurp.so=inktomi
Slurp=Inktomi Slurp
Toutatis 2.5-2=hoppa.com
Hubater=hubat.com
IlTrovatore-Setaccio=iltrovatore.it
IncyWincy=incywincy.com
UltraSeek=infoseek.com
InfoSeek Sidewinder=infoseek.com
Mole2=intags.de
MP3Bot=mp3bot.de
C-PBWF-ip3000.com-crawler=ip3000.com
ip3000.com-crawler=ip3000.com
kuloko-bot=kuloko.com
LNSpiderguy=lexis-nexis.com
lookbot=look.com
MantraAgent=looksmart.com
NetResearchServer=loopimprovements.com
Lycos_Spider_(T-Rex)=lycos.com
JoocerBot=joocer.com
HenryTheMiragoRobot=mirago.co.uk
mozDex=mozdex.com
Gulliver=northernlight.com
ObjectsSearch=objectssearch.com
PicoSearch=picosearch.com
PJspider=portaljuice.com
DIIbot=powerinter.net
nttdirectory_robot=navi.ocn.ne.jp
griffon=navi.ocn.ne.jp
Spider/maxbot.com=maxbot.com
various=Multiple Robots
gazz=Unknown
NationalDirectory-SuperSpider=nationaldirectory.com
dloader(NaverRobot)=naver.com
dumrobo(NaverRobot)=naver.com
Openfind piranha=openfind.com
Shark=openfind.com
Openbot=Chinese Robot
psbot=picsearch.org
CrawlerBoy=pinpoint.com
Pinpoint.com=pinpoint.com
AlkalineBOT=vestris.com
Fluffy the spider=searchhippo.com
Scrubby=scrubtheweb.com
asterias=singingfish.com
speedfind=speedfind.de
ramBot=speedfind.de
xtreme=speedfind.de
Kototoi=s.u-tokyo.ac.jp
Searchspider=searchspider.com
SightQuestBot=sightquest.com
Spider_Monkey=spidermonkey.ca
Surfnomore Spider v1.1=surfnomore.com
Robot@SuperSnooper.Com=supersnooper.com
teoma_agent1=teoma.com=teoma.com
Teradex_Mapper=mapper.teradex.com
ESISmartSpider=travel-finder.com
Spider TraficDublu=traficdublu.ro
Tutorial Crawler=tutorgig.com
UK Searcher Spider=uksearcher.co.uk
Vivante Link Checker=vivante.com
appie=walhello.com
Nazilla=websmostlinked.com
www.WebWombat.com.au=webwombat.com.au
marvin/infoseek=webseek.de
MuscatFerret=webtop.com
WhizBang! Lab=whizbanglabs.com
ZyBorg=wisenut.com
WIRE WebRefiner:=wire.co.uk
WSCbot=worldsearchcenter.com
Yandex=yandex.com
Yellopet-Spider=yellowpet.com
libwww-perl=client sites
Iron33=verno.ueda.info.waseda.ac.jp

Attached File(s)

 ROBOT_LIST.txt ( 3.14K ) Number of downloads: 12