黑帽联盟

 找回密码
 会员注册
查看: 1528|回复: 0
打印 上一主题 下一主题

[其它] 屏蔽掉所有垃圾蜘蛛大全

[复制链接]
yun 黑帽联盟官方人员 

920

主题

37

听众

1364

积分

超级版主

Rank: 8Rank: 8

  • TA的每日心情
    奋斗
    2019-10-18 11:20
  • 签到天数: 678 天

    [LV.9]以坛为家II

    # robots.txtgenerated
    User-agent:larbin
    Disallow: /

    User-agent:b2w/0.1
    Disallow: /

    User-agent:Copernic
    Disallow: /

    User-agent:psbot
    Disallow: /

    User-agent:Python-urllib
    Disallow: /

    User-agent:URL_Spider_Pro
    Disallow: /

    User-agent:CherryPicker
    Disallow: /

    User-agent:EmailCollector
    Disallow: /

    User-agent:EmailSiphon
    Disallow: /

    User-agent:WebBandit
    Disallow: /

    User-agent:EmailWolf
    Disallow: /

    User-agent:ExtractorPro
    Disallow: /

    User-agent:CopyRightCheck
    Disallow: /

    User-agent:Crescent
    Disallow: /

    User-agent:SiteSnagger
    Disallow: /

    User-agent:ProWebWalker
    Disallow: /

    User-agent:CheeseBot
    Disallow: /

    User-agent:LNSpiderguy
    Disallow: /

    User-agent:Alexibot
    Disallow: /

    User-agent:Teleport
    Disallow: /

    User-agent:TeleportPro
    Disallow: /

    User-agent:MIIxpc
    Disallow: /

    User-agent:Telesoft
    Disallow: /

    User-agent:Website Quester
    Disallow: /

    User-agent:WebZip
    Disallow: /

    User-agent:moget/2.1
    Disallow: /

    User-agent:WebZip/4.0
    Disallow: /

    User-agent:WebStripper
    Disallow: /

    User-agent:WebSauger
    Disallow: /

    User-agent:WebCopier
    Disallow: /

    User-agent:NetAnts
    Disallow: /

    User-agent:Mister PiX
    Disallow: /

    User-agent:WebAuto
    Disallow: /

    User-agent:TheNomad
    Disallow: /

    User-agent:WWW-Collector-E
    Disallow: /

    User-agent: RMA
    Disallow: /

    User-agent:libWeb/clsHTTP
    Disallow: /

    User-agent:asterias
    Disallow: /

    User-agent:httplib
    Disallow: /

    User-agent:turingos
    Disallow: /

    User-agent:spanner
    Disallow: /

    User-agent:InfoNaviRobot
    Disallow: /

    User-agent:Harvest/1.5
    Disallow: /

    User-agent:Bullseye/1.0
    Disallow: /

    User-agent:Mozilla/4.0 (compatible; BullsEye; Windows 95)
    Disallow: /

    User-agent:Crescent Internet ToolPak HTTP OLE Control v.1.0
    Disallow: /

    User-agent:CherryPickerSE/1.0
    Disallow: /

    User-agent:CherryPickerElite/1.0
    Disallow: /

    User-agent:WebBandit/3.50
    Disallow: /

    User-agent:NICErsPRO
    Disallow: /

    User-agent:Microsoft URL Control – 5.01.4511
    Disallow: /

    User-agent: NotYour Business!
    Disallow: /

    User-agent:Hidden-Referrer
    Disallow: /

    User-agent:DittoSpyder
    Disallow: /

    User-agent:Foobot
    Disallow: /

    User-agent:WebmasterWorldForumBot
    Disallow: /

    User-agent:SpankBot
    Disallow: /

    User-agent:BotALot
    Disallow: /

    User-agent:lwp-trivial/1.34
    Disallow: /

    User-agent:lwp-trivial
    Disallow: /

    User-agent:BunnySlippers
    Disallow: /

    User-agent:Microsoft URL Control – 6.00.8169
    Disallow: /

    User-agent: URLyWarning
    Disallow: /

    User-agent:Wget/1.6
    Disallow: /

    User-agent:Wget/1.5.3
    Disallow: /

    User-agent: Wget
    Disallow: /

    User-agent:LinkWalker
    Disallow: /

    User-agent:cosmos
    Disallow: /

    User-agent:moget
    Disallow: /

    User-agent:hloader
    Disallow: /

    User-agent:humanlinks
    Disallow: /

    User-agent:LinkextractorPro
    Disallow: /

    User-agent:Offline Explorer
    Disallow: /

    User-agent: MataHari
    Disallow: /

    User-agent:LexiBot
    Disallow: /

    User-agent: WebImage Collector
    Disallow: /

    User-agent: TheIntraformant
    Disallow: /

    User-agent:True_Robot/1.0
    Disallow: /

    User-agent:True_Robot
    Disallow: /

    User-agent:BlowFish/1.0
    Disallow: /

    User-agent:JennyBot
    Disallow: /

    User-agent: MIIxpc/4.2
    Disallow: /

    User-agent:BuiltBotTough
    Disallow: /

    User-agent:ProPowerBot/2.14
    Disallow: /

    User-agent:BackDoorBot/1.0
    Disallow: /

    User-agent:toCrawl/UrlDispatcher
    Disallow: /

    User-agent:WebEnhancer
    Disallow: /

    User-agent:suzuran
    Disallow: /

    User-agent:TightTwatBot
    Disallow: /

    User-agent: VCIWebViewer VCI WebViewer Win32
    Disallow: /

    User-agent: VCI
    Disallow: /

    User-agent:Szukacz/1.4
    Disallow: /

    User-agent:QueryN Metasearch
    Disallow: /

    User-agent:Openfind data gathere
    Disallow: /

    User-agent:Openfind
    Disallow: /

    User-agent:Xenu’s Link Sleuth 1.1c
    Disallow: /

    User-agent:Xenu’s
    Disallow: /

    User-agent: Zeus
    Disallow: /

    User-agent:RepoMonkey Bait & Tackle/v1.01
    Disallow: /

    User-agent:RepoMonkey
    Disallow: /

    User-agent:Microsoft URL Control
    Disallow: /

    User-agent:Openbot
    Disallow: /

    User-agent: URLControl
    Disallow: /

    User-agent: ZeusLink Scout
    Disallow: /

    User-agent: Zeus32297 Webster Pro V2.9 Win32
    Disallow: /

    User-agent:Webster Pro
    Disallow: /

    User-agent:EroCrawler
    Disallow: /

    User-agent:LinkScan/8.1a Unix
    Disallow: /

    User-agent:Keyword Density/0.9
    Disallow: /

    User-agent:Kenjin Spider
    Disallow: /

    User-agent:Iron33/1.0.2
    Disallow: /

    User-agent:Bookmark search tool
    Disallow: /

    User-agent:GetRight/4.2
    Disallow: /

    User-agent:FairAd Client
    Disallow: /

    User-agent:Gaisbot
    Disallow: /

    User-agent:Aqua_Products
    Disallow: /

    User-agent:Radiation Retriever 1.1
    Disallow: /

    User-agent:WebmasterWorld Extractor
    Disallow: /

    User-agent:Flaming AttackBot
    Disallow: /

    User-agent:Oracle Ultra Search
    Disallow: /

    User-agent:MSIECrawler
    Disallow: /

    User-agent:PerMan
    Disallow: /

    User-agent:searchpreview
    Disallow: /

    User-agent: *
    Disallow:
    Disallow: /bin/
    Sitemap:/sitemap.php


    帖子永久地址: 

    黑帽联盟 - 论坛版权1、本主题所有言论和图片纯属会员个人意见,与本论坛立场无关
    2、本站所有主题由该帖子作者发表,该帖子作者与黑帽联盟享有帖子相关版权
    3、其他单位或个人使用、转载或引用本文时必须同时征得该帖子作者和黑帽联盟的同意
    4、帖子作者须承担一切因本文发表而直接或间接导致的民事或刑事法律责任
    5、本帖部分内容转载自其它媒体,但并不代表本站赞同其观点和对其真实性负责
    6、如本帖侵犯到任何版权问题,请立即告知本站,本站将及时予与删除并致以最深的歉意
    7、黑帽联盟管理员和版主有权不事先通知发贴者而删除本文

    您需要登录后才可以回帖 登录 | 会员注册

    发布主题 !fastreply! 收藏帖子 返回列表 搜索
    回顶部