设为首页收藏本站黑币充值VIP购买

黑帽联盟

 找回密码
 会员注册
黑帽联盟»论坛 › 黑帽学院 › 白帽seo › Apache与Nginx环境屏蔽YisouSpider爬虫的方法
返回列表 发新帖
查看: 1982|回复: 0
打印 上一主题 下一主题

[技术文章] Apache与Nginx环境屏蔽YisouSpider爬虫的方法

[复制链接]
yun 黑帽联盟官方人员 

920

主题

37

听众

1364

积分

超级版主

Rank: 8Rank: 8

  • TA的每日心情
    奋斗
    2019-10-18 11:20

    • 签到天数: 678 天

    [LV.9]以坛为家II
    跳转到指定楼层
    楼主
    |只看该作者 |倒序浏览
    早上一个朋友告知其没有流量的博客突然出现访问很慢，然后打不开提示信息主机资源超量等信息（具体的提示我也忘记复制下来），大概意思就是占用资源过大短期被屏蔽。因为是虚拟主机，采用的是APACHE架构服务，所以比较简单，直接帮他登录CPANEL面板取得日志文件，然后查看后吓一跳。
    16.png

    原来是来自YisouSpider蜘蛛的爬行导致占用内存和CPU过大导致的资源消耗，一个小小的博客没有流量的怎么可能会这样呢，我看他的博客只有几篇文章而已。
    解决方法很简单，我们直接在.htaccess文件中屏蔽这个IP就可以。
    1. Order Allow,Deny
    2. Allow from all
    3. Deny from 42.120.161.43
    复制代码
    在.htaccess文件中增加上面三行，如果有多个IP地址的话，用空格隔开。搜索了一下，发现出现这样问题的还真不少，而且还看到有人专门整理了所有的YisouSpider蜘蛛IP地址，我们可以全部屏蔽未雨绸缪，也可以等有问题的时候一个个添加。
    网友整理的所有YiSouSpider IP地址：
    1. Deny from 42.156.136.95
    2. Deny from 42.156.137.67
    3. Deny from 42.156.137.62
    4. Deny from 42.156.137.63
    5. Deny from 42.156.137.60
    6. Deny from 42.156.138.113
    7. Deny from 42.120.161.37
    8. Deny from 42.120.161.32
    9. Deny from 42.120.161.65
    10. Deny from 42.156.139.95
    11. Deny from 42.156.136.113
    12. Deny from 42.156.136.44
    13. Deny from 42.156.136.42
    14. Deny from 42.156.136.40
    15. Deny from 42.156.139.51
    16. Deny from 42.156.139.53
    17. Deny from 42.156.139.55
    18. Deny from 42.156.139.54
    19. Deny from 42.156.139.57
    20. Deny from 42.156.138.63
    21. Deny from 42.156.138.62
    22. Deny from 42.156.138.61
    23. Deny from 42.156.138.60
    24. Deny from 42.156.138.67
    25. Deny from 42.120.161.62
    26. Deny from 42.156.138.65
    27. Deny from 42.120.160.67
    28. Deny from 42.120.161.95
    29. Deny from 42.120.160.113
    30. Deny from 42.120.160.63
    31. Deny from 42.120.160.62
    32. Deny from 42.120.160.61
    33. Deny from 42.120.160.60
    34. Deny from 42.156.138.81
    35. Deny from 42.156.138.80
    36. Deny from 42.156.138.83
    37. Deny from 42.156.138.82
    38. Deny from 42.156.136.23
    39. Deny from 42.156.137.53
    40. Deny from 42.156.137.51
    41. Deny from 42.156.137.57
    42. Deny from 42.156.138.104
    43. Deny from 42.156.139.5
    44. Deny from 42.156.139.4
    45. Deny from 42.120.161.23
    46. Deny from 42.120.161.24
    47. Deny from 42.156.136.107
    48. Deny from 42.156.137.61
    49. Deny from 42.156.136.50
    50. Deny from 42.156.136.51
    51. Deny from 42.156.136.53
    52. Deny from 42.156.136.54
    53. Deny from 42.156.136.55
    54. Deny from 42.156.136.57
    55. Deny from 42.156.139.42
    56. Deny from 42.156.138.75
    57. Deny from 42.156.139.40
    58. Deny from 42.156.138.77
    59. Deny from 42.156.138.74
    60. Deny from 42.120.160.74
    61. Deny from 42.120.160.75
    62. Deny from 42.120.160.109
    63. Deny from 42.120.160.77
    64. Deny from 42.156.137.40
    65. Deny from 42.156.138.95
    66. Deny from 42.120.160.16
    67. Deny from 42.120.160.17
    68. Deny from 42.120.160.15
    69. Deny from 42.120.160.13
    70. Deny from 42.120.160.10
    71. Deny from 42.120.160.28
    72. Deny from 42.156.139.77
    73. Deny from 42.120.161.57
    74. Deny from 42.156.139.74
    75. Deny from 42.120.161.51
    76. Deny from 42.120.161.53
    77. Deny from 42.120.161.18
    78. Deny from 42.156.136.83
    79. Deny from 42.156.136.27
    80. Deny from 42.120.160.81
    81. Deny from 42.120.160.80
    82. Deny from 42.120.160.83
    83. Deny from 42.120.160.82
    84. Deny from 42.156.136.28
    85. Deny from 42.120.161.63
    86. Deny from 42.156.137.32
    87. Deny from 42.156.137.37
    88. Deny from 42.120.160.23
    89. Deny from 42.156.136.82
    90. Deny from 42.156.136.81
    91. Deny from 42.156.136.80
    92. Deny from 42.120.160.24
    93. Deny from 42.156.136.4
    94. Deny from 42.156.136.5
    95. Deny from 42.156.136.2
    96. Deny from 42.120.160.9
    97. Deny from 42.120.160.4
    98. Deny from 42.120.160.5
    99. Deny from 42.120.160.2
    100. Deny from 42.156.139.60
    101. Deny from 42.156.139.61
    102. Deny from 42.156.139.62
    103. Deny from 42.156.139.63
    104. Deny from 42.120.161.42
    105. Deny from 42.120.161.40
    106. Deny from 42.156.139.67
    107. Deny from 42.156.138.20
    108. Deny from 42.156.138.16
    109. Deny from 42.156.138.17
    110. Deny from 42.156.138.15
    111. Deny from 42.156.136.37
    112. Deny from 42.120.160.95
    113. Deny from 42.156.136.32
    114. Deny from 42.156.136.31
    115. Deny from 42.156.136.24
    116. Deny from 42.156.137.23
    117. Deny from 42.156.137.20
    118. Deny from 42.120.160.37
    119. Deny from 42.120.160.32
    120. Deny from 42.156.137.28
    121. Deny from 42.156.139.82
    122. Deny from 42.156.139.83
    123. Deny from 42.156.139.80
    124. Deny from 42.156.139.81
    125. Deny from 42.120.161.4
    126. Deny from 42.156.138.107
    127. Deny from 42.156.138.24
    128. Deny from 42.156.138.23
    129. Deny from 42.120.161.74
    130. Deny from 42.120.160.19
    131. Deny from 42.156.138.28
    132. Deny from 42.156.139.15
    133. Deny from 42.156.139.17
    134. Deny from 42.156.139.19
    135. Deny from 42.156.139.18
    136. Deny from 42.120.160.40
    137. Deny from 42.120.160.42
    138. Deny from 42.120.161.20
    139. Deny from 42.156.138.19
    140. Deny from 42.120.161.68
    141. Deny from 42.156.137.2
    142. Deny from 42.120.161.66
    143. Deny from 42.120.161.67
    144. Deny from 42.120.161.60
    145. Deny from 42.120.161.61
    146. Deny from 42.156.137.5
    147. Deny from 42.156.137.4
    148. Deny from 42.156.138.32
    149. Deny from 42.156.138.68
    150. Deny from 42.156.136.18
    151. Deny from 42.156.136.19
    152. Deny from 42.156.138.37
    153. Deny from 42.156.136.15
    154. Deny from 42.156.136.16
    155. Deny from 42.156.136.17
    156. Deny from 42.156.136.13
    157. Deny from 42.156.139.28
    158. Deny from 42.120.160.53
    159. Deny from 42.120.160.50
    160. Deny from 42.120.160.51
    161. Deny from 42.120.161.5
    162. Deny from 42.120.160.57
    163. Deny from 42.120.160.55
    164. Deny from 42.120.161.9
    165. Deny from 42.120.161.19
    166. Deny from 42.156.139.107
    167. Deny from 42.156.139.22
    168. Deny from 42.120.161.15
    169. Deny from 42.120.161.17
    170. Deny from 42.156.137.24
    171. Deny from 42.120.161.83
    172. Deny from 42.156.136.61
    173. Deny from 42.156.136.60
    174. Deny from 42.156.136.63
    175. Deny from 42.156.136.62
    176. Deny from 42.156.136.67
    177. Deny from 42.156.137.107
    178. Deny from 42.156.139.32
    179. Deny from 42.156.139.31
    180. Deny from 42.120.161.81
    181. Deny from 42.156.139.37
    182. Deny from 42.156.138.40
    183. Deny from 42.156.137.19
    184. Deny from 42.156.138.42
    185. Deny from 42.156.137.17
    186. Deny from 42.156.137.15
    187. Deny from 42.156.137.13
    188. Deny from 42.156.137.10
    189. Deny from 42.120.161.55
    190. Deny from 42.120.161.113
    191. Deny from 42.120.161.77
    192. Deny from 42.156.137.74
    193. Deny from 42.156.137.77
    194. Deny from 42.156.137.95
    195. Deny from 42.156.138.2
    196. Deny from 42.156.138.4
    197. Deny from 42.156.138.5
    198. Deny from 42.156.139.113
    199. Deny from 42.120.160.20
    200. Deny from 42.156.137.80
    201. Deny from 42.156.137.81
    202. Deny from 42.156.137.82
    203. Deny from 42.156.137.83
    204. Deny from 42.156.137.113
    205. Deny from 42.156.139.71
    206. Deny from 42.156.139.24
    207. Deny from 42.156.139.20
    208. Deny from 42.156.136.77
    209. Deny from 42.156.136.74
    210. Deny from 42.156.139.23
    211. Deny from 42.120.161.82
    212. Deny from 42.156.138.57
    213. Deny from 42.120.161.80
    214. Deny from 42.156.138.55
    215. Deny from 42.156.138.53
    216. Deny from 42.156.138.50
    217. Deny from 42.156.138.51
    复制代码
    PS：如果我们是NGINX环境系统，我们也可以到自己当前网站的.CONF文件中添加
    1. deny 42.120.161.43;
    复制代码
    这样我们一样可以屏蔽当前这个IP地址的访问。
    PS：这里在补充Nginx环境屏蔽YisouSpider爬虫的方法，我们可以直接将整个爬虫User Agent端口屏蔽，反正也没有多大的用途。
    1. if ($http_user_agent ~* "YisouSpider") {
    2. return 403;
    3. }
    复制代码
    在我们当前站点的.conf文件中添加上面的脚本，保存替换之后重启Nginx生效，这样我们就可以让所有的YisouSpider返回403。


    屏蔽垃圾爬虫, seo优化, YISOUSPIDER
    帖子永久地址： 

    黑帽联盟 - 论坛版权1、本主题所有言论和图片纯属会员个人意见，与本论坛立场无关
    2、本站所有主题由该帖子作者发表，该帖子作者与黑帽联盟享有帖子相关版权
    3、其他单位或个人使用、转载或引用本文时必须同时征得该帖子作者和黑帽联盟的同意
    4、帖子作者须承担一切因本文发表而直接或间接导致的民事或刑事法律责任
    5、本帖部分内容转载自其它媒体，但并不代表本站赞同其观点和对其真实性负责
    6、如本帖侵犯到任何版权问题，请立即告知本站，本站将及时予与删除并致以最深的歉意
    7、黑帽联盟管理员和版主有权不事先通知发贴者而删除本文

    相关帖子
    回复

    使用道具 举报

    返回列表 发新帖
    高级模式
    B Color Image Link Quote Code Smilies
    您需要登录后才可以回帖 登录 | 会员注册

    发布主题 !fastreply! 收藏帖子 返回列表 搜索

    Archiver手机版

    Copyright © 2013-2023 黑帽联盟 (https://bbs.cnblackhat.com/) 版权所有 All Rights Reserved.

    免责声明: 本网不承担任何由内容提供商提供的信息所引起的争议和法律责任。客服QQ: 1074189538

    黑帽联盟 苏ICP备16048252号-2

    回顶部