书接上回，大量mysql数据库查询是不是蜘蛛造成的？

acg88

zsj1259 发表于 2024-8-14 13:15
是amazon的爬虫ip，你是不是在亚马逊上发了什么？你在哪些社交媒体发过？这个可能是审核爬虫。或者是社交ca ...

没有，这只能是总裁或者排名竞争对手搞的鬼。上次就给我移动线路搞假Q，现在移动线路没有Q了，所以是假的，真的Q不可能突然放出来。它可能利用被枪域名指向我的域名搞的，CF都没用，移动针对域名封锁

bilpam

是的，最近蜘蛛就是这样

acg88

NodeLoc 发表于 2024-8-14 13:16
针对游客可以只给缓存数据

超过我的处理能力

zsj1259

acg88 发表于 2024-8-14 13:24
超过我的处理能力

你如果不追求谷歌收录或者啥的，直接把谷歌的asn禁掉就好了。

acg88

172.70.43.174 - - [15/Aug/2024:01:14:09 +0800] "GET /bofang/161872-1-34.html HTTP/1.1" 499 0 "-" "meta-externalagent/1.1 (+https://developers.FB.com/docs/sharing/webmasters/crawler)"
172.70.38.56 - - [15/Aug/2024:01:14:09 +0800] "GET /sousuo/-----Takahashi--------.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.70.175.98 - - [15/Aug/2024:01:14:09 +0800] "GET /bofang/75799-8-448.html HTTP/1.1" 499 0 "-" "meta-externalagent/1.1 (+https://developers.FB.com/docs/sharing/webmasters/crawler)"
172.70.42.7 - - [15/Aug/2024:01:14:10 +0800] "GET /neirong/152263.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.69.67.206 - - [15/Aug/2024:01:14:10 +0800] "GET /neirong/75194.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.71.166.141 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/260846-3-4.html HTTP/1.1" 499 0 "-" "meta-externalagent/1.1 (+https://developers.FB.com/docs/sharing/webmasters/crawler)"
172.68.27.194 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/98393-13-32.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.68.245.99 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/31-4-69.html HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.71.147.121 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/98046-8-134.html HTTP/1.1" 499 0 "https://gqc4.top/" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
172.70.174.151 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/265769-2-52.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.70.135.141 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/139863-5-9.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.70.135.21 - - [15/Aug/2024:01:14:10 +0800] "GET /vodshow/duanju-%E5%BE%B7%E5%9B%BD-score---------1996.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.68.15.183 - - [15/Aug/2024:01:14:10 +0800] "GET /bofang/234948-1-4.html HTTP/1.1" 499 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"

大量的爬

acg88

已封掉IP段，太恶心了，全是meta , fb的，就这几个IP端

acg88

NB啊，添加那几个黑名单IP，网站都打不开，520了，删除掉黑名单IP后网站就能打开，这什么逻辑

原因 https://blog.csdn.net/qq_36746815/article/details/87194074/

acg88

只好改用UA黑名单了。虽然由一秒钟十几条爬虫信息变为几条，但还是IP段黑名单好用，不过为什么设置IP黑名单CF连接会错误呢

meta-externalagent        删除
FBexternalhit        删除
SemrushBot        删除
Amazonbot        删除
Googlebot        删除

acg88acg88

加上UA过滤
(Bytespider|toutiao|YodaoBot|msnbot|Googlebot|Amazonbot|dataforseo|DataForSeoBot|semrush|babbar|mj12bot|116.0.0.0)

状态码  444了  恶意User-Agent  186
172.70.34.160 - - [15/Aug/2024:02:09:14 +0800] "GET /bofang/168172-2-75.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.68.245.125 - - [15/Aug/2024:02:09:14 +0800] "GET /neirong/210549.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.70.38.138 - - [15/Aug/2024:02:09:14 +0800] "GET /bofang/244475-10-25.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
162.158.78.228 - - [15/Aug/2024:02:09:15 +0800] "GET /neirong/223243.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.68.92.138 - - [15/Aug/2024:02:09:15 +0800] "GET /neirong/118989.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.68.92.158 - - [15/Aug/2024:02:09:15 +0800] "GET /bofang/253542-2-10.html HTTP/1.1" 444 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.68.23.177 - - [15/Aug/2024:02:09:15 +0800] "GET /bofang/152780-11-175.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
162.158.41.253 - - [15/Aug/2024:02:09:16 +0800] "GET /bofang/21-13-1337.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected])"
172.68.92.157 - - [15/Aug/2024:02:09:16 +0800] "GET /bofang/30-11-64.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.68.23.34 - - [15/Aug/2024:02:09:17 +0800] "GET /bofang/7288-13-28.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
172.70.34.41 - - [15/Aug/2024:02:09:18 +0800] "GET /bofang/244475-10-24.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.70.39.42 - - [15/Aug/2024:02:09:18 +0800] "GET /bofang/168172-2-74.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.68.245.8 - - [15/Aug/2024:02:09:18 +0800] "GET /neirong/223216.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.68.92.149 - - [15/Aug/2024:02:09:18 +0800] "GET /neirong/106312.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.70.175.192 - - [15/Aug/2024:02:09:18 +0800] "GET /neirong/1074.html HTTP/1.1" 444 0 "-" "FBexternalhit/1.1 (+http://www.FB.com/externalhit_uatext.php)"
172.68.92.131 - - [15/Aug/2024:02:09:18 +0800] "GET /bofang/178294-3-1.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
172.70.38.51 - - [15/Aug/2024:02:09:19 +0800] "GET /neirong/210610.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
172.70.38.127 - - [15/Aug/2024:02:09:19 +0800] "GET /bofang/210273-4-3.html HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"