PDA

View Full Version : Barrage of repeated requests for large files on my server frombots... what to do?



richard k zad
05-06-08, 09:48 PM
Hello all, first quickly I'm sorry if this is the wrong place, but it
seemed the most appropriate usenet group.

My server is currently receiving a barrage of requests straight for
the largest files (videos) on my web server, which was causing the
server to become dreadfully slow to respond (at least a minute before
it hints a response) and sometimes not even at all.

This is the 3rd day it's been going on.

I've been keeping track of stats from the bots for the last 16
hours...
- Requests were from 845 different IP addresses
- Nearly all the IP addresses are from China
- 30,377 requests for video files (not the video pages)
- 40 distinct video files were requested
- One host made 13,000 requests within those 40 files.

I'm running a dedicated server.

I've had a few similar "attacks" (or algorithm malfunction?) in the
past, but it was only by 3 or 4 different IP addresses and so it was
easy enough to just block them using iptables.

The only way I'm able to block the current "attack" and get these
stats is that for some reason they all have the same referer:
http://www.173la.cn/u/egao/ -- which I don't know about you but it
doesn't even seem like a real website to me.

If the requests didn't have this referer, I would not have a way to
block these requests and my server would be beaten to a pulp. Right
now I'm redirecting any request with that referer to my logger script.



Any ideas of a better way to solve this once and for all? I'm
passively blocking them, but they are still trying to jam the same
requests down my server's throat. And I'm afraid that they'll get rid
of the referer if I try any sort of active block rather than my
passive method, and I'll be screwed.

The IP addresses also seem to be from several different hosts (judging
from the whois data), so it's not exactly easy to inform them all...


An example of one of the bots from the access_log:
211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/
439_poppy2606.wmv HTTP/1.1" 403 317 "http://www.173la.cn/u/egao/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )"
(Notice it's 403 because I'm blocking them myself)

Any tips at all?

Thanks in advance!!
- Richard

bz
05-07-08, 04:30 AM
richard k zad <richkzad@gmail.com> wrote in news:c9f6d507-a64d-463b-95a7-
6ccb7d13f778@w1g2000prd.googlegroups.com:

> An example of one of the bots from the access_log:
> 211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/
> 439_poppy2606.wmv HTTP/1.1" 403 317 "http://www.173la.cn/u/egao/"
> "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )"
> (Notice it's 403 because I'm blocking them myself)
>

according to whois -h whois.apnic.net
[quote]
Please send abuse e-mail to
remarks: chenxi@gd.chinamobile.com
remarks: Please send probe e-mail to
remarks: chenxi@gd.chinamobile.com
[unquote]

Of course you could bloock all china.

05/07/08 04:24:16 Fast traceroute 211.139.255.10
Trace 211.139.255.10 ...
.....
7 4.69.132.77 61ms 55ms 56ms TTL: 0 (ae-
3.ebr2.LosAngeles1.Level3.net ok)
8 4.69.137.22 55ms 53ms 52ms TTL: 0 (ae-72-
72.csw2.LosAngeles1.Level3.net ok)
9 4.68.20.69 51ms 57ms 52ms TTL: 0 (ae-23-
79.car3.LosAngeles1.Level3.net ok)
10 62.156.138.69 54ms 89ms 55ms TTL: 0 (No rDNS)
11 62.154.14.105 310ms 354ms 207ms TTL: 0 (lax-
sa1.LAX.US.net.DTAG.DE ok)
12 62.154.5.205 210ms 208ms 208ms TTL: 0 (hkg-
sa1.HKG.HK.net.DTAG.DE ok)
13 217.239.41.10 208ms 209ms 348ms TTL: 0 (No rDNS)
14 217.6.24.138 280ms 247ms 265ms TTL: 0 (No rDNS)
15 218.200.253.237 262ms 250ms 250ms TTL: 0 (No rDNS)
16 211.136.3.57 243ms 236ms 235ms TTL: 0 (No rDNS)
17 211.136.6.34 270ms 259ms 262ms TTL: 0 (No rDNS)
18 211.139.135.70 269ms 256ms 260ms TTL: 0 (No rDNS)
19 211.139.254.3 278ms 254ms 267ms TTL: 0 (No rDNS)
20 211.139.255.10 271ms 264ms * TTL:244 (No rDNS)

The lack of rDNS shows they are rather clueless.
You might complain to HKG.HK.net.DTAG.DE about the bots
It looks like HKG may be Hongcong and



--
bz

please pardon my infinite ignorance, the set-of-things-I-do-not-know is an
infinite set.

bz+csm@ch100-5.chem.lsu.edu remove ch100-5 to avoid spam trap

Leythos
05-07-08, 06:51 AM
In article <c9f6d507-a64d-463b-95a7-6ccb7d13f778
@w1g2000prd.googlegroups.com>, richkzad@gmail.com says...
> Any ideas of a better way to solve this once and for all? I'm
> passively blocking them, but they are still trying to jam the same
> requests down my server's throat. And I'm afraid that they'll get rid
> of the referer if I try any sort of active block rather than my
> passive method, and I'll be screwed.

Enter a subnet block for their providers IP range in your firewall so
that no address in that range can reach your server.

You do have a proper firewall, right?

The following list works for me, and I'm inside the USA and don't deal
with customers/users outside the USA, the entire 211.0.0.0/8 network is
blocked in my list below:

Updated 2008MAR01

12.144.182.0/24
12.45.203.0/24
12.98.139.0/24
121.0.0.0/8
124.0.0.0/8
125.172.237.0/24
125.213.42.0/24
134.159.0.0/16
134.160.0.0/16
140.109.0.0/16
140.110.0.0/15
140.112.0.0/12
140.128.0.0/13
140.136.0.0/15
140.138.0.0/16
155.48.106.0/24
162.40.0.0/16
168.126.0.0/16
172.184.111.203
190.3.209.0/24
193.248.60.0/24
193.251.0.0/16
193.252.0.0/16
193.253.0.0/16
194.170.0.0/16
195.174.0.0/16
195.175.16.0/20
195.229.0.0/23
195.58.124.0/24
200.181.0.0/16
200.244.0.0/16
200.30.203.0/24
201.0.0.0/8
201.130.192.0/18
201.230.0.0/16
201.240.0.0/16
202.40.148.0-202.40.149.255
202.84.128.0-202.84.255.255
202.88.186.0/24
203.150.101.0/24
203.152.22.0/24
203.162.0.0-203.162.255.255
203.210.128.0-203.210.255.255
205.251.79.0/24
210.0.0.0/8
211.0.0.0/8
212.150.124.0/24
212.162.8.0/24
212.18.57.0/24
212.202.178.0/24
212.27.32.0-212.27.63.255
212.64.0.0/16
212.9.7.0/24
213.13.26.0/24
213.192.0.0-213.192.255.255
216.184.97.0/24
216.76.35.0/24
217.118.224.0-217.118.239.255
217.160.110.0/24
218.164.28.0/24
218.234.0.0-218.239.255.255
218.252.74.0/24
218.67.128.0-218.76.255.255
219.115.214.0/24
219.212.4.0/24
219.56.0.0/24
219.97.93.0/24
220.0.0.0/8
222.0.0.0/8
41.221.19.0/24
60.0.0.0/8
61.135.148.0/24
61.175.239.0/24
61.181.0.0/16
61.218.19.0/24
61.33.206.0/24
61.48.18.0/24
62.154.0.0/17
62.240.161.0-62.240.161.127
64.230.125.0/24
66.250.125.0/24
66.250.32.0/24
66.28.35.131
66.57.133.0/24
71.184.44.154
78.48.8.16
80.0.0.0/8
81.0.0.0/8
82.0.0.0/8
83.0.0.0/8
85.17.255.0-85.255.255.255
87.0.0.0/8
88.0.0.0/8
89.0.0.0/8
91.76.56.0/24


--
- Igitur qui desiderat pacem, praeparet bellum.
- Calling an illegal alien an "undocumented worker" is like calling a
drug dealer an "unlicensed pharmacist"
spam999free@rrohio.com (remove 999 for proper email address)

Moe Trin
05-08-08, 02:59 PM
On Wed, 7 May 2008, in the Usenet newsgroup comp.security.firewalls, in article
<Xns9A972DD9C9FA3WQAHBGMXSZHVspammote@130.39.198.139>, bz wrote:

>according to whois -h whois.apnic.net

That's good, but most find that the abuse contacts are ignored.

>Of course you could bloock all china.

[compton ~]$ zgrep -c CN APNIC.gz
1430
[compton ~]$

Don't forget Hong Kong and Macau (.hk and .mo, which adds 648 more
networks) which are now special districts of China, and some people
consider Taiwan (.tw with 397 more networks) as part of China for a
total of 2475 address blocks, ranging from 200 /24s (255.255.255.0)
on up to 4 /10s (255.192.0.0) - but hey, that's only 174,089,728 IPv4
addresses in total.

[compton ~]$ zgrep -E (CN|HK|MO|TW) APNIC.gz | cut -d' ' -f2 | cut -d'.'
-f1 | sort -n | uniq -c | column
51 58 85 124 2 161 3 204
42 59 48 125 1 162 2 206
50 60 3 134 4 163 161 210
115 61 1 137 1 165 44 211
63 116 2 139 2 166 96 218
41 117 21 140 1 167 58 219
44 118 1 143 4 168 29 220
86 119 1 144 2 169 68 221
29 120 1 147 103 192 70 222
50 121 4 152 4 198
44 122 2 158 708 202
58 123 1 159 269 203
[compton ~]$

Translation: 51 address blocks in the 58.x.x.x range, 42 in the 59.x.x.x
range, 50 in the 60.x.x.x range, and so on. And before you use a /8
rule, you should know that there are other countries using those blocks,

[compton ~]$ zgrep -h ' 58\.' [ALR]* | cut -d' ' -f1 | sort | uniq -c |
column
1 AF 3 ID 2 NZ 4 TW
24 AU 3 IN 2 PH 1 VN
5 BD 30 JP 5 PK
43 CN 15 KR 6 SG
4 HK 4 MY 8 TH
[compton ~]$ ^58^59
zgrep -h ' 59\.' [ALR]* | cut -d' ' -f1 | sort | uniq -c | column
8 AU 4 HK 6 KR 4 TW
1 BD 7 IN 1 PK
34 CN 16 JP 1 SG
[compton ~]$

That's just two /8s. Do you know your ISO-3166 country codes?

>The lack of rDNS shows they are rather clueless.

and also makes it hard to use rDNS to identify Chinese blocks - but what
else is new? This happens in a lot of countries.

>05/07/08 04:24:16 Fast traceroute 211.139.255.10

And what IF ANYTHING does this have to do with the spam?

>11 62.154.14.105 310ms 354ms 207ms TTL: 0 (lax-
>sa1.LAX.US.net.DTAG.DE ok)
>12 62.154.5.205 210ms 208ms 208ms TTL: 0 (hkg-
>sa1.HKG.HK.net.DTAG.DE ok)

>You might complain to HKG.HK.net.DTAG.DE about the bots

Why - they happen to be an intermediate step on YOUR route to China,
but this is not as likely for the rest of us.

>It looks like HKG may be Hongcong and

HKG is probably Hong Kong - a former British enclave on the coast of
China around 22.2 degrees North, as LAX is Los Angeles. Your point?

Old guy

spam@gnostheos.org
06-23-08, 05:03 PM
On May 6, 10:48 pm, richard k zad <richk...@gmail.com> wrote:
> Hello all, first quickly I'm sorry if this is the wrong place, but it
> seemed the most appropriate usenet group.
>
> My server is currently receiving a barrage of requests straight for
> the largest files (videos) on my web server, which was causing the
> server to become dreadfully slow to respond (at least a minute before
> it hints a response) and sometimes not even at all.
>
> This is the 3rd day it's been going on.
>
> I've been keeping track of stats from the bots for the last 16
> hours...
> - Requests were from 845 different IP addresses
> - Nearly all the IP addresses are from China
> - 30,377 requests for video files (not the video pages)
> - 40 distinct video files were requested
> - One host made 13,000 requests within those 40 files.
>

Since the same IP requests the file a bunch of times, maybe
you could put a CGI interface infront of the file instead of
serving the file directly then log the IPs in a database with
a time stamp (or use the webserver logs or whatever) to
see if multiple requests have been made in a short period of
time. If the same IP is requesting it over and over again

while true
do
sleep 1
transmit byte
done

Transmit the file really slowly so that the connection
stays active, but does not use up your bandwidth.

You could also check the referer in the CGI and block
anything which does not have a referer page at least
from the same domain.

There is probably a way to do a better job with a
rate limiting firewall rule. Maybe spamd on OpenBSD
perhaps?

iptables has a rate limiting option

limit
This module matches at a limited rate using a token bucket
filter. A
rule using this extension will match until this limit is
reached
(unless the ‘!’ flag is used). It can be used in combination
with the
LOG target to give limited logging, for example.

--limit rate
Maximum average matching rate: specified as a number,
with an
optional ‘/second’, ‘/minute’, ‘/hour’, or ‘/day’
suffix; the
default is 3/hour.

--limit-burst number
Maximum initial number of packets to match: this
number gets
recharged by one every time the limit specified above
is not
reached, up to this number; the default is 5.

Maybe you could create a rule limiting the average rate and the burst
rate for these subnets. That way they can still connect but only
use a fraction of your bandwidth.

> I'm running a dedicated server.
>
> I've had a few similar "attacks" (or algorithm malfunction?) in the
> past, but it was only by 3 or 4 different IP addresses and so it was
> easy enough to just block them using iptables.
>
> The only way I'm able to block the current "attack" and get these
> stats is that for some reason they all have the same referer:http://www.173la.cn/u/egao/-- which I don't know about you but it
> doesn't even seem like a real website to me.
>
> If the requests didn't have this referer, I would not have a way to
> block these requests and my server would be beaten to a pulp. Right
> now I'm redirecting any request with that referer to my logger script.
>
> Any ideas of a better way to solve this once and for all? I'm
> passively blocking them, but they are still trying to jam the same
> requests down my server's throat. And I'm afraid that they'll get rid
> of the referer if I try any sort of active block rather than my
> passive method, and I'll be screwed.
>
> The IP addresses also seem to be from several different hosts (judging
> from the whois data), so it's not exactly easy to inform them all...
>
> An example of one of the bots from the access_log:
> 211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/
> 439_poppy2606.wmv HTTP/1.1" 403 317 "http://www.173la.cn/u/egao/"
> "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )"
> (Notice it's 403 because I'm blocking them myself)
>
> Any tips at all?
>
> Thanks in advance!!
> - Richard

----
http://www.1150riverviewdr.com/