Rate limiting for some git-related requests

Rare and precious forum-wide announcements.
User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Rate limiting for some git-related requests

Postby fox » 13 Apr 2017, 15:27

I've noticed an unusual amount of git-related HTTP requests in nginx logs which cause a noticable increase in overall traffic sent per month (each request is only ~16K but with the rate of 1 per 3-4 seconds per IP it stacks up quite quickly - going by webalizer this specific URL endpoint caused 88% of monthly hits and 36% overall bytes sent in April, however this could be legit git traffic, at least partially):

Code: Select all

(ip redacted) - - [09/Apr/2017:08:46:00 +0300] "GET /gitlab/fox/tt-rss.git/info/refs?service=git-upload-pack HTTP/1.1" 200 13638 "-" "git/1.9.1"


Only three IP addresses originate the *vast* majority of the above traffic, one of those registered to Digital Ocean. Maybe they NAT all outbound traffic for their hosted sites under one address? Maybe some special person out there decided to git pull every second? Who knows, really.

Anyway, for the time being I'm implementing a rate limit if the following two conditions match: request URI contains "service=git-upload-pack" and user agent contains "git".

If you're screwed by this and can't git pull anymore because Digital Ocean or whoever *is* putting everyone behind one colossal NAT please post here so I can rework or remove this limit.

cpforbes
Bear Rating Trainee
Bear Rating Trainee
Posts: 1
Joined: 13 Apr 2017, 17:47

Re: Rate limiting for some git-related requests

Postby cpforbes » 13 Apr 2017, 18:18

I've got a server running at Digital Ocean and far as I know it there is no NAT involved.

My server better not be attempting to pull every second, but if (ip redacted) ends in 228 let me know.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Rate limiting for some git-related requests

Postby fox » 13 Apr 2017, 18:43

nah. nevertheless, since April 9:

Code: Select all

  41521 x.x.30.57
  49819 x.x.82.78
  90014 x.x.178.187
 104441 x.x.214.130
 121943 x.x.84.0
 122200 x.x.24.105


bottom to top first two IPs are Digital Ocean, the 3rd one is "LeaseWeb Netherlands", the 4th one is some hosting provider in France (online.net).

gg guys

JustAMacUser
Bear Rating Overlord
Bear Rating Overlord
Posts: 373
Joined: 20 Aug 2013, 23:13

Re: Rate limiting for some git-related requests

Postby JustAMacUser » 13 Apr 2017, 18:52

Sounds like an automated script gone bad. You're a lot nicer than I would be; I'd just block the IPs at the firewall and be done with it... Of course then you'd get a new forum member complaining that they can access Git locally but not from their VPS. :)

As mentioned, I don't think Digital Ocean does anything but VPSs with dedicated IPs.

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: Rate limiting for some git-related requests

Postby sleeper_service » 13 Apr 2017, 22:05

JustAMacUser wrote:Sounds like an automated script gone bad. You're a lot nicer than I would be; I'd just block the IPs at the firewall and be done with it... Of course then you'd get a new forum member complaining that they can access Git locally but not from their VPS. :)

and then you'd know who the troublemaker is :D

JustAMacUser
Bear Rating Overlord
Bear Rating Overlord
Posts: 373
Joined: 20 Aug 2013, 23:13

Re: Rate limiting for some git-related requests

Postby JustAMacUser » 13 Apr 2017, 22:30

sleeper_service wrote:and then you'd know who the troublemaker is :D


Win-win for everyone! :)

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Rate limiting for some git-related requests

Postby fox » 13 Apr 2017, 22:34

hehe

joeyteel
Bear Rating Trainee
Bear Rating Trainee
Posts: 7
Joined: 17 Aug 2014, 11:52

Re: Rate limiting for some git-related requests

Postby joeyteel » 14 Apr 2017, 07:44

Digital Ocean definitely doesn't do outbound NAT or half my sites wouldn't function correctly. I can also confirm that those aren't either of my instances.

If they're hitting you that often I'd report it personally. Digital Ocean takes abuse reports pretty seriously.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Rate limiting for some git-related requests

Postby fox » 14 Apr 2017, 08:15

whatever it was it stopped now so it's all cool *shrug emoji*

maybe the mysterious owner reads this forum, lol

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Rate limiting for some git-related requests

Postby fox » 15 Apr 2017, 23:18

Image


Return to “Announcements”

Who is online

Users browsing this forum: No registered users and 3 guests