Skip to main content


@Hypolite Petovan on my server I use nginx and set rate limits by subnet.

It didn't take much to get these absurd scans off my back, and I was mostly getting hit by facebook for some wild reason.

@Shiri Bailem Thanks, I'm not ready to migrate to nginx, it would be an even bigger mountain to climb. I've enabled an Apache DDoS protection module but apparently it isn't working with the specific Apache worker module I use to attribute vhost to individual users to prevent lateral attacks. 😞

@Hypolite Petovan I mostly figured you might be able to translate the ideas, Apache has a rate limit module I believe that you can use.

I'll also note, worst comes to worst... you can put nginx [em]in front[/em] of Apache.

My setup since I run multiple servers out of my home is actually nginx in one VM in front of another nginx server so I can use the public IP for multiple things.

@Shiri Bailem @Hypolite Petovan honestly, if it's hitting a 404 hard enough to cause slowdown, it's probably resource exhaustion and you just need more forks. But also, you probably want the limit farther up the stack, since delivering a 404 is low resources.
@silverwizard @Shiri Bailem Many websites I'm hosting are PHP CMSes with little to no caching because human traffic has never been that heavy, so machines come in with tens of requests by second and wreck the whole server.
@Shiri Bailem I've thought about easing the transition between Apache and nginx by putting the latter in front of the former, but I ended up being deprived from the nginx experience by the separation of concerns at my current job where the DevOps team completely handles the hosting architecture. I'm just another tenant in that model, whereas I used to get my hands dirty with Apache before.

@Hypolite Petovan it's daunting to start, just because everything is configured in unfamiliar ways... but once you get going it's pretty straightforward.

I'm more than happy to chat sometime and run you through the basics.

How about PHP-FPM vs regular PHP? that makes an even bigger impact than just switching to nginx. (I'm not even sure nginx can use non-fpm php?)

@Shiri Bailem Nginx doesn't have a built-in module like Apache has, so it has to use PHP-FPM. I have considered it, but I would like to replicate the system where each vhost is served by its own user to enforce separation of concerns.

I let a few people host stuff on my server, so if one of them gets hacked, I want to avoid lateral traversal and limit the damage to that user's home directory. I'm not sure how to do this with php-fpm pools without exposing the server to resource depletion since resources are allocated per pool and not globally.

Although I'm already fighting with resource depletion, so I'm not sure what I'm afraid of.

It's been difficult lately, I have had issue wrapping my head around situations I face at work and off work, and I feel utterly inadequate. I might take you up on your offer to chat, even if little is achieved technically!

@Hypolite Petovan it definitely adds some complications to have multiple user accounts like that, but not necessarily huge ones.

The big advantage of PHP-FPM even under Apache is separating out the resource usage. You can more easily have a narrow limit on PHP processes while letting the lighter webserver process handle static content.

For that situation I think you can set up some rules in nginx that share across sites and limit the number of active PHP calls at one time. In that fashion I think it would be similar to Apache's limits.

I do think however that multi-tenant setups these days tend to just be nginx in front of a container for each tenant.

@Shiri Bailem Containers would have been smart, including to have multiple versions of PHP running at the same time depending on when the websites were initially published. I spent way too much time trying to fix deprecation issues in archives websites.

Alas, that ship has sailed too.

@Hypolite Petovan You probably just need a rule for fail2ban - then use a couple of retries. I can, uh, probably make that more cogent if you need.

[INCLUDES]

before = common.conf

[Definition]

failregex = ^<HOST> .+HEAD \/ok.+ HTTP.+ [0-9]+

ignoreregex =
@silverwizard Ah, thanks, I can wrap my head around this. the couple of retries is maxretry= ?

@Hypolite Petovan yeah, in this jail config. I use 2 for endpoints like this, once is happenstance, twice is enemy action imo.

So yeah, maxretry=2 is what I meant

@silverwizard Thanks, I have a new conundrum now. I need a glob expression to cover all *.log in /var/log/apache2 AND all its sub-folders. I tried /var/log/apache2/**/*.log doesn't pick up the files in /var/log/apache2 itself, I'm not sure what else to try.
@Hypolite Petovan Was the two stars an error in formatting? I would think /var/log/apache2/*/*.log would work - though I've not tried - I generally flatten my logs to small numbers of files and then pivot them with grep so I dunno.
@silverwizard No, the double stars are supposed to match sub-folders recursively. I believe /var/log/apache2/*/*.log would only match log files in the first-level subfolders.

@Hypolite Petovan Oh, yeah, possibly. I think that might be special case globbing - and I don't know how fail2ban globs.

logpath
filename(s) of the log files to be monitored, separated by new lines.
Globs -- paths containing * and ? or [0-9] -- can be used however only the files that exist at start up matching this glob pattern will be considered.

Optional space separated option 'tail' can be added to the end of the path to cause the log file to be read from the end, else default 'head' option reads file from the be‐
ginning

Ensure syslog or the program that generates the log file isn't configured to compress repeated log messages to "*last message repeated 5 time*s" otherwise it will fail to
detect. This is called RepeatedMsgReduction in rsyslog and should be Off.


Based on jail.conf's manpage I think it might be a problem if you're rotating logs constantly in a glob.

Can you explain how your apache logs are configured? I just am not sure what patterns to use.

@silverwizard Sure, I have the various vhosts' logs in subfolders in /var/log/apache2, but there is a root access.log in that folder also for all the requests that go to the top-level domain, i.e. not what I call a vhost even if it's also one in the Apache configuration. Push come to shove, I can probably ignore it in favor of the subfolder ones.
@Hypolite Petovan Yeah - I'd recommend just shoving the TLD into a folder then as the easiest way to solve this. Even if it's a little kludgy. File globbing is a bit weird.

@silverwizard I did it! I put on my big sysadmin pants, I straightened my Apache2 vhost configuration, neatly stored access logs in subfolders, then created my user-agent matching pattern in fail2ban and banned my first IPs for an hour.

I did it but man did it take a lot out of me. I'm reasonably satisfied though. Now I don't entirely remember which domains I took down to save the CPU. My sites-available folder is full of dead domains. πŸ™ƒ

@Hypolite Petovan Oooof - sysadminning is the worst! You deserve a break.

But hey - glad it's working out! And in the future, if you need more help, feel free to ask me throw more effort at these things (you've put a lot of valuable volunteer time into something I value - and I'm happy to return the favour).

But also - hey! Nice. Hopefully it helps.

@silverwizard You've been a great help by providing stubs which sometimes is all I need. The initial complexity is like a solid wall and a stub is the first few rungs of a ladder I didn't realize existed.
@Hypolite Petovan Oh yeah! I figured it would help! And always glad to!
@Brad Koehn β˜‘οΈ I thought about that at first, and they all seem to come from 15.177.0.0/32, but even this feels daunting to me.

While it's not recommended, fail2ban has the "apache-badbots" config you can enable. You could then add a line/rule to block "Amazon-Route53-Health-Check-Service"

There's some details here: https://askubuntu.com/questions/1116001/block-badbot-with-fail2ban-via-user-agents-in-access-log

This entry was edited (6 days ago)
@Robert "Anaerin" Johnston Thanks, looking into this filter shows it was last updated in 2013, not sure how relevant it still is.
# DEV Notes:
# List of bad bots fetched from http://www.user-agents.org
# Generated on Thu Nov  7 14:23:35 PST 2013 by files/gen_badbots.
The file itself isn't, but the framework it uses still is.

If you have ufw installed, it’s just ufw deny from <ip or cidr> to any.

Or in Apache you can

<RequireAll>
    Require all granted
    Require not ip 1.2.3.4
</RequireAll>
@Brad Koehn β˜‘οΈ Thanks, do you know if I have to copy this block in every vhost?
@Hypolite Petovan I'm not sure; I haven't used apache in ages. You may be able to put it in the main httpd.conf.
⇧