Skip to main content
#help #sysadmin #sysadmining #systemadministration
Hypolite Petovan
Hypolite Petovan  
•
@Shiri Bailem Thanks, I'm not ready to migrate to nginx, it would be an even bigger mountain to climb. I've enabled an Apache DDoS protection module but apparently it isn't working with the specific Apache worker module I use to attribute vhost to individual users to prevent lateral attacks. 😞
@Shiri Bailem
silverwizard
silverwizard  
•

@Hypolite Petovan You probably just need a rule for fail2ban - then use a couple of retries. I can, uh, probably make that more cogent if you need.

[INCLUDES]

before = common.conf

[Definition]

failregex = ^<HOST> .+HEAD \/ok.+ HTTP.+ [0-9]+

ignoreregex =
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@silverwizard Ah, thanks, I can wrap my head around this. the couple of retries is maxretry= ?
@silverwizard
silverwizard
silverwizard  
•

@Hypolite Petovan yeah, in this jail config. I use 2 for endpoints like this, once is happenstance, twice is enemy action imo.

So yeah, maxretry=2 is what I meant

@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@silverwizard Thanks, I have a new conundrum now. I need a glob expression to cover all *.log in /var/log/apache2 AND all its sub-folders. I tried /var/log/apache2/**/*.log doesn't pick up the files in /var/log/apache2 itself, I'm not sure what else to try.
@silverwizard
silverwizard
silverwizard  
•
@Hypolite Petovan Was the two stars an error in formatting? I would think /var/log/apache2/*/*.log would work - though I've not tried - I generally flatten my logs to small numbers of files and then pivot them with grep so I dunno.
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@silverwizard No, the double stars are supposed to match sub-folders recursively. I believe /var/log/apache2/*/*.log would only match log files in the first-level subfolders.
@silverwizard
silverwizard
silverwizard  
•

@Hypolite Petovan Oh, yeah, possibly. I think that might be special case globbing - and I don't know how fail2ban globs.

logpath
filename(s) of the log files to be monitored, separated by new lines.
Globs -- paths containing * and ? or [0-9] -- can be used however only the files that exist at start up matching this glob pattern will be considered.

Optional space separated option 'tail' can be added to the end of the path to cause the log file to be read from the end, else default 'head' option reads file from the be‐
ginning

Ensure syslog or the program that generates the log file isn't configured to compress repeated log messages to "*last message repeated 5 time*s" otherwise it will fail to
detect. This is called RepeatedMsgReduction in rsyslog and should be Off.


Based on jail.conf's manpage I think it might be a problem if you're rotating logs constantly in a glob.

Can you explain how your apache logs are configured? I just am not sure what patterns to use.

@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@silverwizard Sure, I have the various vhosts' logs in subfolders in /var/log/apache2, but there is a root access.log in that folder also for all the requests that go to the top-level domain, i.e. not what I call a vhost even if it's also one in the Apache configuration. Push come to shove, I can probably ignore it in favor of the subfolder ones.
@silverwizard
silverwizard
silverwizard  
•
@Hypolite Petovan Yeah - I'd recommend just shoving the TLD into a folder then as the easiest way to solve this. Even if it's a little kludgy. File globbing is a bit weird.
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•

@silverwizard I did it! I put on my big sysadmin pants, I straightened my Apache2 vhost configuration, neatly stored access logs in subfolders, then created my user-agent matching pattern in fail2ban and banned my first IPs for an hour.

I did it but man did it take a lot out of me. I'm reasonably satisfied though. Now I don't entirely remember which domains I took down to save the CPU. My sites-available folder is full of dead domains. 🙃

@silverwizard
silverwizard
silverwizard  
•

@Hypolite Petovan Oooof - sysadminning is the worst! You deserve a break.

But hey - glad it's working out! And in the future, if you need more help, feel free to ask me throw more effort at these things (you've put a lot of valuable volunteer time into something I value - and I'm happy to return the favour).

But also - hey! Nice. Hopefully it helps.

@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@silverwizard You've been a great help by providing stubs which sometimes is all I need. The initial complexity is like a solid wall and a stub is the first few rungs of a ladder I didn't realize existed.
@silverwizard
silverwizard
silverwizard  
•
@Hypolite Petovan Oh yeah! I figured it would help! And always glad to!
@Hypolite Petovan
Brad Koehn ☑️
Brad Koehn ☑️  
•
Firewall the IP?
Hypolite Petovan
Hypolite Petovan  
•
@Brad Koehn ☑️ I thought about that at first, and they all seem to come from 15.177.0.0/32, but even this feels daunting to me.
@Brad Koehn ☑️
Robert "Anaerin" Johnston
Robert "Anaerin" Johnston  
•

While it's not recommended, fail2ban has the "apache-badbots" config you can enable. You could then add a line/rule to block "Amazon-Route53-Health-Check-Service"

There's some details here: https://askubuntu.com/questions/1116001/block-badbot-with-fail2ban-via-user-agents-in-access-log


Block badbot with fail2ban via user agents in access.log

How can I create a filter to block these with fail2ban? 476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) 892 ltx71 - (http://ltx71.com/) 5367 Mozilla/5.0 (compat...
Ask Ubuntu
This entry was edited (2 months ago)
Hypolite Petovan
Hypolite Petovan  
•
@Robert "Anaerin" Johnston Thanks, looking into this filter shows it was last updated in 2013, not sure how relevant it still is.
# DEV Notes:
# List of bad bots fetched from http://www.user-agents.org
# Generated on Thu Nov  7 14:23:35 PST 2013 by files/gen_badbots.
@Robert "Anaerin" Johnston
Robert "Anaerin" Johnston
Robert "Anaerin" Johnston  
•
The file itself isn't, but the framework it uses still is.
silverwizard
silverwizard  
•
@Shiri Bailem @Hypolite Petovan honestly, if it's hitting a 404 hard enough to cause slowdown, it's probably resource exhaustion and you just need more forks. But also, you probably want the limit farther up the stack, since delivering a 404 is low resources.
@Hypolite Petovan @Shiri Bailem
Hypolite Petovan
Hypolite Petovan  
•
@silverwizard @Shiri Bailem Many websites I'm hosting are PHP CMSes with little to no caching because human traffic has never been that heavy, so machines come in with tens of requests by second and wreck the whole server.
@silverwizard @Shiri Bailem
Hypolite Petovan
Hypolite Petovan  
•
@Shiri Bailem I've thought about easing the transition between Apache and nginx by putting the latter in front of the former, but I ended up being deprived from the nginx experience by the separation of concerns at my current job where the DevOps team completely handles the hosting architecture. I'm just another tenant in that model, whereas I used to get my hands dirty with Apache before.
@Shiri Bailem
Brad Koehn ☑️
Brad Koehn ☑️  
•

If you have ufw installed, it’s just ufw deny from <ip or cidr> to any.

Or in Apache you can

<RequireAll>
    Require all granted
    Require not ip 1.2.3.4
</RequireAll>
Hypolite Petovan
Hypolite Petovan  
•
@Brad Koehn ☑️ Thanks, do you know if I have to copy this block in every vhost?
@Brad Koehn ☑️
Hypolite Petovan
Hypolite Petovan  
•

@Shiri Bailem Nginx doesn't have a built-in module like Apache has, so it has to use PHP-FPM. I have considered it, but I would like to replicate the system where each vhost is served by its own user to enforce separation of concerns.

I let a few people host stuff on my server, so if one of them gets hacked, I want to avoid lateral traversal and limit the damage to that user's home directory. I'm not sure how to do this with php-fpm pools without exposing the server to resource depletion since resources are allocated per pool and not globally.

Although I'm already fighting with resource depletion, so I'm not sure what I'm afraid of.

It's been difficult lately, I have had issue wrapping my head around situations I face at work and off work, and I feel utterly inadequate. I might take you up on your offer to chat, even if little is achieved technically!

@Shiri Bailem
Brad Koehn ☑️
Brad Koehn ☑️  
•
@Hypolite Petovan I'm not sure; I haven't used apache in ages. You may be able to put it in the main httpd.conf.
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•

@Shiri Bailem Containers would have been smart, including to have multiple versions of PHP running at the same time depending on when the websites were initially published. I spent way too much time trying to fix deprecation issues in archives websites.

Alas, that ship has sailed too.

@Shiri Bailem
⇧