Hypolite Petovan

1 month ago •

Hypolite Petovan
1 month ago •

Once again my personal web server is on its knees, this time thanks to Amazon who's probing a non-existent Health Check endpoint with a rare intensity. In the Apache access log, it looks like this:

<domain>:80 15.177.10.187 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.26.71 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.42.155 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.30.95 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.50.106 -

Once again my personal web server is on its knees, this time thanks to Amazon who's probing a non-existent Health Check endpoint with a rare intensity. In the Apache access log, it looks like this:

<domain>:80 15.177.10.187 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.26.71 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.42.155 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.30.95 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"
<domain>:80 15.177.50.106 - - [25/Sep/2025:22:04:50 +0000] "GET /ok HTTP/1.1" 404 457 "-" "Amazon-Route53-Health-Check-Service (ref 0c1421fb-b0fe-4dbd-af57-dc05457a9d2e; report http://amzn.to/1vsZADi)"

I followed the link provided and submitted a report, but I'm guessing they are only handled during US business hours. To be able to access my other web sites, I took offline the target vhost, and Amazon immediately switched to another one. I filed a report, took it offline, etc... Four vhosts down the line, the web form prevented me from submitting yet another report because of a rate limiting feature.

I am seething with rage and I want AWS IPs off my web server, but I'm off my depth in system administration. I see I could do it with iptables (by compiling a list of IP blocks from the JSON provided by Amazon) and I was hoping fail2ban would have a ready-made jail, but it seems to be meant for repeated authentication errors, not for crawler errors.

Neither of these solutions feel workable, would anyone have an easier method to cut AWS off my box?

Thanks a bunch!

#SystemAdministration #SysAdmin #SysAdmining #Help

Shiri Bailem

1 month ago •

@Hypolite Petovan on my server I use nginx and set rate limits by subnet.

It didn't take much to get these absurd scans off my back, and I was mostly getting hit by facebook for some wild reason.

@Hypolite Petovan

Shiri Bailem

1 month ago •

@Hypolite Petovan Been over a year since I set it up, I had forgotten some of the nonsense I put together in there.

I have a redirect that checked if facebook was in the user agent, and if so I forced a delay before it even responded (since it's nginx + php-fpm a request only really takes resources if it's hitting php... so this only took resources if they were still there after the wait lol).

    location / {
        if ($http_user_agent ~* "^facebook") { 
            return 302 https://foggyminds.com/forcedelay$request_uri;
        }

        limit_req zone=global_limit burst=30 delay=2;
        limit_req_status 429;

    location /forcedelay {
        echo_sleep 5;
        rewrite ^/forcedelay(.*)$ $1 break;
        echo_exec @proxy;
    }

And I had two different rate limits set, so posts were limited more than gets, plus a remap to do rate limits off of sub... show more

@Hypolite Petovan Been over a year since I set it up, I had forgotten some of the nonsense I put together in there.

    location / {
        if ($http_user_agent ~* "^facebook") { 
            return 302 https://foggyminds.com/forcedelay$request_uri;
        }

        limit_req zone=global_limit burst=30 delay=2;
        limit_req_status 429;

    location /forcedelay {
        echo_sleep 5;
        rewrite ^/forcedelay(.*)$ $1 break;
        echo_exec @proxy;
    }

And I had two different rate limits set, so posts were limited more than gets, plus a remap to do rate limits off of subnet as opposed to individual IP (because so often they were requests cycling through addresses on a single subnet).:

map $remote_addr $remote_subnet {
        default $remote_addr;
        "~^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$" $1;
        "~^(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$" $1;
}

limit_req_zone 1 zone=post_zone:10m rate=10r/s;
limit_req_zone $remote_subnet zone=global_limit:32m rate=20r/s;

    location /inbox {
        limit_req zone=post_zone;

(Also see the location / under the forcedelay)

@Hypolite Petovan

like this

Hypolite Petovan

1 month ago •

@Shiri Bailem Thanks, I'm not ready to migrate to nginx, it would be an even bigger mountain to climb. I've enabled an Apache DDoS protection module but apparently it isn't working with the specific Apache worker module I use to attribute vhost to individual users to prevent lateral attacks. 😞

@Shiri Bailem

Shiri Bailem

1 month ago •

@Hypolite Petovan I mostly figured you might be able to translate the ideas, Apache has a rate limit module I believe that you can use.

I'll also note, worst comes to worst... you can put nginx [em]in front[/em] of Apache.

My setup since I run multiple servers out of my home is actually nginx in one VM in front of another nginx server so I can use the public IP for multiple things.

@Hypolite Petovan

Hypolite Petovan likes this.

silverwizard

1 month ago from ZoobopDeDoDop! •

@Shiri Bailem @Hypolite Petovan honestly, if it's hitting a 404 hard enough to cause slowdown, it's probably resource exhaustion and you just need more forks. But also, you probably want the limit farther up the stack, since delivering a 404 is low resources.

@Hypolite Petovan @Shiri Bailem

Hypolite Petovan

1 month ago •

@silverwizard @Shiri Bailem Many websites I'm hosting are PHP CMSes with little to no caching because human traffic has never been that heavy, so machines come in with tens of requests by second and wreck the whole server.

@silverwizard @Shiri Bailem

Shiri Bailem likes this.

Hypolite Petovan

1 month ago •

@Shiri Bailem I've thought about easing the transition between Apache and nginx by putting the latter in front of the former, but I ended up being deprived from the nginx experience by the separation of concerns at my current job where the DevOps team completely handles the hosting architecture. I'm just another tenant in that model, whereas I used to get my hands dirty with Apache before.

@Shiri Bailem

Shiri Bailem likes this.

Shiri Bailem

1 month ago •

@Hypolite Petovan it's daunting to start, just because everything is configured in unfamiliar ways... but once you get going it's pretty straightforward.

I'm more than happy to chat sometime and run you through the basics.

How about PHP-FPM vs regular PHP? that makes an even bigger impact than just switching to nginx. (I'm not even sure nginx can use non-fpm php?)

@Hypolite Petovan

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@Shiri Bailem Nginx doesn't have a built-in module like Apache has, so it has to use PHP-FPM. I have considered it, but I would like to replicate the system where each vhost is served by its own user to enforce separation of concerns.

I let a few people host stuff on my server, so if one of them gets hacked, I want to avoid lateral traversal and limit the damage to that user's home directory. I'm not sure how to do this with php-fpm pools without exposing the server to resource depletion since resources are allocated per pool and not globally.

Although I'm already fighting with resource depletion, so I'm not sure what I'm afraid of.

It's been difficult lately, I have had issue wrapping my head around situations I face at work and off work, and I feel utterly inadequate. I might take you up on your offer to chat, even if little is achieved technically!

@Shiri Bailem

Shiri Bailem

1 month ago •

@Hypolite Petovan it definitely adds some complications to have multiple user accounts like that, but not necessarily huge ones.

The big advantage of PHP-FPM even under Apache is separating out the resource usage. You can more easily have a narrow limit on PHP processes while letting the lighter webserver process handle static content.

For that situation I think you can set up some rules in nginx that share across sites and limit the number of active PHP calls at one time. In that fashion I think it would be similar to Apache's limits.

I do think however that multi-tenant setups these days tend to just be nginx in front of a container for each tenant.

@Hypolite Petovan

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@Shiri Bailem Containers would have been smart, including to have multiple versions of PHP running at the same time depending on when the websites were initially published. I spent way too much time trying to fix deprecation issues in archives websites.

Alas, that ship has sailed too.

@Shiri Bailem

Shiri Bailem likes this.

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan You probably just need a rule for fail2ban - then use a couple of retries. I can, uh, probably make that more cogent if you need.

[INCLUDES]

before = common.conf

[Definition]

failregex = ^<HOST> .+HEAD \/ok.+ HTTP.+ [0-9]+

ignoreregex =

@Hypolite Petovan

like this

Hypolite Petovan

1 month ago •

@silverwizard Ah, thanks, I can wrap my head around this. the couple of retries is maxretry= ?

@silverwizard

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan yeah, in this jail config. I use 2 for endpoints like this, once is happenstance, twice is enemy action imo.

So yeah, maxretry=2 is what I meant

@Hypolite Petovan

Hypolite Petovan

1 month ago •

@silverwizard Thanks, I have a new conundrum now. I need a glob expression to cover all *.log in /var/log/apache2 AND all its sub-folders. I tried /var/log/apache2/**/*.log doesn't pick up the files in /var/log/apache2 itself, I'm not sure what else to try.

@silverwizard

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan Was the two stars an error in formatting? I would think /var/log/apache2/*/*.log would work - though I've not tried - I generally flatten my logs to small numbers of files and then pivot them with grep so I dunno.

@Hypolite Petovan

Hypolite Petovan

1 month ago •

@silverwizard No, the double stars are supposed to match sub-folders recursively. I believe /var/log/apache2/*/*.log would only match log files in the first-level subfolders.

@silverwizard

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan Oh, yeah, possibly. I think that might be special case globbing - and I don't know how fail2ban globs.

logpath
filename(s) of the log files to be monitored, separated by new lines.
Globs -- paths containing * and ? or [0-9] -- can be used however only the files that exist at start up matching this glob pattern will be considered.
Optional space separated option 'tail' can be added to the end of the path to cause the log file to be read from the end, else default 'head' option reads file from the be‐
ginning
Ensure syslog or the program that generates the log file isn't configured to compress repeated log messages to "*last message repeated 5 time*s" otherwise it will fail to
detect. This is called RepeatedMsgReduction in rsyslog and should be Off.

Based on jail.conf's manpage I think it might be a problem if you're rotating logs constantly in a glob.

Can you explain how your apache logs are configured? I just am not sure what patterns to use.

@Hypolite Petovan

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@silverwizard Sure, I have the various vhosts' logs in subfolders in /var/log/apache2, but there is a root access.log in that folder also for all the requests that go to the top-level domain, i.e. not what I call a vhost even if it's also one in the Apache configuration. Push come to shove, I can probably ignore it in favor of the subfolder ones.

@silverwizard

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan Yeah - I'd recommend just shoving the TLD into a folder then as the easiest way to solve this. Even if it's a little kludgy. File globbing is a bit weird.

@Hypolite Petovan

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@silverwizard I did it! I put on my big sysadmin pants, I straightened my Apache2 vhost configuration, neatly stored access logs in subfolders, then created my user-agent matching pattern in fail2ban and banned my first IPs for an hour.

I did it but man did it take a lot out of me. I'm reasonably satisfied though. Now I don't entirely remember which domains I took down to save the CPU. My sites-available folder is full of dead domains. 🙃

@silverwizard

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan Oooof - sysadminning is the worst! You deserve a break.

But hey - glad it's working out! And in the future, if you need more help, feel free to ask me throw more effort at these things (you've put a lot of valuable volunteer time into something I value - and I'm happy to return the favour).

But also - hey! Nice. Hopefully it helps.

@Hypolite Petovan

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@silverwizard You've been a great help by providing stubs which sometimes is all I need. The initial complexity is like a solid wall and a stub is the first few rungs of a ladder I didn't realize existed.

@silverwizard

silverwizard likes this.

silverwizard

1 month ago from ZoobopDeDoDop! •

@Hypolite Petovan Oh yeah! I figured it would help! And always glad to!

@Hypolite Petovan

Hypolite Petovan likes this.

Brad Koehn ☑️

1 month ago •

Firewall the IP?

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@Brad Koehn ☑️ I thought about that at first, and they all seem to come from 15.177.0.0/32, but even this feels daunting to me.

@Brad Koehn ☑️

Robert "Anaerin" Johnston

1 month ago •

While it's not recommended, fail2ban has the "apache-badbots" config you can enable. You could then add a line/rule to block "Amazon-Route53-Health-Check-Service"

There's some details here: https://askubuntu.com/questions/1116001/block-badbot-with-fail2ban-via-user-agents-in-access-log

Block badbot with fail2ban via user agents in access.log

How can I create a filter to block these with fail2ban? 476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) 892 ltx71 - (http://ltx71.com/) 5367 Mozilla/5.0 (compat...

^{Ask Ubuntu}

This entry was edited (1 month ago)

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@Robert "Anaerin" Johnston Thanks, looking into this filter shows it was last updated in 2013, not sure how relevant it still is.

# DEV Notes:
# List of bad bots fetched from http://www.user-agents.org
# Generated on Thu Nov  7 14:23:35 PST 2013 by files/gen_badbots.

@Robert "Anaerin" Johnston

Robert "Anaerin" Johnston

1 month ago •

The file itself isn't, but the framework it uses still is.

Hypolite Petovan likes this.

Brad Koehn ☑️

1 month ago •

If you have ufw installed, it’s just ufw deny from <ip or cidr> to any.

Or in Apache you can

<RequireAll>
    Require all granted
    Require not ip 1.2.3.4
</RequireAll>

Hypolite Petovan likes this.

Hypolite Petovan

1 month ago •

@Brad Koehn ☑️ Thanks, do you know if I have to copy this block in every vhost?

@Brad Koehn ☑️

Brad Koehn ☑️

1 month ago •

@Hypolite Petovan I'm not sure; I haven't used apache in ages. You may be able to put it in the main httpd.conf.

@Hypolite Petovan

Hypolite Petovan likes this.

⇧