Skip to main content

Oh look, the websites I host are under siege again. I don't have to look at the logs to figure out that it's AI scrapers again, and people wonder why I'm so gung-ho against the 5 biggest LLM models that are sucking the soul out of my body without me having to ever use them.

#AI #LLM

#AI #llm

reshared this

Hypolite Petovan
Hypolite Petovan  
•

That's yet another website I host that I had to take down because I don't want to have to do the necessary work to make it accessible to a stupid bot that requests non-existing pages 10-100 times per second.

That's two now that have gone dark because of this AI insanity. It absolutely didn't have to be this way.

Marco R.
Marco R.  
•
Did you try countermeasures like Anubis or this one?
https://github.com/bediger4000/infinite-fake-website
Hypolite Petovan
Hypolite Petovan  
•
@Marco R. Thank you for the suggestions, the latter one seems to be relying on recognizable User-Agent strings, which AI crawlers notoriously lie about. Additionally, I'm getting DDOSed as it is, I'd rather not serve additional content to bots because their hit frequency is simply too much for my server to handle.
@Marco R.
Hypolite Petovan
Hypolite Petovan  
•
@Marco R. Looks like I could enable Anubis even on my antiquated setup, I'll look into that.
@Marco R.
Hypolite Petovan
Hypolite Petovan  
•
@Marco R. Anubis is now live on the two main target sites I host!
@Marco R.
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan This is why I block them with cloudflare
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@TekNo ⚝ aEvl Good for you, but I don’t self-host to be dependent on a commercial service.
@TekNo ⚝ aEvl
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan I don't like opening ports, to the internet, does your domain name provider have any security features you can tunnel through?
I know tailscale has funnel, but they do not have anti bot securities. I was not trying to sound like an ass.
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•

@TekNo ⚝ aEvl Sorry I was snappy, CloudFlare’s de facto monopoly on reverse proxy is scary to me.

I didn’t get what you meant about opening ports and domain name provider security feature tunnel?

@TekNo ⚝ aEvl
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan the only other way i know to host is opening ports on my router to the Internet, but my experience showed me that this is the easiest way to be hacked. I know cloudflare can see the data flow and i avoid anything that could get me attacked at least for the most part. Domain registrar is what i meant. Cloudflared creates a secure tunnel from your device to the Internet with cloudflare as a firewall protecting the device from a long list of attack vectors. Tailscale does the tunnels also but with no security. I'm not sure if anything else provides this. If your domain registrar provides security between your server and the Internet then that could be a useful way to mitigate attacks.
@Hypolite Petovan
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan anubis is new to me, so I will be looking into this also.
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@TekNo ⚝ aEvl Ah, I seem to have misled you with “self-hosting”. I rent the hardware from a datacenter in France, so even if my server gets pwnd, it won’t get past through my home network.
@TekNo ⚝ aEvl
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan To me that's not self-hosting. Vlan's are great for self-hosting. And using tunnels make it better.
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•

@TekNo ⚝ aEvl The tricky part in self-hosting at home is making sure you have a permanent IP or a dynamic DNS record, having an ISP that allows inbound ports 80 and 443 traffic, and a custom router configuration to route these requests to the right machine on the LAN.

I don’t think opening a couple standard ports on your home router makes it inherently insecure, but correctly setting up your home network is work that I never bothered to do because I’m not a networking engineer and don’t plan on becoming one. And then when you mess up it can get dicey.

@TekNo ⚝ aEvl
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•

@Hypolite Petovan

having an ISP that allows inbound ports 80 and 443 traffic, and a custom router configuration to route these requests to the right machine on the LAN.


Yes that is the tough part, I know how to, and even used some ways to do it a few years ago, but the software that made it work became insecure with bad updates, and I noticed it, tried to find better options and realized it was just not the way I wanted to do it. Used a VPS for a while then the price hikes were almost the same cost I would have spent if i just used tunnels on my own hardware. Now I have learned full stack and networking from these things and I have spent a lot less money than what I would have otherwise.

@Hypolite Petovan
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan I'm deleting my other comments for security reasons.
@Hypolite Petovan
TekNo ⚝ aEvl
TekNo ⚝ aEvl  
•
@Hypolite Petovan Funny story, after having this conversation, my site urbanmind has been getting invisibly dossd hundreds of database requests and my machine has had to use a lot if power to break even on the message queue. Cant find the culprit of all this bs requests so this has been fun. I was winning yesterday. but today the queues going up and down, and not clearly clearing it out. This war has been on going for weeks, but today they amped up the numbers,
@Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
I've now set up Anubis in front of the two offending virtual hosts, and it has significantly reduced server load. Unfortunately I had to disable OpenGraph allowance because of course AI crawler bots were taking advantage of it, but both websites are archives and so thumbnails and summaries are less important for them.
⇧