Skip to main content

Search

Items tagged with: robotsTxt


#fediAdmin #fediTips #fediVerse

To begin with I wonder what happens if our sites and profiles display CC-BY-SA-NC as #copyright notice. Any use by #AI scrapers should become illegal and indemnisation inforcable.
Also if you search for #robotsTXT in google, this is what you get.

> Ignoring robots.txt instructions can result in your scraping activities being considered unethical or even illegal.

@maxschrems
@markus_netzpolitik
@ankedb

A screen shot from today of a google search for robots.txt.
The informational text for standard questions by google itself reads:
"Is it illegal to scrape a site if there is no robots.txt present?
Ignoring robots. txt instructions can result in your scraping activities being considered unethical or even illegal.3 nov 2023"


#meanWhile ..

.. the #mastodon community wastes it's time trying to pimp up the stars of it's #APP in #googlePlay, the #robotsTxt of it's instances disallows exactly one #AI bot scrapper to not search for all public data available about the #fediVerse. Not only on it's mother ship but on all instances, so the elonGated can create his target lists of "the enemy inside".

.. good job, well done! ..

> User-agent: GPTBot
> Disallow: /

https://mastodon.social/robots.txt

#fediAdmin