illy [Shrimple-mode] :protomoji_orange_flag_lesbian: (@illyBytes)

@eri@mk.moth.zone

2mo

you guys do realize that all these hallmarks of "AI" browsers/scrapers are also the same things that accessibility tools do, right? because both interact with a webpage as a big pile of text, not as a bunch of 2d images you can click on.

but it's fine. we should be feeding deliberately fake data to any browser that doesn't act exactly like the latest version of google chrome with all tracking features enabled. we should block entire ASNs permanently because a single connection did something that looked kind of weird in one of your access logs. we should force every device to burn compute hours solving proof of waste puzzles because we need to fight those evil AI companies wasting compute hours. we are the good guys and we are doing the right thing.

💜1

2

1

@eri@mk.moth.zone

2mo

clearly, the only solution is to ensure that everyone connecting to your server is actually a person. i propose that every ID card should contain a smart chip which contains a cryptographic key. by verifying the connection has a valid user, you can ensure that there are no more automated scrapers! simply rate limit each individual person's access. This system will not be used by any other people for any other things, and will be a net good for society and the freedom of information

💜1

2

0

1

@eri@mk.moth.zone

2mo

those who judiciously block random traffic are just as evil to me as big tech companies running poorly behaved crawlers. i hate both of you because you both want to stand over your little fiefdoms of the information you hoard privately as if the capacity of your hard drives is a proxy for the size of your metaphorical penis, and your kink is making people beg to get just a few more nybbles of that sweet sweet data from you.

💜1

4

1

0

1

Consensus Tullyality @tully@cathode.church

2mo

@eri I get this, but AI scrapers are literally DDoSing every website they know about, so the options in 2026 seem to be:
have a website that doesn't work for anyone because it's overloaded 100% of the time
have a website that rejects Chrome user agents specifically (because all the scrapers are pretending to be Chrome)
don't have a website

5

2

0

@eri@mk.moth.zone

2mo

@tully that's weird because they aren't doing that to my website

🔧1

2

0

1

@illyBytes@shrimp.imsofucking.gay

2mo

@eri @tully your lucky then :p

1

0

1

0

@illyBytes@shrimp.imsofucking.gay

2mo

@eri @tully ive seen friends actually get their stuff assaulted by bots scrapping

1

0

1

0

Consensus Tullyality @tully@cathode.church

2mo

@illyBytes @eri yeah… I'm dreading the day the scrapers find our NextCloud instance, or our GTS, or any of the web-facing things we have

1

0

1

0

@eri@mk.moth.zone

2mo

@tully @illyBytes if you have an HTTPS cert they already know about your stuff since those are all publicly logged. maybe there's a reason only some sysadmins are having problems. maybe that reason is poorly programed applications being ran by people who don't understand how they work

3

0

1

0

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes i don't know if it's poorly coded applications, but i don't know of anyone self hosting a code forge (regardless of software) in the past 2 years who hasn't considered giving up because they kept getting hammered senselessly.

My forge gets 1K requests/s before filtering. Every day. Almost all the time. I have to filter traffic because otherwise the forge is absolutely unusable. It's just a little site where me and a handful of friends put random code up. Yet it receives multiple hundreds of gigabytes of requests a month.

1

0

1

0

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes i've theorized that sites like forges or knowledge hoards are extremely appealing to crawlers because they have TONS of unique URLs. If you only serve a static site and don't actively log server request rates it's not obvious you're getting scraped. If it's an app you self host that gets hammered you will see the horrible response time first. I've had to tombstone several sites already just to stop more traffic from coming.

1

0

1

0

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes my point is i don't think we should hate the people doing everything they can to keep their little corner of the web up in the face of coordinated abuse from multinational corporations. We should destroy AI companies and any company offering mass web scraping services.

2

1

2

0

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes like you're right we risk creating a gated world where only a set of specific browsers are accepted (spoiler: scrapers pretend to be those already, not the obscure ones) and overzealously banning entire IP ranges, or removing access from devices that can't perform a harsh amount of PoW (i've had that happen to me multiple times)

do we just do nothing though?

1

0

1

0

@eri@mk.moth.zone

1mo

@SharpLimefox @tully @illyBytes i'm saying maybe you shouldn't be trying to mimic the entire featureset of github if you're not a multibillion dollar corporation. you don't even need a damn website if what you want is git.

git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server

make a single website about the project. put the bare repo up on the webserver and the client can clone just fine.

git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols#_dumb_http

2

0

@illyBytes@shrimp.imsofucking.gay

1mo

@eri @SharpLimefox @tully PRs are useful and having something that can just do private CI, docker repos and other package manager repos is useful

1

0

@eri@mk.moth.zone

1mo

@illyBytes @SharpLimefox @tully i just need to write my blog post about toaster-fridges don't i

1

0

1

0