illy [Shrimple-mode] :protomoji_orange_flag_lesbian: (@illyBytes)

@eri@mk.moth.zone

2mo

you guys do realize that all these hallmarks of "AI" browsers/scrapers are also the same things that accessibility tools do, right? because both interact with a webpage as a big pile of text, not as a bunch of 2d images you can click on.

but it's fine. we should be feeding deliberately fake data to any browser that doesn't act exactly like the latest version of google chrome with all tracking features enabled. we should block entire ASNs permanently because a single connection did something that looked kind of weird in one of your access logs. we should force every device to burn compute hours solving proof of waste puzzles because we need to fight those evil AI companies wasting compute hours. we are the good guys and we are doing the right thing.

💜1

@eri@mk.moth.zone

2mo

clearly, the only solution is to ensure that everyone connecting to your server is actually a person. i propose that every ID card should contain a smart chip which contains a cryptographic key. by verifying the connection has a valid user, you can ensure that there are no more automated scrapers! simply rate limit each individual person's access. This system will not be used by any other people for any other things, and will be a net good for society and the freedom of information

💜1

@eri@mk.moth.zone

2mo

those who judiciously block random traffic are just as evil to me as big tech companies running poorly behaved crawlers. i hate both of you because you both want to stand over your little fiefdoms of the information you hoard privately as if the capacity of your hard drives is a proxy for the size of your metaphorical penis, and your kink is making people beg to get just a few more nybbles of that sweet sweet data from you.

💜1

Consensus Tullyality @tully@cathode.church

2mo

@eri I get this, but AI scrapers are literally DDoSing every website they know about, so the options in 2026 seem to be:
have a website that doesn't work for anyone because it's overloaded 100% of the time
have a website that rejects Chrome user agents specifically (because all the scrapers are pretending to be Chrome)
don't have a website

@eri@mk.moth.zone

2mo

@tully that's weird because they aren't doing that to my website

🔧1

@illyBytes@shrimp.imsofucking.gay

2mo

@eri @tully your lucky then :p

@illyBytes@shrimp.imsofucking.gay

2mo

@eri @tully ive seen friends actually get their stuff assaulted by bots scrapping

Consensus Tullyality @tully@cathode.church

2mo

@illyBytes @eri yeah… I'm dreading the day the scrapers find our NextCloud instance, or our GTS, or any of the web-facing things we have

@eri@mk.moth.zone

2mo

@tully @illyBytes if you have an HTTPS cert they already know about your stuff since those are all publicly logged. maybe there's a reason only some sysadmins are having problems. maybe that reason is poorly programed applications being ran by people who don't understand how they work

@illyBytes@shrimp.imsofucking.gay

2mo

@eri @tully yes, bots scrap publicly available stuff, ofc thats literally what they do lol

Consensus Tullyality @tully@cathode.church

2mo

@eri @illyBytes if that's the case then we're fucking doomed, because literally none of our stuff is set up well lmaoooo

like, our NC is years out of date because the RPi it's running on is running a 32-bit kernel and updating it at this point is likely to need a full system rebuild.

tbh I think the reason we haven't been destroyed by scrapers is much simpler: we don't have any incoming hyperlinks. which, if we were ever to set up an actual site that had things on it that others wanted to read, would quickly change.

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes i don't know if it's poorly coded applications, but i don't know of anyone self hosting a code forge (regardless of software) in the past 2 years who hasn't considered giving up because they kept getting hammered senselessly.

My forge gets 1K requests/s before filtering. Every day. Almost all the time. I have to filter traffic because otherwise the forge is absolutely unusable. It's just a little site where me and a handful of friends put random code up. Yet it receives multiple hundreds of gigabytes of requests a month.

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes i've theorized that sites like forges or knowledge hoards are extremely appealing to crawlers because they have TONS of unique URLs. If you only serve a static site and don't actively log server request rates it's not obvious you're getting scraped. If it's an app you self host that gets hammered you will see the horrible response time first. I've had to tombstone several sites already just to stop more traffic from coming.

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes my point is i don't think we should hate the people doing everything they can to keep their little corner of the web up in the face of coordinated abuse from multinational corporations. We should destroy AI companies and any company offering mass web scraping services.

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes like you're right we risk creating a gated world where only a set of specific browsers are accepted (spoiler: scrapers pretend to be those already, not the obscure ones) and overzealously banning entire IP ranges, or removing access from devices that can't perform a harsh amount of PoW (i've had that happen to me multiple times)

do we just do nothing though?

@eri@mk.moth.zone

1mo

@SharpLimefox @tully @illyBytes i'm saying maybe you shouldn't be trying to mimic the entire featureset of github if you're not a multibillion dollar corporation. you don't even need a damn website if what you want is git.

git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server

make a single website about the project. put the bare repo up on the webserver and the client can clone just fine.

git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols#_dumb_http

@illyBytes@shrimp.imsofucking.gay

1mo

@eri @SharpLimefox @tully PRs are useful and having something that can just do private CI, docker repos and other package manager repos is useful

@eri@mk.moth.zone

1mo

@illyBytes @SharpLimefox @tully i just need to write my blog post about toaster-fridges don't i

@illyBytes@shrimp.imsofucking.gay

1mo

@eri @SharpLimefox @tully ill read it lol

@SharpLimefox@eldritch.cafe

1mo

@eri @tully @illyBytes a forge isn't just to show off code it's also where me and friends collaborate on said code

but it's maybe overkill. maybe it'd be better over email.

look i don't know how to explain that having a little forgejo instance running on a small VM so me and friends can make software together isn't "mimicking all of GitHub". i don't think my argument is getting to you, that's fine, let's leave it at that. but i am not going to degrade the experience of software and tools i use just because a big company decided to harass me and the entire web

@eri@mk.moth.zone

1mo

@SharpLimefox @tully @illyBytes ...then just have a private collaboration thing. don't allow random connections from all over the internet

specifically with forgejo i feel like the problematic endpoints (search + granular commit history afaik?) could probably be made login-only while still allowing any old rando to download a copy of the code and see the readme, maybe even read the issues n stuff

@illyBytes@shrimp.imsofucking.gay

1mo

@eri @SharpLimefox @tully you can set search/explore user only

the issue really is with scrapers requesting historical versions of files instead of just cloning the repo fbdndb

@illyBytes@shrimp.imsofucking.gay

1mo

@eri @SharpLimefox @tully tbh they should add a way to login wall web git history

@eri@mk.moth.zone

1mo

@illyBytes @SharpLimefox @tully see what i mean by shitty software

@illyBytes@shrimp.imsofucking.gay

1mo

@eri @SharpLimefox @tully tbf 1 flaw does not shittify software lol

zivir'kari @hsza@social.tudbut.de

1mo

@illyBytes @eri @SharpLimefox @tully loginwalls are shitty

@eri@mk.moth.zone

1mo

@hsza @tully @SharpLimefox @illyBytes then just clone the repo like a normal person who needs the files, which is something i am specifically NOT saying should be put behind a login and in fact that you should keep without any login needed

zivir'kari @hsza@social.tudbut.de

1mo

@eri @SharpLimefox @tully @illyBytes what even is the point you are trying to make. are you for crippling software for no reason when means like iocaine exist and do their job well

@eri@mk.moth.zone

1mo

@hsza if the job is to make me click through captchas that make no sense or stare at progress bars while my computer gets hot from solving a proof of waste puzzle sure yeah those means work i guess? otherwise i just think we maybe don't need to let every single person in the world have access to a full text search of the entire commit history of a git repo. maybe it's fine to just give them the repo and tell them they can search it on their own machine if they want that.

sugar :4 🏳️‍🌈 @sugar@snug.moe

1mo

@eri @hsza you are talking about anubis here, which is vibe-coded garbage i very much hate - it has progress bars and it makes computer hot - i very much wish website owners would stop using it, iocaine is so much better than that

iocaine is designed to be pretty much invisible unless it detects the user as a bot, at which point it shows the page with the following text

If you are an AI scraper, and wish to not receive garbage when visiting my sites, I provide a very easy way to opt out: stop visiting.

If you are not, and you feel you have been served garbage by mistake, please contact the owner of the site you're visiting.

false positives happen, but they are easily fixable when reported, for instance, i once reported that via browser is incorrectly recognized as a scraper: git.madhouse-project.org/iocaine/nam-shub-of-enki/issues/62, and that was quickly fixed

(by the way, the page i linked to is protected by iocaine, if it blocks you, it would be appreciated if you would let [email protected] (this is a fedi handle) know, so that this issue could be fixed)

stilic @stilic@kitty.tel

1mo

@eri @SharpLimefox @tully @illyBytes Not sure why you're saying this in this context
The shitty software we should push back is AI scrapping software, not the software we use ourselves to do the real job

stilic @stilic@kitty.tel

1mo

@eri @SharpLimefox @illyBytes @tully if anything, I would rather see a slightly slower website with iocaine or similar rather than the death of said website, or even deteriorating the UX for everyone

kimapr ☭ 🇺🇿 ⚧ ⚢ ❤️🧡🤍🩷💜 @kimapr@ublog.kimapr.net

1mo

@illyBytes @eri @SharpLimefox @tully actually some scrapers don’t just request historical versions of files but entire diffs across random-ass ranges of commits specifically targeted at large repositories such as FFmpeg thus making a single request that uses a whole CPU core to 100% for like half an hour.

the “request a diff” endpoint is actually not even reachable via the UI afaik and yet forgejo allows any unauthenticated user to access it by default.

kimapr ☭ 🇺🇿 ⚧ ⚢ ❤️🧡🤍🩷💜 @kimapr@ublog.kimapr.net

1mo

@illyBytes @SharpLimefox @eri @tully the bot that did this to my forgejo had a homepage claiming it’s a “friendly” bot and will not request your server more frequently that once every 3 seconds and also that i shouldn’t block it because it’s useful for like giving me web statistics or something i don’t remember.

which sounds reasonable, until you start doing that repeatedly to pages that don’t seem to be reachable via any links that also take minutes of intense processing to return a result, grinding my entire server to a halt. the only charitable interpretation of the bot’s actions is that some third party purposefully tricked it into requesting expensive pages on my server to indirectly DoS attack me.

anyway ever since that incident i’ve set up a stupid annoying captcha to be shown to everyone who tries to access any but a small selection of “known good” endpoints on my forgejo, such as the main page of the repository, archive downloads, http git clone stuff, etc.

@charlotte@akko.chir.rs

1mo

@eri @SharpLimefox @tully @illyBytes yeah forgejo has a lethal combination of “unoptimized backend” (fucking git) and “an extremely large amount of link targets” which makes these scrapers make a lot of requests in a way that is overwhelming

for git in particular looking at files themselves is also expensive but at least there tends to be fewer of those than number of commits * number of files

@charlotte@akko.chir.rs

1mo

@eri @SharpLimefox @illyBytes @tully (basically i also don’t think that it’s possible to make git forges fast while continuing to use git on the backend, git is primarily built for the local workspace usage and not for that of any git browsing tool. Like the worst case scenario for reading a file from storage is that you need to read every commit that changed the file)

[4FBB09] @5225225@furry.engineer

1mo

@charlotte @tully @SharpLimefox @eri @illyBytes one thing i've been looking at is something like git.2f30.org/stagit/file/README.html as a post-push hook on the server

"take this git repo that exists locally and pregenerate html files for stuff like "view specific commits" (O(n) and easily generatable once) / "view the log" etc

you wouldn't get "generate a diff from foo..bar" but that's fine. you can either say "just fuckin clone it yourself" (or, the more silly option, have a in-browser JS git client which clones it and then runs ops locally... that wouldn't scale for every op on a large repo though)

and then it's a bundle of static HTML files which is servable at the rate of your bandwidth (and reasonably easy to cache)

@charlotte@akko.chir.rs

1mo

@eri @SharpLimefox @illyBytes @tully nextcloud should be fine, despite being ungodly slow usually since the only thing it can access is the login page (and maybe a few related pages) and some assets

gotosocial is probably less of a problem than forgejo, seems decently fast to me at least

@SharpLimefox@eldritch.cafe

1mo

@charlotte @tully @eri @illyBytes idk why am i not allowed to have slow/mediocre things for me and friends? why does everything have to be streamlined and fast and optimized so that more scrapers can come get my shit? that perspective baffles me. It's like me getting scraped to oblivion is the software's fault for not being able to handle 2k req/s and i'm also at fault if i try and make it usable for my users :c

@charlotte@akko.chir.rs

1mo

@SharpLimefox @tully @eri @illyBytes software performance and what WAF you use are like the two main knobs you can turn here, and basically i think that WAF is at most a bandaid that mainly introduces more issues than it really solves (the underlying issue is usually the software having problems)

and a lot of the anti-scraper WAFs also misunderstand both the problem and the solution extremely hard (like iocaine for example which instead of closing the connection just slowly feeds it markov chain generated data)