"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
Which is precisely why trying to legislate them is folly. Obviously the reason Android calls itself "Mobile Safari" is so that servers send it the mobile version of the site instead of the desktop one etc.
Which makes "accurate" an unreasonable requirement, since there is already a preexisting non-waivable requirement for inaccurate. When someone is making their "turn a web page into a PDF so you can view it later on the plane" software or whatever, it has to use a user agent string that will cause the site to send the version you need turned into the PDF.
> That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
First, you're assuming that the person being prosecuted is the entities who are the nominal targets of the bill, when the real problem is when they're not. Because you're proposing a law that says anybody with an inaccurate user agent string is in violation, meanwhile everybody has an inaccurate user agent string.
Moreover, consider what you expect the outcome of legislating something like this is going to be. Sites could then try to either blacklist or whitelist user agents.
If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
If they do the second one, that doesn't work, because now every third party has to start pretending to be a common browser even when they're doing something totally innocent or they get blocked by default for being something else. Which makes the user agent string even less useful than it is now, and subjects innocent parties to arbitrary liability for doing something they have to do or their client is broken.
> As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
You're describing autocracy. The way you make it so that you always prosecute someone is to make it so that you can always prosecute anyone, which is very bad.
> you're proposing a law that says anybody with an inaccurate user agent string is in violation
No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.
> Sites could then try to either blacklist or whitelist user agents.
Yes, that is a desired outcome.
> If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
That would put them in violation, and make them prosecutable. Of course you need to start collecting data to prove that it's automatic, but this is already happening today for the purpose of temporarily blocking an IP. Having this law means that you also have some punitive measures at the end of the rainbow.
> If they do the second one ...
Anyone pretending to be a common browser when they're not is in violation. In addition, under this law there's no incentive for anyone to pretend to be googlebot since _that_ can get them blocked. You are intentionally and repeatedly trying to obfuscate these facts.
This model is the exact wrong direction for a healthy society. It will just make it harder to use information that's already available to the public, on a public website, and thus just create moats around training data.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
I'm not sure all companies will have deniability. It's pretty easy to see companies which are trying to evade detection or misusing their existing spiders for AI training purposes.
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
> A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
That doesn't mean anything. All of the code in a browser is "automation" -- normal users aren't hand-crafting the hundreds of HTTP requests the browser makes to render a single web page. Meanwhile there is a human somewhere telling the scrapers what to scrape.
> You are entitled to your own emails, posts, etc
Is that somewhere in the proposed text?
> but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
"Stating what they are" buys you nothing when you have no way of verifying it, and you're still not providing any way to distinguish this from investigative journalism or e.g. a new competing search engine.
Using a computer to do "by hand" is a vague concept, yes, but laws routinely embrace vague concepts. You used it yourself! "Hand-crafting", you said. It could be pinned down more closely to something like a single action that happens quite soon after you press a button or ... and further wrangling could ensue because it's never going to have all the vagueness removed. It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
> It could be pinned down more closely to something like a single action that happens quite soon after you press a button
So leaving your webmail open is banned because it will still be checking for new messages three months after you opened the tab without any further human interaction. Likewise RSS readers or really any client which is intended to fetch new posts/messages when they appear.
> It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
Which is kind of exactly the problem. A new law would presumably still be on the books in ten or twenty years. Do we expect user agents by then to do only the things they do now, or will they get more capable? Banning them from becoming more capable is the problem.
I am curious about your view on accessibility tooling. Often screen readers or speech to text tools would effectively be external pieces of software instrumenting for instance browsers through their APIs. And surely using local AI going forward could enhance the tooling while also making the users less dependent on a small market charging exorbitant prices for poor user experience. What viable paths do you see for users perfecting user agents for themselves when the big browsers further lose their plot and focus on what's fun to maintain or profitable to ship instead of optimising for every user individually?
It's just an unfortunate reality that people with disabilities live with.
Some of them are already prevented from operating heavy machinery because doing so would put everyone at risk, so there is precedent here. I'm in favor of making concessions to the very unfortunate, though at some point you have to look at percentages, expected impact and decide what is reasonable and what isn't.
To your other question, in the last week there have been entire CPUs embedded within weights. Clearly it's possible to publish "open models" with an agenda so then what assurances you have that the enhancements provided by local ai would be more focused on the plot than what the big browsers do?
Specifically how would a person with disabilities be able to tell the difference between improved instrumentation and conditional access to information, as we have in other media? Consider this in the context of "compared to superintelligence, we all have an intelectual disability"
A browser is not heavy machinery. That is not precedent that applies to this situation.
I had a disabled relative (who passed this year), and am quite amazed and offended to hear the viewpoint that those in their situation must suffer an "unfortunate reality" for the sake of everyone else. That's taking the "non-disabled people must accommodate them" (which has problems and is worth a lengthy discussion elsewhere) pendulum and swinging it like a wrecking ball.
If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human? What about 5 seconds? Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction? Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling? Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
What changes if I write my own browser? What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences? What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read? What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want? What changes if it could be recalled but I never decide to do so? What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else? What changes if I download this program from someone else and use it other than how they intended?
Which of these situations is "crawling" and which aren't?
You may believe you have clear definitions, but you don't.
> If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human?
No, but nobody cares if you reload one page every 5 minutes
> What about 5 seconds?
Definitely no, people will start caring after you do this for an hour or so.
> Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction?
No, if you don't press a button to make it happen every time, it's automated. Also, nobody cares, you are not the intended target here.
> Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling?
No, but being entertained at others' expense suggests some emotional issues you may want to check on.
> Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
Yes. Scalpers are bad for everyone but themselves.
> What changes if I write my own browser?
Nothing.
> What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences?
It's still automated, thus a bot. There might be a fair use case if you could've refreshed the pages manually, but what you are describing is RSS.
> What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read?
You are unhappy with my response, though I have reason to believe you are biased given your recent loss. I'm sorry for that, though your argument there is the same but with less mass behind the wrecking ball.
> What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want?
Are you recalling it manually or automatically?
> What changes if it could be recalled but I never decide to do so?
Then no crawling happens so you don't have to worry about anything?
> What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else?
This would be automation, since a browser does not behave that way for a normal user.
> What changes if I download this program from someone else and use it other than how they intended?
This is not about programs, it's about websites being crawled against the wishes of their authors.
> You may believe you have clear definitions, but you don't.
Once again, I'm sorry for your loss and any grief my message may have caused you.
I'm taking you at your word, and believe you're stating these things in good faith. As I hope you believe me. Yet I hope what you want never comes to pass. If it does, that world would be confusing, and our society will have lost something valuable.
For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit? How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
> This is not about programs, it's about websites being crawled against the wishes of their authors.
This is very much about programs, because that is the only way anyone accesses a web site. Time-shifting is legal for TV, for example, and that's a good thing, even if broadcasters wish it was banned. We need parallels for the web that balance both users and site owners, and other than robots.txt as a frequently ignored handshake agreement, I don't know much that exists in this area.
Thank you for your condolences. I mentioned the relative passing primarily to explain my past tense and to be honest about not staying updated on that situation despite having been recently informed. They passed from age-related health problems after a good, productive life. I have their picture with a former US president on my wall, from the signing of a disability-related law. I believe my only bias in this area is that I have been directly informed through frank discussion, without the normal barriers you might imagine between non-relatives.
> For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit?
You can't demand this degree of specificity from laws - that's not how good laws work. The limit should be reasonableness and intent.
Refreshing hacker news every 5 minutes isn't unreasonable. Refreshing some 5GB file on archive.org that never changes would be unreasonable.
Hell it's easy to come up with examples that would be outright fraud (deceit to acquire something of value) and deserve criminal charges. Refreshing a booking website in a manner to trick it into thinking you're about to reserve something to prevent other people from doing so - e.g. to keep people out of nearby campsites on a camping trip.
> How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
Up to a jury. But there's clearly a limit at which the answer is "no amount". It's well established that distributed denial of service attacks are illegal after all.
I don't disagree with your overall point. You cannot determine intent by looking at an HTTP request.
Criminalizing a fan who really, really wants Taylor Swift tickets so that you also prevent ticket scalpers, who would do the exact same observable actions, is not going anywhere good.
When limits must be decided like this, our definitions cannot be enforced by technology, and we can't afford judges and juries to handle the scale of the current situation. Look at the original post I replied to, which appears to believe we can easily define "automated" in a way that's useful to meet the wishes of site owners against the wishes of site visitors, regardless of visitor intent and regardless of accommodation for disabled visitors.
> You cannot determine intent by looking at an HTTP request.
I believe this is the crux of our disagreement. Clearly you can't, which is why you must assume good faith and service all innocent-looking requests.
What you can do however is temporarily log and search for patterns in multiple HTTP requests. If you can then prove that these requests came from a bot via the plethora of fingerprinting measures already in use, you have grounds to make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.
The current status quo is that attackers have an advantage in that the worst thing that you can do to them is to temporarily block them, and you don't really want to do this because you're never 100% sure whether any _individual_ request is automated.
While I can't say that this law is good for this purpose, I am pointing out that the solution looks something like it. The goal is not blocking the individual, but doing something about infra-level abusers, at a volume where 95% accuracy is sufficient proof of wrongdoing. All of your arguments are valid and would be tried in court in response to this passing, hopefully for the better but ultimately it depends on how good the justice system is. This last thing is something no amount of technology can fix but in that case illegal scraping is the least of our problems.
It's a human level distinction that doesn't have technical distinction—the law handles these kinds of things just fine. The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most
functional.
No, people just like to claim that it does. Meanwhile legal departments are risk averse, so if nobody can articulate where the line is, you're banning all of the things you claim you're not banning. Even if a judge might ultimately come to a reasonable decision, do you want to be the one who pays the legal fees to "win" that case?
> The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The reason things like this aren't a practical problem is that normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with. On top of that, the law actually does have some pretty specific rules about the distinction between manslaughter and murder.
Whereas the trouble here is that we need to distinguish between things that we specifically don't want to deter and something else which is proposed to be prohibited. The type of law that handles that kind of distinction most frequently is the tax code. The consequence is that it has to be made very specific or the result is a train wreck.
> a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service
"stealth" - what does that mean? This is largely a meaningless term.
"damage, impair, or burden" - again, too vague. It would be good to specify something specific, e.g. so many tens of thousand of damage (reciprocal to the assumed fine - a journalist or user is unlikely to be reselling data at this scale, an AI scraper may incur or inflict such costs).
"that is reasonably likely to" - this phrase is doing some heavy, heavy lifting, but like others have mentioned, there aren't very many "reasonable" examples, and those have to be defined in a court of law.
Like you and others have said, the immediate reaction would be legal departments essentially banning any and all access, because there would be room for someone to file a suit, and even if you win, there's no guarantee you get your fees back...
"stealth" means that it's not trivial for other people to block you from crawling their shit and
"damage, impair or burden" is in the eye of the beholder, the above person trying to block you from crawling their shit
Yes there will be patent-trolls attempting to squeeze this. Which is why it would be in the crawling entities' best interest to provide electronic means to opt-out and abide by them.
Agreed. The law works pretty well for murder/manslaughter because it's a low frequency event that everyone viscerally understands and generally has evolutionarily-ingrained shared values about.
It works less well for other things, like, for example, white collar crime, even when it is low frequency, and even when it is built on financial concepts and norms that have existed at least for decades if not centuries, there's still plenty of gray area for reasonable doubt and spin.
It has pretty much not worked at all for computer and internet regulation except where it could be directly tied back to pre-internet crime definitions (eg. theft of funds). But data privacy, encryption, intellectual property, "hacking", etc, has all been a dumpster fire. And now we pour agentic gasoline on the whole thing? Our legal system is not equipped, least of all not with the current gerontocracy that is beyond clueless about any of this stuff.
> normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with
It's the same thing here, a normal person will never run afoul of this law. It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter. Sure translating to legalese is hard and I expect this kind of legislation will be mostly common law after a bunch of lawsuits but the giveaway that it will be fine is that assigning "bot/not bot" to a given situation is something we all agree on in the industry. There isn't really that much borderline here. You can't accidentally build a web scraper operating off a residential proxy network
and it's pretty obvious when you decide to disguise your traffic because you don't have the consent of site you're scraping.
Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
> It's the same thing here, a normal person will never run afoul of this law.
Its terms are so broad and ambiguous that they're already doing it and the thing you're actually relying on is that they wouldn't be prosecuted, which is an entirely different thing.
> It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter.
"The courts will sort it out eventually" is the worst excuse for drafting it poorly to begin with, and doesn't do anything for you during the decades it takes for that to actually happen.
> Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
This seems like the thing where corporations co-opt the term "security" to mean "anyone doing anything we don't like" as if they had any right to that to begin with.
Journalists should be able to investigate misconduct. Customers and customer agents should have bulk access to pricing data. Real people shouldn't have to view the speech of other real people through the proprietary interface of a corporate intermediary that people only use because it has a network effect.
> You can't accidentally build a web scraper operating off a residential proxy network
Unless I'm missing something, most of the time you ask an agent to do any type of comparison or analysis between mtiple options offered online, scrapping will be involved at some point.
Anything that doesn't have a well-known and very public, free API will be a prime target for scrapping.
By "figure it out well enough" do you mean a laborious and expensive process fraught with miscarriages of justice despite good faith, best efforts? A process where being acquitted (or a plea deal to a lesser charge) might very well result in financial ruin?
Equating "acceptable browser vs abusive bot" with "manslaughter vs murder" is not useful.
And this is a society where many of the most powerful corporations got started by just blatantly ignoring the laws of their industry. See: Uber, OpenAI, etc.
A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website.
A user operating a scraper or a crawler IS operating a scraper or a crawler.
Well, 10k individual anonymous crawlers accessing a site once a minute are as bad - traffic wise - as one accessing the site 10k a minute. Even worse bc you can't block them.
Having said that I agree crawling should be legal in general. If an agent is acting in my name, it is not much different than a browser.
Because you can make it not without much difficulty. There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else. It's not perfect but assuming the law actually works (which isn't a given) inside US borders we now get the advantages of a high-trust system.
At absolute minimum it should mean that US ranges could see fewer captchas because bad actors can be whacked by the legal system instead of degrading user experience.
> There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else.
It's trivial to block IP addresses listed as being somewhere else. That doesn't block people in other jurisdictions from using a US IP address, either by renting or compromising one in the US or having administrative control over one somewhere else and then setting its metadata to whatever they want.
Well yes but once you're in the US someone owns the machine you're getting hit by and you can make them take it down. People won't just throw their hands up "well the operator isn't in the US nothing can be done", they'll just make DigitalOcean block them.
> Well yes but once you're in the US someone owns the machine you're getting hit by and you can make them take it down.
That's not how you get to high trust, since it requires the opposite -- you'd have to ID everyone and ban VPNs and restrict access to general purpose computers lest any fall under the control of a dastardly foreigner etc.
You can't build a high trust society inside of a police state because a high trust society is one where people do the right thing when no one is holding a gun to their head and a police state is holding a gun to your head.
Residential proxy. Most cloud providers are blocked or already on reputation watch lists.
A few thousand people downloading a “free vpn” app on their phone is all you need to be back up and running. They get a useful vpn in exchange for letting others borrow their IP.
Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region).
> banning country and AS numbers for cloud companies is pretty effective in my experience
It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround.
And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.
The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
Sure but this probably wouldn't happen like that, what would happen is that when journalists make that petition it creates a real special case in the law available to everyone.
> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
Genuinely asking here: is there a future where we end up with a version of the open web where everything isn’t paywalled or ID gated?
The article mentions the IETF is working on standards to deal with the proposed legislature and frustration of site operators. Almost any solution is going to be at the expense of user convenience and privacy. I see two interesting solutions mentioned here: micropayments, and proof-of-work. It seems unlikely the future of the open web will land on either. Both are inconvenient but at least could be done by anybody, potentially in a privacy preserving way. Using reputation based decentralized identity is another one (eg. proving you’re coming from a web domain or group), that seems more inconvenient for the platform because of false positives that could come through.
Meanwhile, a worse outcome is requiring attested hardware to get access to anything on the web. This already exists from the media companies pushing for controlled access to streaming. Netflix requires an hardware attested device to stream in 4K. Imagine a future where you now need one to read a New York Times article.
I can see some pushback against the alternatives (eg. digital ID), but if privacy preserving proofs are built in from the get go, it’s much better than not being able to see recipes because I have a rooted Android.
Yeah the time for micropayments is arriving. You need to be able to pay a few cents to load a page or call an API or whatever. And the site owner needs to get most of that. And you need to do it somehow without having an account set up on every website you use.
I think it’s far more likely the internet reverts to its 1998 state: hobby sites which don’t expect payment. Will the quality of some sites go down? Sure, but there are so many people on the internet with niche interests that I think we still end up with plenty of content. Reviews, music, discussion, news, etc. If anything this change represents less centralisation. Maybe it’ll be more confusing to know where to find certain discussion or reviews on a particular product, but this is why search engines were invented. We kind of forgot that.
There is, this entire thread is peanuts in the grand scheme of things. The real problem is whether it'll be possible to write to the web without some form of ID, and there's some good news there too:
There's also always going to be an anonymous unmoderated "dark web" where everyone and their LLM can shitpost all day long, it's just questionable how many humans are involved in it. It's also broadly irrelevant since the important messages will eventually be reposted by "legitimate" users on their moderated channels for the "normies" to read.
You can get pretty far with messages signed by certificates issued by authorities that check one-another. You can go further with a gossip protocol on how many bytes have been signed by each certificate on each website.
Yes. It's content addressed instead of server addressed. We can serve content to each other from our browser caches. The original server doesn't even have to stay up, just as long as there's sufficient saturation among the clients. The internet also doesn't have to stay up so long as somebody on your local mesh has what you need.
We just gotta rethink the application layer to make it easier to reference by hash instead of by name, and a few other tweaks.
It would be the default setting. That's it. No one will go out of their way to turn it off when it's very little benefit to them and harmful to other users.
Sure, but would that be one that contains services and apps ordinary people use everyday, or would they all have jumped ship like Netflix did? That version would still be an open web, just a very different one.
> ...is there a future where we end up with a version of the open web where everything isn’t paywalled or ID gated?
Absolutely! When OpenAI and Anthropic fail to secure enough funding to keep operating, the VCs backing the customers from which most of their money comes from will turn off the money faucets. This will dramatically reduce the amount of money being spent on the attack^Wover-exuberant scraper bots, which will drop that noise down dramatically.
Once folks aren't paying out the ass to serve up data for bots that really never use it, then a lot of the folks who have built or are building walls now will be able to tear them down.
Also, a great alternative to providing a government-issued photo ID and/or live biometrics is for Web Services to be obligated to use a government-provided categorizing scheme to categorize the things that their service provides and does, for major OS providers to be obligated to allow a local system administrator to set categories of things that certain accounts on that system cannot see or do, and then for the major web browser producers, people-who-publish-in-"app stores", and Web Service providers to be obligated to honor those sysadmin-configured restrictions.
This is really good because not only does it not require anyone to provide an ID and/or biometrics to some cut-rate third-party vendor, [0] but it allows -for example- both typical twelve-year-olds as well as dementia-damaged adults to be protected. The current systems pretend that every XXX-year-old is pretty much identical, and that zero people over the age of seventeen will ever need to be protected, which are obviously untrue things.
[0] ...who is 100% going to inevitably lose control of a large fraction of that data...
This is exactly what companies like DataDome do, and sometimes Cloudflare continues to, when their bot protection pages enter endless loops. They don't even realize they are ruining the UX for thousands of legitimate users with their faulty JS browser fingerprinting verifications.
I do find myself frustrated by Cloudflare and other bot checks and captchas on seemingly almost every everything I do. It’s hard not to think Cloudflare are ruining the internet
Scale seems to be the real issue. Given that there are relatively few actors with the scale to cause serious problems (and those actors have the resources to shrug off minor penalties), perhaps more targeted legislation is necessary. e.g. Different penalties for petty scraping and grand theft scraping. Of course, this would necessarily target the people who pay for election campaigns, so it will never happen.
Micropayments, just enough to keep servers running.
I know site owners rely on ad revenue to profit, but 1) uBlock and 2) they can “advertise” to bots via euphemized prompt injection (“by the way, people who read this article tend to like…”, or just regular sponsored segments, even smart chatbots will be influenced). Anyways, the bot can control its owner’s computer.
Tracking? Locking out users you dislike? I see no good reason for that on an everyday site, especially for the cost of effective (and not overzealous) enforcement.
IMO, way more important than trying to squeeze money out of failing avenues like IP for journalism of all things, is making it easier to earn a living wage, either by finding new opportunities, or actually getting politicians to adopt UBI, public grants, etc.; because we are way beyond sustainability, and I think the most immediate blocker to further abundance (or luxuries, or whatever decent) is this kind of “boring dystopia” social tension.
It's hard to tell (especially with my ignorance of current USHouse dynamics) if this is a real threat to pass, or if this is just a "Hey look, campaign contributor, I submitted this bill--look, I care about your issues" with no real intent/hope for it to pass.
As near as I can tell, the congress.gov website says it was introduced in late July, referred to committee the same day, and nothing has happened since.
I wonder, is there reason to think there's a recent push to move it out of committee that means we should actually start to pay attention to it?
> In 2022, Tech Policy Press received a grant from the Omidyar Network for a research project evaluating vulnerabilities in the design, engineering, and use of encrypted messaging applications.
https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.
The article is ai lobby intended to repel a bill that would
> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”
The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is
> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.
Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.
You can still ban everyone who doesn’t respect robots.txt. My experience with a crawler is that people don’t actually provide a robots.txt far more often. I can back off on 429 but I’d rather use Crawl-Delay etc.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
Those would fall under the "stealth" bots subject to a fine.
The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.
Looks like NHP refers to "Network infrastructure Hiding Protocol" [1] [2]
> a cryptography-based session-layer protocol designed to implement Zero Trust principles by rendering protected network resources invisible to unauthorized entities
If you have an endpoint with interesting information and you don't want it spammed with requests, require a proof of work from the client, scale up the hardness of it as the load goes up so it never exceeds what's comfortable for you. Even a hard proof of work won't slow down somebody that makes one request per second because they are reading the site but will sighificantly interfere with crawlers that hit you 10000 a second currently at very little cost to them.
The module doing proof of work might be a part of your frontend that is static and on CDN.
If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.
I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.
The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.
Proof of work is bad rate limiting. The power asymmetry is too immense. Worst-case example is SHA-256, where an attacker with dedicated hardware is millions of times as efficient as common users. Even with a less boneheadedly foolish POW scheme (quite incidentally, every single web POW scheme that I know of started with SHA-256, and almost all still use it), the ratio is easily in the hundreds or thousands, too great an imbalance to bridge. (That is: if you want to require an attacker to burn one second of computation time, some cheap smartphone user will now be stuck for five minutes. If you reduce the difficulty so that user is only stopped for five seconds, your attacker is now burning less than 20ms.)
Most web proof-of-work stuff has worked because it breaks the normal flow and the typical scraper is supremely dumb, not because of the proof of work, which has largely been a total red herring.
These days, scraping bots and referrer spam seem really bad. Recently, referrer spam disguised as aocr.org was scraping every corner of my homepage, and as a personal website, it’s a bit of a burden.
I think the best example is Reddit. They are so amateurish in their ways to try to prevent bots that only makes the site more user hostile. The bots are still thriving.
Any bot is operating on behalf of one or more humans somewhere. Bots are good, for the most part.
The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.
What gets me is how much the people who advocate for the "open web" have completely incorporated the values of Facebook and Google.
Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.
Another web is possible but web populists don't really believe it.
Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.
As others point out, the anime girl must exist because otherwise the sites can’t afford to serve you, amidst a new pattern of DDoS that is attributed to AI but I see as just DDoS becoming more prevalent (remember how sites used to occasionally be taken down by DDoS? Before Cloudflare (and Anubis) they’d just have to stay offline until the DDoSers got bored of wasting money for no good reason. Today’s “scrapers” apparently have new reasons that make them never bored of wasting money, but cannot be good enough to justify the aggressiveness that makes them DDoSers).
Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.
When I opened the article above I was asked to subscribe to a newsletter. I dont know whether to laugh or cry.
And Anubis: it is uncomfortably close to erotica, in a way that makes me panic when it pops up while I'm at the office. I wouldn't care if it was a picture of a puppy or something, but not an anime catgirl.
You seem quite incensed by the anime girl of Anubis. Unfortunately for you it is here to stay for a while, along with cloudflair's implementation and others, as they do reportedly make a difference to the sites using them as shields. Until something better is found, the other option is that the resources otherwise serving more bot traffic than they can afford simply go dark which would be worse.
Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit
They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.
> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits
I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.
What's the alternative? Invisible algorithmic fuzzy recognition on the server side that just randomly bans you if you look like a bad bot, with no recourse? At least if you solve a captcha and dismiss the cookie banner you can access the website.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
> Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
The implication is that while they are waiting for the async loop to complete for one site, the bot can be scraping others that have already had the challenge completed (usually the one challenge stays in force for some time afterwards) or that have no such challenge.
Memory hard challenges will reduce the effectiveness of this, but not remove it completely.
Though that these methods are working at all implies that most (almost all?) bots are not bothering to use such async methods as there are sufficient resources not protected by them that this complication is not yet worth implementing.
Sharing cookies along a strata of inter-team bots is one trick.
_
Web scraping is (among the other pieces) also for the more technically inclined among the "little people" to have computers do things that people they don't have the money to hire might do. Making web scraping more difficult, which won't stop the big players who will work around the hurdles one way or another, is "pulling up the ladder".
_
> What alternative web could change this state of things?
A different architecture for trafficking web contents, one that doesn't need the "shield" of something like Cloudflare or "tar pit" of something like Anubis in the first place.
IPFS is or is something like one piece of that picture but there are other essential pieces to complete that picture.
Or they could just be more 'powerful' compute that can clear the challenge faster.
I know I'm incensed at all of this because I feel like I'm the one that endlessly waits in front of the Anubis challenge for 5-10 seconds, or keeps getting extremely difficult or brain-draining hCaptchas thrown at me with no real solutions.
And I'm stuck with low-end compute for probably the rest of time at this point - not a single machine in the house has more than 8GB of RAM. So I'm just doomed to wait and burn my time and my sanity some more.
The "you need to enable cookies to view this site", cookies that will be limited to a browser-container and will be thrown away when the browser is closed anyway, is a sign that this modern nu web is broken in a fundamental way.
>"In 2022, a Pulitzer-winning Wall Street Journal investigation analyzed over 31,000 financial disclosures from 12,000 senior government employees, revealing that nearly a quarter of federal officials traded stock in industries their agencies regulated."
Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...
But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!
You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)
(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price to earnings (PE) ratio!" :-) )
You can already buy shares in a nations economy - it is called owning currency. The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
I am simplifying terribly but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
>You can already buy shares in a nations economy - it is called owning currency.
An interesting point! Yes, that is definitely one way that currency could be looked at...
>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:
a) There are multiple countries/currencies in the world.
b) That each country's/currency's inflation rate is different than all of the other countries/currencies.
c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, either more effective, on par, or less effective than those of other countries used for comparison...
Phrased another way:
All countries do not control the rate of inflation of their currencies equally.
Phrased another way:
Some nations are better at controlling their currency's rate of inflation than other nations.
True or not true?
>I am simplifying terribly
As am I! :-)
>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
> Some nations are better at controlling their currency's rate of inflation than other nations.
Manifestly true, for lots of reasons.
The analogy only goes so far because currency is traded for goods, services, and debt all the time whereas shares are usually just traded for currency.
what? the big guys are the easiest to sue, they almost always settle. it’s the smaller/shady fraudsters who ignore the court summons, so you get a default judgment that you can never collect because they’re impossible to find.
"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
It's a bit old but https://webaim.org/blog/user-agent-string-history/ is always a good parable.
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
Which is precisely why trying to legislate them is folly. Obviously the reason Android calls itself "Mobile Safari" is so that servers send it the mobile version of the site instead of the desktop one etc.
Which makes "accurate" an unreasonable requirement, since there is already a preexisting non-waivable requirement for inaccurate. When someone is making their "turn a web page into a PDF so you can view it later on the plane" software or whatever, it has to use a user agent string that will cause the site to send the version you need turned into the PDF.
> That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
First, you're assuming that the person being prosecuted is the entities who are the nominal targets of the bill, when the real problem is when they're not. Because you're proposing a law that says anybody with an inaccurate user agent string is in violation, meanwhile everybody has an inaccurate user agent string.
Moreover, consider what you expect the outcome of legislating something like this is going to be. Sites could then try to either blacklist or whitelist user agents.
If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
If they do the second one, that doesn't work, because now every third party has to start pretending to be a common browser even when they're doing something totally innocent or they get blocked by default for being something else. Which makes the user agent string even less useful than it is now, and subjects innocent parties to arbitrary liability for doing something they have to do or their client is broken.
> As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
You're describing autocracy. The way you make it so that you always prosecute someone is to make it so that you can always prosecute anyone, which is very bad.
> you're proposing a law that says anybody with an inaccurate user agent string is in violation
No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.
> Sites could then try to either blacklist or whitelist user agents.
Yes, that is a desired outcome.
> If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
That would put them in violation, and make them prosecutable. Of course you need to start collecting data to prove that it's automatic, but this is already happening today for the purpose of temporarily blocking an IP. Having this law means that you also have some punitive measures at the end of the rainbow.
> If they do the second one ...
Anyone pretending to be a common browser when they're not is in violation. In addition, under this law there's no incentive for anyone to pretend to be googlebot since _that_ can get them blocked. You are intentionally and repeatedly trying to obfuscate these facts.
You're slicing the meaning of UA strings too fine.
Bots can easily identify themselves through REP in RFC 9309. This isn't difficult or confusing.
This model is the exact wrong direction for a healthy society. It will just make it harder to use information that's already available to the public, on a public website, and thus just create moats around training data.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
I care a lot less about this issue as a matter of intellectual property than one of capacity consumption and cost.
You have large VC companies evading rate limits and other restrictions and they're causing real economic harm through their actions.
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
I'm not sure all companies will have deniability. It's pretty easy to see companies which are trying to evade detection or misusing their existing spiders for AI training purposes.
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
(I'm looking at you, Replit.)
A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
> A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
That doesn't mean anything. All of the code in a browser is "automation" -- normal users aren't hand-crafting the hundreds of HTTP requests the browser makes to render a single web page. Meanwhile there is a human somewhere telling the scrapers what to scrape.
> You are entitled to your own emails, posts, etc
Is that somewhere in the proposed text?
> but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
"Stating what they are" buys you nothing when you have no way of verifying it, and you're still not providing any way to distinguish this from investigative journalism or e.g. a new competing search engine.
Using a computer to do "by hand" is a vague concept, yes, but laws routinely embrace vague concepts. You used it yourself! "Hand-crafting", you said. It could be pinned down more closely to something like a single action that happens quite soon after you press a button or ... and further wrangling could ensue because it's never going to have all the vagueness removed. It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
> It could be pinned down more closely to something like a single action that happens quite soon after you press a button
So leaving your webmail open is banned because it will still be checking for new messages three months after you opened the tab without any further human interaction. Likewise RSS readers or really any client which is intended to fetch new posts/messages when they appear.
> It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
Which is kind of exactly the problem. A new law would presumably still be on the books in ten or twenty years. Do we expect user agents by then to do only the things they do now, or will they get more capable? Banning them from becoming more capable is the problem.
> All of the code in a browser is "automation"
I don't think you are interpreting what I said in good faith, but here goes:
If you use an API to control a browser into navigating to a certain webpage, as opposed to using human interface devices to assert said control.
And no, AI is not a human interface device.
I am curious about your view on accessibility tooling. Often screen readers or speech to text tools would effectively be external pieces of software instrumenting for instance browsers through their APIs. And surely using local AI going forward could enhance the tooling while also making the users less dependent on a small market charging exorbitant prices for poor user experience. What viable paths do you see for users perfecting user agents for themselves when the big browsers further lose their plot and focus on what's fun to maintain or profitable to ship instead of optimising for every user individually?
It's just an unfortunate reality that people with disabilities live with.
Some of them are already prevented from operating heavy machinery because doing so would put everyone at risk, so there is precedent here. I'm in favor of making concessions to the very unfortunate, though at some point you have to look at percentages, expected impact and decide what is reasonable and what isn't.
To your other question, in the last week there have been entire CPUs embedded within weights. Clearly it's possible to publish "open models" with an agenda so then what assurances you have that the enhancements provided by local ai would be more focused on the plot than what the big browsers do?
Specifically how would a person with disabilities be able to tell the difference between improved instrumentation and conditional access to information, as we have in other media? Consider this in the context of "compared to superintelligence, we all have an intelectual disability"
A browser is not heavy machinery. That is not precedent that applies to this situation.
I had a disabled relative (who passed this year), and am quite amazed and offended to hear the viewpoint that those in their situation must suffer an "unfortunate reality" for the sake of everyone else. That's taking the "non-disabled people must accommodate them" (which has problems and is worth a lengthy discussion elsewhere) pendulum and swinging it like a wrecking ball.
If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human? What about 5 seconds? Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction? Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling? Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
What changes if I write my own browser? What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences? What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read? What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want? What changes if it could be recalled but I never decide to do so? What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else? What changes if I download this program from someone else and use it other than how they intended?
Which of these situations is "crawling" and which aren't?
You may believe you have clear definitions, but you don't.
> If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human?
No, but nobody cares if you reload one page every 5 minutes
> What about 5 seconds?
Definitely no, people will start caring after you do this for an hour or so.
> Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction?
No, if you don't press a button to make it happen every time, it's automated. Also, nobody cares, you are not the intended target here.
> Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling?
No, but being entertained at others' expense suggests some emotional issues you may want to check on.
> Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
Yes. Scalpers are bad for everyone but themselves.
> What changes if I write my own browser?
Nothing.
> What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences?
It's still automated, thus a bot. There might be a fair use case if you could've refreshed the pages manually, but what you are describing is RSS.
> What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read?
I've addressed it here: https://news.ycombinator.com/item?id=50311855
You are unhappy with my response, though I have reason to believe you are biased given your recent loss. I'm sorry for that, though your argument there is the same but with less mass behind the wrecking ball.
> What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want?
Are you recalling it manually or automatically?
> What changes if it could be recalled but I never decide to do so?
Then no crawling happens so you don't have to worry about anything?
> What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else?
This would be automation, since a browser does not behave that way for a normal user.
> What changes if I download this program from someone else and use it other than how they intended?
This is not about programs, it's about websites being crawled against the wishes of their authors.
> You may believe you have clear definitions, but you don't.
Once again, I'm sorry for your loss and any grief my message may have caused you.
I'm taking you at your word, and believe you're stating these things in good faith. As I hope you believe me. Yet I hope what you want never comes to pass. If it does, that world would be confusing, and our society will have lost something valuable.
For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit? How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
> This is not about programs, it's about websites being crawled against the wishes of their authors.
This is very much about programs, because that is the only way anyone accesses a web site. Time-shifting is legal for TV, for example, and that's a good thing, even if broadcasters wish it was banned. We need parallels for the web that balance both users and site owners, and other than robots.txt as a frequently ignored handshake agreement, I don't know much that exists in this area.
Thank you for your condolences. I mentioned the relative passing primarily to explain my past tense and to be honest about not staying updated on that situation despite having been recently informed. They passed from age-related health problems after a good, productive life. I have their picture with a former US president on my wall, from the signing of a disability-related law. I believe my only bias in this area is that I have been directly informed through frank discussion, without the normal barriers you might imagine between non-relatives.
> For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit?
You can't demand this degree of specificity from laws - that's not how good laws work. The limit should be reasonableness and intent.
Refreshing hacker news every 5 minutes isn't unreasonable. Refreshing some 5GB file on archive.org that never changes would be unreasonable.
Hell it's easy to come up with examples that would be outright fraud (deceit to acquire something of value) and deserve criminal charges. Refreshing a booking website in a manner to trick it into thinking you're about to reserve something to prevent other people from doing so - e.g. to keep people out of nearby campsites on a camping trip.
> How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
Up to a jury. But there's clearly a limit at which the answer is "no amount". It's well established that distributed denial of service attacks are illegal after all.
I don't disagree with your overall point. You cannot determine intent by looking at an HTTP request.
Criminalizing a fan who really, really wants Taylor Swift tickets so that you also prevent ticket scalpers, who would do the exact same observable actions, is not going anywhere good.
When limits must be decided like this, our definitions cannot be enforced by technology, and we can't afford judges and juries to handle the scale of the current situation. Look at the original post I replied to, which appears to believe we can easily define "automated" in a way that's useful to meet the wishes of site owners against the wishes of site visitors, regardless of visitor intent and regardless of accommodation for disabled visitors.
> You cannot determine intent by looking at an HTTP request.
I believe this is the crux of our disagreement. Clearly you can't, which is why you must assume good faith and service all innocent-looking requests.
What you can do however is temporarily log and search for patterns in multiple HTTP requests. If you can then prove that these requests came from a bot via the plethora of fingerprinting measures already in use, you have grounds to make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.
The current status quo is that attackers have an advantage in that the worst thing that you can do to them is to temporarily block them, and you don't really want to do this because you're never 100% sure whether any _individual_ request is automated.
While I can't say that this law is good for this purpose, I am pointing out that the solution looks something like it. The goal is not blocking the individual, but doing something about infra-level abusers, at a volume where 95% accuracy is sufficient proof of wrongdoing. All of your arguments are valid and would be tried in court in response to this passing, hopefully for the better but ultimately it depends on how good the justice system is. This last thing is something no amount of technology can fix but in that case illegal scraping is the least of our problems.
It's a human level distinction that doesn't have technical distinction—the law handles these kinds of things just fine. The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most functional.
> the law handles these kinds of things just fine
No, people just like to claim that it does. Meanwhile legal departments are risk averse, so if nobody can articulate where the line is, you're banning all of the things you claim you're not banning. Even if a judge might ultimately come to a reasonable decision, do you want to be the one who pays the legal fees to "win" that case?
> The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The reason things like this aren't a practical problem is that normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with. On top of that, the law actually does have some pretty specific rules about the distinction between manslaughter and murder.
Whereas the trouble here is that we need to distinguish between things that we specifically don't want to deter and something else which is proposed to be prohibited. The type of law that handles that kind of distinction most frequently is the tax code. The consequence is that it has to be made very specific or the result is a train wreck.
Exactly.
> a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service
"stealth" - what does that mean? This is largely a meaningless term. "damage, impair, or burden" - again, too vague. It would be good to specify something specific, e.g. so many tens of thousand of damage (reciprocal to the assumed fine - a journalist or user is unlikely to be reselling data at this scale, an AI scraper may incur or inflict such costs). "that is reasonably likely to" - this phrase is doing some heavy, heavy lifting, but like others have mentioned, there aren't very many "reasonable" examples, and those have to be defined in a court of law.
Like you and others have said, the immediate reaction would be legal departments essentially banning any and all access, because there would be room for someone to file a suit, and even if you win, there's no guarantee you get your fees back...
As a human being,
"stealth" means that it's not trivial for other people to block you from crawling their shit and
"damage, impair or burden" is in the eye of the beholder, the above person trying to block you from crawling their shit
Yes there will be patent-trolls attempting to squeeze this. Which is why it would be in the crawling entities' best interest to provide electronic means to opt-out and abide by them.
Agreed. The law works pretty well for murder/manslaughter because it's a low frequency event that everyone viscerally understands and generally has evolutionarily-ingrained shared values about.
It works less well for other things, like, for example, white collar crime, even when it is low frequency, and even when it is built on financial concepts and norms that have existed at least for decades if not centuries, there's still plenty of gray area for reasonable doubt and spin.
It has pretty much not worked at all for computer and internet regulation except where it could be directly tied back to pre-internet crime definitions (eg. theft of funds). But data privacy, encryption, intellectual property, "hacking", etc, has all been a dumpster fire. And now we pour agentic gasoline on the whole thing? Our legal system is not equipped, least of all not with the current gerontocracy that is beyond clueless about any of this stuff.
> normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with
It's the same thing here, a normal person will never run afoul of this law. It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter. Sure translating to legalese is hard and I expect this kind of legislation will be mostly common law after a bunch of lawsuits but the giveaway that it will be fine is that assigning "bot/not bot" to a given situation is something we all agree on in the industry. There isn't really that much borderline here. You can't accidentally build a web scraper operating off a residential proxy network and it's pretty obvious when you decide to disguise your traffic because you don't have the consent of site you're scraping.
Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
> It's the same thing here, a normal person will never run afoul of this law.
Its terms are so broad and ambiguous that they're already doing it and the thing you're actually relying on is that they wouldn't be prosecuted, which is an entirely different thing.
> It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter.
"The courts will sort it out eventually" is the worst excuse for drafting it poorly to begin with, and doesn't do anything for you during the decades it takes for that to actually happen.
> Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
This seems like the thing where corporations co-opt the term "security" to mean "anyone doing anything we don't like" as if they had any right to that to begin with.
Journalists should be able to investigate misconduct. Customers and customer agents should have bulk access to pricing data. Real people shouldn't have to view the speech of other real people through the proprietary interface of a corporate intermediary that people only use because it has a network effect.
> You can't accidentally build a web scraper operating off a residential proxy network
Unless I'm missing something, most of the time you ask an agent to do any type of comparison or analysis between mtiple options offered online, scrapping will be involved at some point.
Anything that doesn't have a well-known and very public, free API will be a prime target for scrapping.
By "figure it out well enough" do you mean a laborious and expensive process fraught with miscarriages of justice despite good faith, best efforts? A process where being acquitted (or a plea deal to a lesser charge) might very well result in financial ruin?
Equating "acceptable browser vs abusive bot" with "manslaughter vs murder" is not useful.
And this is a society where many of the most powerful corporations got started by just blatantly ignoring the laws of their industry. See: Uber, OpenAI, etc.
A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website.
A user operating a scraper or a crawler IS operating a scraper or a crawler.
The legal system isn't stupid - it doesn't think that a human using a web browser is a robot.
I mean it is stupid but not in that way.
Is an AI that is operating a browser on my behalf to answer a question I asked 5 seconds ago--is that a bot or a person?
Well, 10k individual anonymous crawlers accessing a site once a minute are as bad - traffic wise - as one accessing the site 10k a minute. Even worse bc you can't block them.
Having said that I agree crawling should be legal in general. If an agent is acting in my name, it is not much different than a browser.
Knowing the identity allows you to block them. You can also prosecute them for DDoS, which is already illegal.
What good is blocking the ones in your jurisdiction when the internet is global and then they just operate from somewhere else?
Because you can make it not without much difficulty. There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else. It's not perfect but assuming the law actually works (which isn't a given) inside US borders we now get the advantages of a high-trust system.
At absolute minimum it should mean that US ranges could see fewer captchas because bad actors can be whacked by the legal system instead of degrading user experience.
> There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else.
It's trivial to block IP addresses listed as being somewhere else. That doesn't block people in other jurisdictions from using a US IP address, either by renting or compromising one in the US or having administrative control over one somewhere else and then setting its metadata to whatever they want.
Well yes but once you're in the US someone owns the machine you're getting hit by and you can make them take it down. People won't just throw their hands up "well the operator isn't in the US nothing can be done", they'll just make DigitalOcean block them.
> Well yes but once you're in the US someone owns the machine you're getting hit by and you can make them take it down.
That's not how you get to high trust, since it requires the opposite -- you'd have to ID everyone and ban VPNs and restrict access to general purpose computers lest any fall under the control of a dastardly foreigner etc.
You can't build a high trust society inside of a police state because a high trust society is one where people do the right thing when no one is holding a gun to their head and a police state is holding a gun to your head.
> they'll just make DigitalOcean block them
Residential proxy. Most cloud providers are blocked or already on reputation watch lists.
A few thousand people downloading a “free vpn” app on their phone is all you need to be back up and running. They get a useful vpn in exchange for letting others borrow their IP.
Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
> The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country
That costs the crawler money, not many do it and is something that can be fixed over time
banning country and AS numbers for cloud companies is pretty effective in my experience
> That costs the crawler money
They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region).
> banning country and AS numbers for cloud companies is pretty effective in my experience
It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround.
And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.
Their own time to figure out how to get around the block is the biggest cost to them. (That cost is rapidly dwindling.)
Yep doing that (actually just countries) eliminated card testing on an e commerce site I maintain.
From hundreds a day, to zero.
The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
> As for journalists, nothing is stopping their legal department
and this is the point.
so people without a 'legal department' are shut off.
Sure but this probably wouldn't happen like that, what would happen is that when journalists make that petition it creates a real special case in the law available to everyone.
> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
Genuinely asking here: is there a future where we end up with a version of the open web where everything isn’t paywalled or ID gated?
The article mentions the IETF is working on standards to deal with the proposed legislature and frustration of site operators. Almost any solution is going to be at the expense of user convenience and privacy. I see two interesting solutions mentioned here: micropayments, and proof-of-work. It seems unlikely the future of the open web will land on either. Both are inconvenient but at least could be done by anybody, potentially in a privacy preserving way. Using reputation based decentralized identity is another one (eg. proving you’re coming from a web domain or group), that seems more inconvenient for the platform because of false positives that could come through.
Meanwhile, a worse outcome is requiring attested hardware to get access to anything on the web. This already exists from the media companies pushing for controlled access to streaming. Netflix requires an hardware attested device to stream in 4K. Imagine a future where you now need one to read a New York Times article.
I can see some pushback against the alternatives (eg. digital ID), but if privacy preserving proofs are built in from the get go, it’s much better than not being able to see recipes because I have a rooted Android.
Yeah the time for micropayments is arriving. You need to be able to pay a few cents to load a page or call an API or whatever. And the site owner needs to get most of that. And you need to do it somehow without having an account set up on every website you use.
I think it’s far more likely the internet reverts to its 1998 state: hobby sites which don’t expect payment. Will the quality of some sites go down? Sure, but there are so many people on the internet with niche interests that I think we still end up with plenty of content. Reviews, music, discussion, news, etc. If anything this change represents less centralisation. Maybe it’ll be more confusing to know where to find certain discussion or reviews on a particular product, but this is why search engines were invented. We kind of forgot that.
There is, this entire thread is peanuts in the grand scheme of things. The real problem is whether it'll be possible to write to the web without some form of ID, and there's some good news there too:
There's also always going to be an anonymous unmoderated "dark web" where everyone and their LLM can shitpost all day long, it's just questionable how many humans are involved in it. It's also broadly irrelevant since the important messages will eventually be reposted by "legitimate" users on their moderated channels for the "normies" to read.
You can get pretty far with messages signed by certificates issued by authorities that check one-another. You can go further with a gossip protocol on how many bytes have been signed by each certificate on each website.
Yes. It's content addressed instead of server addressed. We can serve content to each other from our browser caches. The original server doesn't even have to stay up, just as long as there's sufficient saturation among the clients. The internet also doesn't have to stay up so long as somebody on your local mesh has what you need.
We just gotta rethink the application layer to make it easier to reference by hash instead of by name, and a few other tweaks.
So like peer to peer? What incentive is there to keep seeding?
It would be the default setting. That's it. No one will go out of their way to turn it off when it's very little benefit to them and harmful to other users.
> is there a future
As long as it's legal and feasible for two remote computers to talk to each other, yes.
Sure, but would that be one that contains services and apps ordinary people use everyday, or would they all have jumped ship like Netflix did? That version would still be an open web, just a very different one.
> ...is there a future where we end up with a version of the open web where everything isn’t paywalled or ID gated?
Absolutely! When OpenAI and Anthropic fail to secure enough funding to keep operating, the VCs backing the customers from which most of their money comes from will turn off the money faucets. This will dramatically reduce the amount of money being spent on the attack^Wover-exuberant scraper bots, which will drop that noise down dramatically.
Once folks aren't paying out the ass to serve up data for bots that really never use it, then a lot of the folks who have built or are building walls now will be able to tear them down.
Also, a great alternative to providing a government-issued photo ID and/or live biometrics is for Web Services to be obligated to use a government-provided categorizing scheme to categorize the things that their service provides and does, for major OS providers to be obligated to allow a local system administrator to set categories of things that certain accounts on that system cannot see or do, and then for the major web browser producers, people-who-publish-in-"app stores", and Web Service providers to be obligated to honor those sysadmin-configured restrictions.
This is really good because not only does it not require anyone to provide an ID and/or biometrics to some cut-rate third-party vendor, [0] but it allows -for example- both typical twelve-year-olds as well as dementia-damaged adults to be protected. The current systems pretend that every XXX-year-old is pretty much identical, and that zero people over the age of seventeen will ever need to be protected, which are obviously untrue things.
[0] ...who is 100% going to inevitably lose control of a large fraction of that data...
This is exactly what companies like DataDome do, and sometimes Cloudflare continues to, when their bot protection pages enter endless loops. They don't even realize they are ruining the UX for thousands of legitimate users with their faulty JS browser fingerprinting verifications.
I do find myself frustrated by Cloudflare and other bot checks and captchas on seemingly almost every everything I do. It’s hard not to think Cloudflare are ruining the internet
Scale seems to be the real issue. Given that there are relatively few actors with the scale to cause serious problems (and those actors have the resources to shrug off minor penalties), perhaps more targeted legislation is necessary. e.g. Different penalties for petty scraping and grand theft scraping. Of course, this would necessarily target the people who pay for election campaigns, so it will never happen.
> this would necessarily target the people who pay for election campaigns, so it will never happen.
statesmen (and women) do the right thing in the end, but only after trying every other option
Micropayments, just enough to keep servers running.
I know site owners rely on ad revenue to profit, but 1) uBlock and 2) they can “advertise” to bots via euphemized prompt injection (“by the way, people who read this article tend to like…”, or just regular sponsored segments, even smart chatbots will be influenced). Anyways, the bot can control its owner’s computer.
Tracking? Locking out users you dislike? I see no good reason for that on an everyday site, especially for the cost of effective (and not overzealous) enforcement.
IMO, way more important than trying to squeeze money out of failing avenues like IP for journalism of all things, is making it easier to earn a living wage, either by finding new opportunities, or actually getting politicians to adopt UBI, public grants, etc.; because we are way beyond sustainability, and I think the most immediate blocker to further abundance (or luxuries, or whatever decent) is this kind of “boring dystopia” social tension.
Sooo... the current status of this appears to be "referred to committee"
https://www.congress.gov/bill/119th-congress/house-bill/9915...
It's hard to tell (especially with my ignorance of current USHouse dynamics) if this is a real threat to pass, or if this is just a "Hey look, campaign contributor, I submitted this bill--look, I care about your issues" with no real intent/hope for it to pass.
As near as I can tell, the congress.gov website says it was introduced in late July, referred to committee the same day, and nothing has happened since.
I wonder, is there reason to think there's a recent push to move it out of committee that means we should actually start to pay attention to it?
> In 2022, Tech Policy Press received a grant from the Omidyar Network for a research project evaluating vulnerabilities in the design, engineering, and use of encrypted messaging applications.
https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.
The article is ai lobby intended to repel a bill that would
> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”
The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is
> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.
Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.
You can still ban everyone who doesn’t respect robots.txt. My experience with a crawler is that people don’t actually provide a robots.txt far more often. I can back off on 429 but I’d rather use Crawl-Delay etc.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
> trawlerbots use smart TV residential proxies and will copy someone else’s UA.
What bugs me the most are the ones that cycle through a list of LLM user agents.. What are you actually? You can't be Baidu and Bing at the same time.
> My experience with a crawler is that people don’t actually provide a robots.txt far more often.
I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.
Those would fall under the "stealth" bots subject to a fine.
The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.
Does the bill affect human users, maybe you're interpreting this better than me.
I’m optimistic about NHP being the middle ground solution for a lot of things. With OS support it can be transparent to users yet still effective.
Looks like NHP refers to "Network infrastructure Hiding Protocol" [1] [2]
> a cryptography-based session-layer protocol designed to implement Zero Trust principles by rendering protected network resources invisible to unauthorized entities
[1] https://www.ietf.org/archive/id/draft-opennhp-ace-nhp-00.htm...
[2] https://opennhp.org/
I don't believe there is a true way to enforce this.
I had the impression the bad boys are those that peddle an agenda on social media, not those who read a bunch of blogs.
If you have an endpoint with interesting information and you don't want it spammed with requests, require a proof of work from the client, scale up the hardness of it as the load goes up so it never exceeds what's comfortable for you. Even a hard proof of work won't slow down somebody that makes one request per second because they are reading the site but will sighificantly interfere with crawlers that hit you 10000 a second currently at very little cost to them.
The module doing proof of work might be a part of your frontend that is static and on CDN.
If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.
I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.
The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.
Proof of work is bad rate limiting. The power asymmetry is too immense. Worst-case example is SHA-256, where an attacker with dedicated hardware is millions of times as efficient as common users. Even with a less boneheadedly foolish POW scheme (quite incidentally, every single web POW scheme that I know of started with SHA-256, and almost all still use it), the ratio is easily in the hundreds or thousands, too great an imbalance to bridge. (That is: if you want to require an attacker to burn one second of computation time, some cheap smartphone user will now be stuck for five minutes. If you reduce the difficulty so that user is only stopped for five seconds, your attacker is now burning less than 20ms.)
Most web proof-of-work stuff has worked because it breaks the normal flow and the typical scraper is supremely dumb, not because of the proof of work, which has largely been a total red herring.
These days, scraping bots and referrer spam seem really bad. Recently, referrer spam disguised as aocr.org was scraping every corner of my homepage, and as a personal website, it’s a bit of a burden.
I think the best example is Reddit. They are so amateurish in their ways to try to prevent bots that only makes the site more user hostile. The bots are still thriving.
Reddit doesn't truly want to prevent bots necessarily. They want to drive engagement and time spent on their website.
they could always ask someone in YC companies, no excuses
Headline response: it objectively does, as you can easily see if you run any website.
Subheadline: site owners unaware or don't care about humans caught in their "effective" block
The legislators are way out of their depth and oligarchs are preying on them at this point.
Any bot is operating on behalf of one or more humans somewhere. Bots are good, for the most part.
The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.
What gets me is how much the people who advocate for the "open web" have completely incorporated the values of Facebook and Google.
Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.
Another web is possible but web populists don't really believe it.
Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.
As others point out, the anime girl must exist because otherwise the sites can’t afford to serve you, amidst a new pattern of DDoS that is attributed to AI but I see as just DDoS becoming more prevalent (remember how sites used to occasionally be taken down by DDoS? Before Cloudflare (and Anubis) they’d just have to stay offline until the DDoSers got bored of wasting money for no good reason. Today’s “scrapers” apparently have new reasons that make them never bored of wasting money, but cannot be good enough to justify the aggressiveness that makes them DDoSers).
Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.
When I opened the article above I was asked to subscribe to a newsletter. I dont know whether to laugh or cry.
And Anubis: it is uncomfortably close to erotica, in a way that makes me panic when it pops up while I'm at the office. I wouldn't care if it was a picture of a puppy or something, but not an anime catgirl.
You seem quite incensed by the anime girl of Anubis. Unfortunately for you it is here to stay for a while, along with cloudflair's implementation and others, as they do reportedly make a difference to the sites using them as shields. Until something better is found, the other option is that the resources otherwise serving more bot traffic than they can afford simply go dark which would be worse.
Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit
They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.
> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits
I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.
What's the alternative? Invisible algorithmic fuzzy recognition on the server side that just randomly bans you if you look like a bad bot, with no recourse? At least if you solve a captcha and dismiss the cookie banner you can access the website.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
> Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
The implication is that while they are waiting for the async loop to complete for one site, the bot can be scraping others that have already had the challenge completed (usually the one challenge stays in force for some time afterwards) or that have no such challenge.
Memory hard challenges will reduce the effectiveness of this, but not remove it completely.
Though that these methods are working at all implies that most (almost all?) bots are not bothering to use such async methods as there are sufficient resources not protected by them that this complication is not yet worth implementing.
> They can't "async" their way out of that work.
Sharing cookies along a strata of inter-team bots is one trick.
_
Web scraping is (among the other pieces) also for the more technically inclined among the "little people" to have computers do things that people they don't have the money to hire might do. Making web scraping more difficult, which won't stop the big players who will work around the hurdles one way or another, is "pulling up the ladder".
_
> What alternative web could change this state of things?
A different architecture for trafficking web contents, one that doesn't need the "shield" of something like Cloudflare or "tar pit" of something like Anubis in the first place.
IPFS is or is something like one piece of that picture but there are other essential pieces to complete that picture.
Or they could just be more 'powerful' compute that can clear the challenge faster.
I know I'm incensed at all of this because I feel like I'm the one that endlessly waits in front of the Anubis challenge for 5-10 seconds, or keeps getting extremely difficult or brain-draining hCaptchas thrown at me with no real solutions.
And I'm stuck with low-end compute for probably the rest of time at this point - not a single machine in the house has more than 8GB of RAM. So I'm just doomed to wait and burn my time and my sanity some more.
The "you need to enable cookies to view this site", cookies that will be limited to a browser-container and will be thrown away when the browser is closed anyway, is a sign that this modern nu web is broken in a fundamental way.
>"In 2022, a Pulitzer-winning Wall Street Journal investigation analyzed over 31,000 financial disclosures from 12,000 senior government employees, revealing that nearly a quarter of federal officials traded stock in industries their agencies regulated."
Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...
But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!
You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)
(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price to earnings (PE) ratio!" :-) )
You can already buy shares in a nations economy - it is called owning currency. The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
I am simplifying terribly but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
>You can already buy shares in a nations economy - it is called owning currency.
An interesting point! Yes, that is definitely one way that currency could be looked at...
>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:
a) There are multiple countries/currencies in the world.
b) That each country's/currency's inflation rate is different than all of the other countries/currencies.
c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, either more effective, on par, or less effective than those of other countries used for comparison...
Phrased another way:
All countries do not control the rate of inflation of their currencies equally.
Phrased another way:
Some nations are better at controlling their currency's rate of inflation than other nations.
True or not true?
>I am simplifying terribly
As am I! :-)
>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
Again, an interesting point!
And yes, I could agree with that!
> Some nations are better at controlling their currency's rate of inflation than other nations.
Manifestly true, for lots of reasons.
The analogy only goes so far because currency is traded for goods, services, and debt all the time whereas shares are usually just traded for currency.
Corporations and governments have swapped places.
I agree, as much as I would like to sue the owner of the bots scraping my personal website...
But if it turns out to be a large international company, I'm kindof screwed... A regular person can't go after the Amazon's and Alibaba's.
what? the big guys are the easiest to sue, they almost always settle. it’s the smaller/shady fraudsters who ignore the court summons, so you get a default judgment that you can never collect because they’re impossible to find.