If you want to vibe something that gets you 70% of the way to this well funded startup in like 15 minutes just tell your LLM of choice to create a hook or override the web fetching skill to pipe the content through Mozilla’s readability extension that strips the DOM elements out deterministically, leaving only the content. You can then parse it however you want. Can be done entirely client side, in runtime, with a few JavaScript libraries
"the first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time."
now for my actual thoughts, if that works for anyone in production reliably, then they should at-least try to do that.
many of my customers moved from in-house solutions because the engineering time spent on maintaining something like this alone makes it quite the undertaking, the infra needed to support something like a few hundred TPS w a browser fleet would cost more than a sub to us in the majority of cases
Conversely, many of my customers abandoned many of their SaaS solutions because they can now hand roll the 10% of the SaaS solution they need in 15 minutes.
That’s not to say all SaaS is going away. But a good hunk of the single simple solution products are going to suffer
Hi! Congrats on the launch, I gained LOTS of great insights from your comments, particualrly the bits about diffrentiation in a crowded market.
I am building something in agentic automation space, currently it's still under development, but would love to know your journey of ideating -> building -> getting the first customer -> iterating -> and presumably getting into YC.
Am still relatively new to this (19 lol) so I got a long way, but would really appreciate any insights :)
i've yapped quite a bit in this thread already about my journey, but happy to provide any (hopefully useful) insights, you can DM me on X, mention this comment please: https://x.com/mynameisyahia
I like the clarity, tone, and readability of your webpage. Also your FAQ is refreshing
> When Should I talk to sales?
> Talk to sales if you need high-volume pricing beyond 2M credits/month, custom rate limits, SSO / SAML, SCIM provisioning, an uptime SLA, annual invoicing, an MSA / DPA, or a dedicated support channel. Reach us at hello@context.dev or through the contact page.
Would that this were the norm everywhere, rather than (say) a sales rep from Datadog scraping my phone number from who knows where to ask about my company's needs after I sign up for a free account on a whim :)
happy you noticed that, i put quite a bit of love into the ui, i've found engineers care alot about polish & feel of the webapps they use, even for an api product
i'm an engineer by trade, and always hated things like forcing a sales call, or having hidden credit multipliers, i tried to build this with the same ethos i like for my own dependencies (shoutout axiom.co)
Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
What major edge cases have you found so far for the browser scraping method, its really complex like dealing with Auth, or pages that use pop ups, content blockers . etc...
if it needs auth, we don't do any type of logins at the moment, still figuring out how to do it without attracting the wrong type of customer (someone doing shady stuff)
popups / cookie banners are quite easy to get through with enough heuristics for 99%+ of cases
content-blockers are the same thing as the auth scenario, hard to figure out where to draw the line between building something people want and doing something shady
- worst is "extension" based traffic routing where your provider is essentially tricking end users into routing traffic through
- opt-in p2p networks can be described as botnets sometimes, i try to avoid these but not as bad
- good enough dISP proxies can be classified as residential quality, work quite well, although people have argued w me on their quality
- managed business/enterprise residential gateways are ideal, because businesses opt in and allow the traffic
- few other options
Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful.
> Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON API that bypasses almost all bot protections — including Turnstile. I didn't include the ability, but it bypassed the most advanced ChatGPT + Turnstile. Built with self-improving Claude Code agents that rewrite their own instructions until fresh agents consistently succeed.
> Once connected to a page, it intercepts every byte of network traffic — then actively drives the page to surface endpoints that only fire on interaction. It types into forms, clicks buttons, scrolls, triggers modals, paginates, submits searches, and walks through multi-step flows, watching what each action produces on the wire. Every request gets captured with its method, headers, payload shape, and response, then classified by transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub). The result is a complete map of the site's real API surface — including the hidden endpoints that only exist behind a click — turned into typed proxy routes you can curl.
Fascinating, this is definitely a path i was thinking about for a bit but decided against doing it primarily since it changes the business fundamentally
Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted.
> We maintain a caching layer and avoid hammering websites.
This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.
How did you find your differentiation in a highly commoditized space? It's probably one of the most crowded spaces.
Even within YC, there are many competitors that do pretty much the same thing:
- Firecrawl
- BrowserUse
- Browserbase
- CloudCruise
- NotteLabs
- Intuned
- Expand.ai
- Reworkd
And then you have the extremely well-funded web retrieval players like Parallel and Exa.
How do you differentiate to all these?
Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Context respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
the reality is that the space is huge and expanding
many that you mentioned serve different needs and it tends to not be zero sum to everyone's surprise (including mine at first)
e.g.
browseruse, browserbase = browser automation
exa, parallel = search for ai (primarily)
we differentiate by prioritizing the infrastructure use-case, we encourage people to build products & features on top of our APIs and build it accordingly
some direct differentiators are: no hidden credit multipliers, world class brand data, and an api that just works
we're trying to be good samaritans on the web, and cache data whenever possible so we can avoid overloading a website, it's the main reason we don't offer "login with ai" features even though it's trivial at this point, it's quite hard to figure out how to avoid shady behavior while building something people want, i don't have a good answer yet and neither do many others
lastly, to reiterate, lots of our customers migrate from alternatives, but lots also just use multiple providers
firecrawl doesn't have a search index, we're probably closest to them at the moment
tavily, exa, parallel, you = search index for ai
cloudflare's browser offerings are not the best in terms of quality, at-least from our testing + anecdotal reports
anything specific you want me to compare across these companies? to outsiders they look the same but it's actually quite a diverse lineup you've asked about
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up
Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be legible to agents. Some are already adapting but many haven’t yet. Context feels like an important part of that transition
Yahia is a great builder. His pace of expansion has been impressive, excited to see where he takes Context.
Unclear what difference exists against Firecrawl - their team has been shipping great features extremely quickly lately, and their core offerings have become really good.
I am interested in KnifeGeek though - looking for a good OTF (ultratech?)
There's alot of overlap at the moment, but we're diverging quite quickly.
If you want differences as of right now
- 1 credit = 1 scrape, no hidden credit multiplier
- we have world class brand data
- we're focused primarily on the infra use-case, rather than gtm & everything else
- anecdotally, customers have seen their error rates drop quite dramatically
In general it's a huge space, firecrawl is a wonderful company, it's fun to compete with them, planning to add more things soon which should make the differences clear
please do, and if you see anything off let me know, we've yet to lose a single "bakeoff", i normally ask customers to just have cc/codex run it so its somewhat unbiased
on the 2nd point, most industries are not zero-sum, and many of customers use multiple data providers in any case, so agree with you there
oh also on knifegeek, it'll be live again this weekend when i find a moment to fix the bug causing it to crash, i collect knives/watches so it was a super fun project to work on
we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time
Hey Yahia, I recently saw your startup on the YC registry and found it to be one of the most promising of the batch, so good luck with this! Do you mind if I ask how you managed to get your first users on Brand.dev?
hey, happy you like it, to be honest we had no real secret, it takes a ton of time and effort to earn developer trust
i got my first customer after 1 month, and then just worked like hell to make him happy even though it was a very large company paying me $99/mo at the time, they're still a customer too
since then, i just locked in on every customer until they were thrilled, and once we hit 50, word of mouth started to kick in, now we're at 300+ and get a referral every other day which is awesome
i think our community appreciates effort, and when someone goes the extra mile to make sure your problems are solved, you tend to remember that. it doesn't scale super well, but you'd be surprised how far it'll take you
> Websites can opt out of our service, and we respect these requests and add them to our block list.
I.e: "robots.txt already exists and is commonplace but if we actually followed it our solution wouldn't work so all websites on the planet need to do something specific to opt-out our ddos."
lmao
i too would have had this view before i started the business, however it's actually really hard, additionally teams migrate to us from in-house solutions all the time
This seems like a responsibly designed service, but I find it a little odd and baffling that we need such intermediaries for the average hobbyist / small project to reliably access sets of content published on the internet.
i think about this too, perhaps you're right, if (big if) the web was designed with the end goal in place, however it was a series of ~interesting~ mistakes & decisions that led to where we are today
the same could be said of most cybersecurity companies too
im not sure the models do this, rather the tooling around them, however web search/extraction by the model providers is ridiculously expensive and quite slow, so going into production it makes sense to use a provider (like us)
Great, another thing I have to block server side. Reminds me of the image leech protections that had to be in place because bandwidth was expensive. History doesn’t repeat but rhymes as they say.
Yep. It will be interesting to see if things move that way or not. AI scraping might get annoying but a lot of sites (eg e-commerce) have obvious incentives to allow AIs to scrape them for free (or even pay AI companies to prioritize them, like they already do with Google).
yes but we do alot more that may not be clear at first glance, things like brand data, and for scraping handling pdf, ocr, docx, ppt, xlsx automatically
shipping a bunch of new things soon which should make it clearer, but as of today yeah
you're welcome to try it on our demo page (no signup needed), should handle everything just fine, yes we don't mention ips on the homepage
also, while it might seem expensive, we're cheaper than every other option out there, because there's no hidden credit multipliers. every single customer who uses us halved their bill + error rate
im not sure if grok is on here, i think that's an X thing
but if you were to ask me, we're more fully featured than cloudflare, and anecdotally a ton more reliable in terms of error rates. back when it was brand.dev, i actually tried to use cloudflare's apis and it was quite unreliable, so we built our own stack instead
If you want to vibe something that gets you 70% of the way to this well funded startup in like 15 minutes just tell your LLM of choice to create a hook or override the web fetching skill to pipe the content through Mozilla’s readability extension that strips the DOM elements out deterministically, leaving only the content. You can then parse it however you want. Can be done entirely client side, in runtime, with a few JavaScript libraries
"the first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time."
now for my actual thoughts, if that works for anyone in production reliably, then they should at-least try to do that.
many of my customers moved from in-house solutions because the engineering time spent on maintaining something like this alone makes it quite the undertaking, the infra needed to support something like a few hundred TPS w a browser fleet would cost more than a sub to us in the majority of cases
Conversely, many of my customers abandoned many of their SaaS solutions because they can now hand roll the 10% of the SaaS solution they need in 15 minutes.
That’s not to say all SaaS is going away. But a good hunk of the single simple solution products are going to suffer
Correction: is not going to, is already suffering.
touché
the remaining 10% accounts for the other 10,000% of the development time
i see you too are an experienced dev
Sure. The first 70% is a piece of cake. Get the last 5% right.
Readability is far from ideal. check this out: https://github.com/promptware/readweb
It's a custom pipeline I made to address its shortcomings
Funny I just did that today to replace Jina with Readability. It's surprisingly good.
Hi! Congrats on the launch, I gained LOTS of great insights from your comments, particualrly the bits about diffrentiation in a crowded market.
I am building something in agentic automation space, currently it's still under development, but would love to know your journey of ideating -> building -> getting the first customer -> iterating -> and presumably getting into YC.
Am still relatively new to this (19 lol) so I got a long way, but would really appreciate any insights :)
Cheers! Wish you luck with Context.
i've yapped quite a bit in this thread already about my journey, but happy to provide any (hopefully useful) insights, you can DM me on X, mention this comment please: https://x.com/mynameisyahia
I like the clarity, tone, and readability of your webpage. Also your FAQ is refreshing
> When Should I talk to sales? > Talk to sales if you need high-volume pricing beyond 2M credits/month, custom rate limits, SSO / SAML, SCIM provisioning, an uptime SLA, annual invoicing, an MSA / DPA, or a dedicated support channel. Reach us at hello@context.dev or through the contact page.
Would that this were the norm everywhere, rather than (say) a sales rep from Datadog scraping my phone number from who knows where to ask about my company's needs after I sign up for a free account on a whim :)
happy you noticed that, i put quite a bit of love into the ui, i've found engineers care alot about polish & feel of the webapps they use, even for an api product
i'm an engineer by trade, and always hated things like forcing a sales call, or having hidden credit multipliers, i tried to build this with the same ethos i like for my own dependencies (shoutout axiom.co)
Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
hi, no worries at all, it came off perfectly fine
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
What major edge cases have you found so far for the browser scraping method, its really complex like dealing with Auth, or pages that use pop ups, content blockers . etc...
if it needs auth, we don't do any type of logins at the moment, still figuring out how to do it without attracting the wrong type of customer (someone doing shady stuff)
popups / cookie banners are quite easy to get through with enough heuristics for 99%+ of cases
content-blockers are the same thing as the auth scenario, hard to figure out where to draw the line between building something people want and doing something shady
Is "residential proxy" a euphemism for botnets or is there a difference? Genuinely curious.
- worst is "extension" based traffic routing where your provider is essentially tricking end users into routing traffic through - opt-in p2p networks can be described as botnets sometimes, i try to avoid these but not as bad - good enough dISP proxies can be classified as residential quality, work quite well, although people have argued w me on their quality - managed business/enterprise residential gateways are ideal, because businesses opt in and allow the traffic - few other options
Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful.
> Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON API that bypasses almost all bot protections — including Turnstile. I didn't include the ability, but it bypassed the most advanced ChatGPT + Turnstile. Built with self-improving Claude Code agents that rewrite their own instructions until fresh agents consistently succeed.
> Once connected to a page, it intercepts every byte of network traffic — then actively drives the page to surface endpoints that only fire on interaction. It types into forms, clicks buttons, scrolls, triggers modals, paginates, submits searches, and walks through multi-step flows, watching what each action produces on the wire. Every request gets captured with its method, headers, payload shape, and response, then classified by transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub). The result is a complete map of the site's real API surface — including the hidden endpoints that only exist behind a click — turned into typed proxy routes you can curl.
[0] https://github.com/adam-s/intercept
Fascinating, this is definitely a path i was thinking about for a bit but decided against doing it primarily since it changes the business fundamentally
You are smart!
Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted.
> We maintain a caching layer and avoid hammering websites.
This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.
100% yes, i found out about that company recently too, it's a tempting idea for sure
Why not much traction, was the benefit more to websites than businesses/scrapers?
it counts as "reverse engineering" a website which is much worse for the website owner imo, then again im biased
FYI Your docs don't respect system-mode choice until you click into light/dark mode then back to system.
They just show up in dark mode if there's no local storage setting set yet.
is that a problem/annoying? i thought it would be fine
It's clearly a bug. The UI defaults to "system preference" but then gives you dark mode anyway, even if your system preference is light mode.
To get back to actually using system preference, you have to choose light/dark then choose system preference again.
fixed & deployed, thanks for flagging.
How did you find your differentiation in a highly commoditized space? It's probably one of the most crowded spaces.
Even within YC, there are many competitors that do pretty much the same thing:
- Firecrawl
- BrowserUse
- Browserbase
- CloudCruise
- NotteLabs
- Intuned
- Expand.ai
- Reworkd
And then you have the extremely well-funded web retrieval players like Parallel and Exa.
How do you differentiate to all these?
Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Context respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
the reality is that the space is huge and expanding
many that you mentioned serve different needs and it tends to not be zero sum to everyone's surprise (including mine at first)
e.g.
browseruse, browserbase = browser automation exa, parallel = search for ai (primarily)
we differentiate by prioritizing the infrastructure use-case, we encourage people to build products & features on top of our APIs and build it accordingly
some direct differentiators are: no hidden credit multipliers, world class brand data, and an api that just works
we're trying to be good samaritans on the web, and cache data whenever possible so we can avoid overloading a website, it's the main reason we don't offer "login with ai" features even though it's trivial at this point, it's quite hard to figure out how to avoid shady behavior while building something people want, i don't have a good answer yet and neither do many others
lastly, to reiterate, lots of our customers migrate from alternatives, but lots also just use multiple providers
how's this different from offerings from cloudflare, firecrawl, tavily, exa?
firecrawl doesn't have a search index, we're probably closest to them at the moment tavily, exa, parallel, you = search index for ai cloudflare's browser offerings are not the best in terms of quality, at-least from our testing + anecdotal reports
anything specific you want me to compare across these companies? to outsiders they look the same but it's actually quite a diverse lineup you've asked about
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up
Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be legible to agents. Some are already adapting but many haven’t yet. Context feels like an important part of that transition
Yahia is a great builder. His pace of expansion has been impressive, excited to see where he takes Context.
oh wow, happy to see you here, we didn't post this anywhere so its nice to see a customer find us so quickly, thank you for the kind words.
brand data is a shockingly hard problem to get right
Unclear what difference exists against Firecrawl - their team has been shipping great features extremely quickly lately, and their core offerings have become really good.
I am interested in KnifeGeek though - looking for a good OTF (ultratech?)
There's alot of overlap at the moment, but we're diverging quite quickly.
If you want differences as of right now
- 1 credit = 1 scrape, no hidden credit multiplier - we have world class brand data - we're focused primarily on the infra use-case, rather than gtm & everything else - anecdotally, customers have seen their error rates drop quite dramatically
In general it's a huge space, firecrawl is a wonderful company, it's fun to compete with them, planning to add more things soon which should make the differences clear
Interesting - I'll be sure to benchmark it at some point. We've found the best results come from blending providers depending on the task anyways.
Thanks for the quick response - and always happy to see more competition in the space. Best of luck with future features!
please do, and if you see anything off let me know, we've yet to lose a single "bakeoff", i normally ask customers to just have cc/codex run it so its somewhat unbiased
on the 2nd point, most industries are not zero-sum, and many of customers use multiple data providers in any case, so agree with you there
thank you!
oh also on knifegeek, it'll be live again this weekend when i find a moment to fix the bug causing it to crash, i collect knives/watches so it was a super fun project to work on
Congrats Yahia, cool to see all the progress and momentum behind what you all are building.
thank you!
Does this respect llms.txt and robots.txt, or have you found it more effective if agents see what humans see?
we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time
Hey Yahia, I recently saw your startup on the YC registry and found it to be one of the most promising of the batch, so good luck with this! Do you mind if I ask how you managed to get your first users on Brand.dev?
hey, happy you like it, to be honest we had no real secret, it takes a ton of time and effort to earn developer trust
i got my first customer after 1 month, and then just worked like hell to make him happy even though it was a very large company paying me $99/mo at the time, they're still a customer too
since then, i just locked in on every customer until they were thrilled, and once we hit 50, word of mouth started to kick in, now we're at 300+ and get a referral every other day which is awesome
i think our community appreciates effort, and when someone goes the extra mile to make sure your problems are solved, you tend to remember that. it doesn't scale super well, but you'd be surprised how far it'll take you
Excellent way to run a business.
> Websites can opt out of our service, and we respect these requests and add them to our block list.
I.e: "robots.txt already exists and is commonplace but if we actually followed it our solution wouldn't work so all websites on the planet need to do something specific to opt-out our ddos." lmao
Really cool idea! I had something similar a while back. Maybe figure out a way to use MCP so the client can connect directly out of the box
ty, it's here! https://docs.context.dev/install-mcp
This seems useful, in 2010, when AI could not make this possible in a single prompt. Now I don't see the need for this.
i too would have had this view before i started the business, however it's actually really hard, additionally teams migrate to us from in-house solutions all the time
of course, you're a salesman
maybe i'm just telling the truth: https://www.context.dev/customers
Awesome! Been great watching this product improve so quickly, can't wait for what's next :)
thank you, i try to ship updates often: https://docs.context.dev/changelog
This seems like a responsibly designed service, but I find it a little odd and baffling that we need such intermediaries for the average hobbyist / small project to reliably access sets of content published on the internet.
Wasn’t this the point of the web?
i think about this too, perhaps you're right, if (big if) the web was designed with the end goal in place, however it was a series of ~interesting~ mistakes & decisions that led to where we are today
the same could be said of most cybersecurity companies too
Remember when the Semantic Web attempted to make the entire web machine-readable?
https://en.wikipedia.org/wiki/Semantic_Web
you just found my end goal :)
love the design... congrats
will let my designer know, it's a dev-focused product, and we're all so finely tuned to avoid slop, so design & feel was really important
new frontier models do this already
im not sure the models do this, rather the tooling around them, however web search/extraction by the model providers is ridiculously expensive and quite slow, so going into production it makes sense to use a provider (like us)
Great, another thing I have to block server side. Reminds me of the image leech protections that had to be in place because bandwidth was expensive. History doesn’t repeat but rhymes as they say.
Maybe AI-service-blocklist-as-a-service could be a YC company.
Better business model would be some sort of micropayment setup - allow humans, but make scrapers pay a hundredth of a cent for access.
Is this similar? https://blog.cloudflare.com/monetization-gateway/
Yep. It will be interesting to see if things move that way or not. AI scraping might get annoying but a lot of sites (eg e-commerce) have obvious incentives to allow AIs to scrape them for free (or even pay AI companies to prioritize them, like they already do with Google).
So basically web scraping as a service with an API on top?
yes but we do alot more that may not be clear at first glance, things like brand data, and for scraping handling pdf, ocr, docx, ppt, xlsx automatically
shipping a bunch of new things soon which should make it clearer, but as of today yeah
fuck you too.
Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies?
AKA unusable for high value data.
you're welcome to try it on our demo page (no signup needed), should handle everything just fine, yes we don't mention ips on the homepage
also, while it might seem expensive, we're cheaper than every other option out there, because there's no hidden credit multipliers. every single customer who uses us halved their bill + error rate
lessgooo! context dev ftw!
thank you, im not sure who this is though, but please email me if i can help in any way
Nice, @grok how does it compare to Cloudflare that also provides a REST endpoint for structured markdown data and screenshots?
im not sure if grok is on here, i think that's an X thing
but if you were to ask me, we're more fully featured than cloudflare, and anecdotally a ton more reliable in terms of error rates. back when it was brand.dev, i actually tried to use cloudflare's apis and it was quite unreliable, so we built our own stack instead