Kickstarter is full of projects like this where every possible shortcut is taken to get to market. I’ve had some good success with a few Kickstarter projects but I’ve been very selective about which projects I support. More often than not I can identify when a team is in over their heads or think they’re just going to figure out the details later, after the money arrives.
For a period of time it was popular for the industrial designers I knew to try to launch their own Kickstarters. Their belief was that engineering was a commodity that they could hire out to the lowest bidder after they got the money. The product design and marketing (their specialty) was the real value. All of their projects either failed or cost them more money than they brought in because engineering was harder than they thought.
I think we’re in for another round of this now that LLMs give the impression that the software and firmware parts are basically free. All of those project ideas people had previously that were shelved because software is hard are getting another look from people who think they’re just going to prompt Claude until the product looks like it works.
I think you're right. And it's going to be loads of fun to watch.
Not go say there haven't also been very good coders who weren't outsourcing anything, who still got out over their skis with stuff they promised on Kickstarter. I worked on Star Citizen and saw the lure of inflating project scope, responding to the vox populi, go to someone's head in realtime. Where they could still at some point conceivably have done what they had promised if they could just resist promising more stuff.
I find it odd that industrial designers wouldn't have a firmer grasp on what was involved in shipping a product than coders do, since code seems much more prone to mission creep than a physical product would be. But I totally agree that if you're used to outsourcing the build phase of whatever you do, AI is going to be the ultimate mirage.
Is there more context to this? I'm assuming Ben is experimenting and demonstrating the danger of vibe circuit designing? Mostly because I know he has a ton of experience and I'd expect him to not make this mistake (also seems like he told the AI why it was wrong)
I'm not sure, it was posted on HN a couple weeks ago with the same title as the text in his tweet. I'd guess he was experimenting and trying to show the dangers, like you suggested.
People make these mistakes too. Several times in my high school shop class kids shorted out 9V batteries trying to build circuits because they didn't understand how electronics work. At no point did our teacher stop them from doing so - on at least one occasion I unplugged one from a breadboard before it got too toasty to handle (and I was/am an electronics nublet). Similarly there was also a lot of hand-wringing about the Gemini pizza glue in a world where people do wacky stuff like cook fish in a dishwasher or defrost chicken overnight on the counter or put cooked steak on the same plate it was on when raw just a few minutes prior.
LLMs are just surfacing the fact that assessing and managing risk is an acquired, difficult-to-learn skill. Most people don't know what they don't know and fail to think about what might happen if they do something (correctly or otherwise) before they do it, let alone what they'd do if it goes wrong.
> Several times in my high school shop class kids shorted out 9V batteries trying to build circuits because they didn't understand how electronics work. At no point did our teacher stop them from doing so
Yes, and that's okay because the classroom is a learning environment. However, LLMs don't learn; a model that releases the magic smoke in this session will be happy to release it all over again next time.
> LLMs are just surfacing the fact that assessing and managing risk is an acquired, difficult-to-learn skill.
Which makes the problem worse, not better. If risk management is a difficult skill, then that means we can't extrapolate from 'easy' demonstrations of said skill to argue that an LLM is generally safe for more sensitive tasks.
Overall, it seems like LLMs have a long tail of failures. Even while their mean or median performance is good, they seem exponentially more likely than a similarly-competent human to advise something like `rm -rf /`. This is a deeply unintuitive behaviour, precisely because our 'human-like' intuition is engaged with resepct to the average/median skill.
Well said, but I'd add that LLMs are also surfacing the fact that there's a swathe of people out there who will treat the machines as more trustworthy than humans by default, and don't believe they need to do any assessment or risk management in the first place.
People are just lazy. It’s got nothing to do with LLMs having more trust because they’re a machine because most people would happily trust their friend over an expert. They’d trust the first blog post they find online over an expert. Most people are just too lazy and not skilled enough to perform independent review.
And to be fair to those people, coming to topics with a research mindset is genuinely hard and time consuming. So I can’t actually blame people for being lazy.
All LLMs do is provide an even easier way to “research”. But it’s not like people were disbelieving random Facebook posts, online scams, and word-of-mouth before LLMs.
As right as this may be, it elides the crucial difference between asking LLMs and all the other methods of asking questions you enumerated. The difference is not between the quality of information you might get from a friend or a blog versus an LLM. The difference is the centralization and feeding of the same poor quality information to massive numbers of people at scale. At least whatever bonkers theory someone "researches" on their own is going to be a heterodox set of ideas, with a limited blast radius. Even a major search engine up-ranking a site devoted to, like, how horse dewormers can cure covid, doesn't present it as if that link is the answer to how to cure covid, right? LLMs have a pernicious combination of sounding authoritative while speaking gibberish. Their real skill is not in surfacing the truth from a mass of data, it's in presenting a set of assertions as truth in a way that might satisfy the maximum number of people with limited curiosity, and in establishing an artificial sense of trust. That's why LLMs are likely the most demonic thing ever made by man. They are machines built to lie, tell half-truths, obfuscate and flatter at the same time. Doesn't that sound enough like every religion's warning about the devil?
But nothing has changed there. People have been posting intelligent-sounding gibberish on social media and blogs for years before LLMs.
The problem with centralisation isn’t that it gobbles up data. It’s that it allows those weights to be dictated by a small few who might choose to skew the model more favourably to the messaging they’ve want to promote.
And this is a genuine concern. But it’s also not a new problem either. We already have that problem with new broadcasters, newspaper publications, social media ethics teams, and so on and so forth.
The new problem LLMs bring to human interaction isn’t any of the issues described above. It’s with LLMs replacing human contact in situations where you need something with a conscience to step in.
For example, conversations leading to AI promoting negative thoughts from people with mental health problems because the chat history starts to overwhelm the context window, resulting in the system prompt doing a poorer job of weighting the conversation away from dangerous topics like suicide.
This isn’t to say that the points which you’ve addressed aren’t real problems that exist. They definitely do exist. But they’ve also always existed, even before GPT was invented. We’ve just never properly addressed those problems because:
either there’s no incentive to. If you are powerful enough to control the narrative then why would you use that power to turn the narrative against you?
…or there simply isn’t a good way of solving that problem. eg I might hate stupid conspiracy theories, but censoring research is a much worse alternative. So we just have to allow nutters to share their dumb ideas in the hope that enough legitimate research is published, and enough people are sensible enough to read it, that the nutters don’t have any meaningful impact on society.
The AI is being sold as an expert, not a student. These are categorically different things.
The mistake in the post is one that can be avoided by taking a single class at a community college. No PhD required, not even a B.S., not even an electricians certificate.
So I don't get your point. You're comparing a person in a learning environment to the equivalent of a person claiming to have a PhD in electrical engineering. A student letting the magic smoke escape from a basic circuit is a learnable experience (a memorable one that has high impact), especially when done in a learning environment where an expert can ensure more dangerous mistakes are less likely or non existent. But the same action from a PhD educated engineer would make you reasonably question their qualifications. Yes, humans make mistakes but if you follow the AI's instructions and light things on fire you get sued. If you follow the engineer's instructions and set things on fire then that engineer gets fired likely loses their license.
Lawyers are getting in trouble because they use AI and submit fabricated citations about fabricated cases as precedent. A bunch of charges were recently thrown out in Wisconsin because of this, and it's not the first time such behavior has made the news.
The real analog here would be an electronics teacher leading his students to create a circuit that caught fire. If you’re confidently giving faulty information to people that don’t know any better, you’re not teaching them.
I am sure this is true. On the flip side, as someone who is addicted to learning, I've been finding LLMs to be amazing at feeding my addiction. :)
Some recent examples:
* foreign languages ("explain the difference between these two words that have the same English translation", "here's a photo of a mock German exam paper and here is my written answer - mark it & show how I could have done better")
* domains that I'm familiar with but might not know the exact commands off the top of my head (troubleshooting some ARP weirdness across a bunch of OSX/Linux/Windows boxes on an Omada network)
* learning basic skills in a new domain ("I'm building this thing out of 4mm mild steel - how do I go about choosing the right type of threading tap?", "what's the difference between Type B and Type F RCCB?")
Many of these can be easily answered with a web search, but the ability to ask follow-up questions has been a game changer.
I'd love to hear from other addicts - are there areas where LLMs have really accelerated your learning?
Hah, yesterday I was discussing solar panels and moving shadows. I would have wasted money buying a commercial solar panel if I didn’t have this chat.
Learned a lot on how it works, to the point I’m confident that I can go the DIY route and spend my money in AliExpress buying components instead.
Why not ask a pro solar panel installer instead? I live in an apartment, of course they would say it’s not possible to place a solar panel on my terrace. I don’t believe in things not being possible.
But I had two semesters of electronics/robotics in my CS undergrad and I know to not to trust the LLM blindly and verify.
I'm of a similar mind but I think you also need to be careful. I find that people are more willing to believe a chatbot than a search result simply due to the way the information is presented. But if you're thinking "but search results can be wrong too!" then that's exactly my point. The problem is quite similar to people "doing their own research". I'm sure conspiracy theorists do a lot of reading, a lot of searching, and all that cargo cult research stuff. But I say cargo cult because it has all the form of research but none of the substance. That doesn't mean using LLMs is exclusive cargo cult learning but it is also easy to fall into a trap of that, and I'd argue easier than it is to fall into cargo cult learning by searching, which is easier to fall into cargo cult learning than by reading books, which is easier than being in a university lecture. Doesn't mean the tools are bad, but that it's easy to fool ourselves.
Basically if you can't differentiate how your typical conspiracy theorist isn't researching then you're at greater risk. It's worth thinking about that question, as they do do a lot of reading, thinking, and looking things up. It's more subtle, right?
FWIW, a thing I find LLMs really useful for is learning the vernacular of fields I'm unfamiliar or less familiar with. It is especially helpful when searches fail due to overloaded words (and let's be honest, Google's self elected lobotomy), but it is more a launching point. Though this still has the conspiracy problem as it is easy to self-reinforce a belief and not considering the alternatives. Follow-up questions are nice and can really help sifting through large amounts of information, but they certainly have a preference to narrow the view. I think this makes learning feel faster and more direct but have also taught (at the university level) I think it is important to learn all the boring stuff too. That stuff may not be important "now" but a well organized course means that that stuff is going to be important "soon" and "now" is the best time to learn it. No different than how musicians need to practice boring scales and patterns, athletes need to do drills and not just learn by competing (or "simulated" computations), or how children learn to write by boringly writing shapes over and over. I find the LLMs like to avoid the boring parts.
I agree, I always ask to know more if I don’t get it or it’s a new subject. But I think we’re in the minority, it’s easier to just accept the answer and move on, it requires very little effort compared to trying to understand and retain.
Just because a calculator will only ever be used by a subset of the population to type 80085 and giggle, doesn't mean it can't also be used for complex calculations.
AI is a tool that can accelerate learning, or severely inhibit it. I do think the tooling is going to continue to make it easier and easier to get good output without knowing what you're doing, though.
> Just because a calculator will only ever be used by a subset of the population
I'm not sure what your argument is here. I think everyone knows this but also recognizes that the vast majority of people are not using calculators in that way. The vast majority of people are using calculators to replace calculation.
I'll give an example. I tell people I tip by: round the decimal, divide by 10, multiply by 2. Nearly every time I say that people tell me it is too difficult. This includes people with PhD STEM educations...
That's a very strong claim. I don't think people expect their circuits to ignite, LLM instruction or not. But I'd expect learning from a book or dedicated website would be less likely for that to occur. (Even accounting for bad manufacturing)
You're biased because you're not considering that by definition the student is inexperienced. Unknown unknowns. Tons of people don't know very basic things (why would they?) like circuits with capacitors bring dangerous when the power is off.
Why are you defending there LLM? Would you be as nice to a person? I'd expect not because these threads tend to point out a person's idiocy. I'm not sure why we give greater leeway to the machine. I'm not sure why we forgive them as if they are a student learning but someone posting similar instructions on a blog gets (rightfully) thrashed. That blog writer is almost never claiming PhD expertise
I agree that LLMs can greatly aid in learning. But I also think they can greatly hinder learning. I'm not sure why anyone thinks it's any different than when people got access to the internet. We gave people access to all the information in the world and people "do their own research" and end up making egregious errors because they don't know how to research (naively think it's "searching for information"), what questions to ask, or how to interrogate data (and much more). Instead we've ended up with lots of conspiratorial thinking. Now a sycophantic search engine is going to fix that? I'm unconvinced. Mostly because we can observe the result.
> We gave people access to all the information in the world and people "do their own research" and end up making egregious errors because they don't know how to research (naively think it's "searching for information"), what questions to ask, or how to interrogate data (and much more).
You pin pointed a major problem with education, indeed. Personally, I think 3 crucial courses should be taught in school to mitigate that: 1) rational thinking 2) learning how to learn 3) learning how to do a research.
I think so too, but I also think this is part of the failure of math and science education. That is exactly what those topics are. But many courses will focus on the facts and not the substance. Instead of linking Feynman's Cargo Cult Science, which also broaches this I'll link this one instead[0] as I think it better illustrates what I'm saying.
In Norway we eat plenty of salmon which is quite raw or even raw (in sushi). It has to be frozen and thawed first, to kill parasites.
A friend that studied fish production did recommend not eating salmon though and eating trout instead (ørret in Norwegian). Based on scientific evidence difference is pretty small (15% fish not surviving for salmon vs 12% for trout). But rainbow trout does have more DHA per kg.
The difference is that LLMs pretend to be experts on all things. The high school shop kids aren’t under the impression they can build a smart toaster or whatever.
The people doing these kickstarters are outsourcing the work because they can’t do it themselves. If they use an LLM, they don’t know what to look for or even ask for, which is how they get these problems where the production backend uses shared credentials and has no access control.
The LLM got it to “working” state, but the people operating it didn’t understand what it was doing. They just prompt until it looks like it works and then ship it.
This took me a while(I'm slow), but I think GP is saying: "I've seen enough of (expressions) thinking ideas is the key when engineering was; with everyone snorting LLMs, we'll see that replicating in software world" but nicely.
THAT makes sense. Engineering was never cheap nor non-differentiating if normalized by man-hours, only when it was USD normalized. If a large enough number of people were to get the same FALSE impression that software and firmware parts are now basically free and non-differentiating commodities, then there will be tons of spectacular failures in software world in coming years. There has already been early previews of those here.
I’m following exactly, but the parent commenter is off on a tangent unrelated to the topic.
We’re not taking about the parent commenter, we’re talking about unskilled Kickstarter operators making decisions. Not a skilled programmer using an LLM.
> they'd rather vibe code themselves than trust an unproven engineering firm
You could cut the statement short here, and it would still be a reasonable position to take these days.
LLMs are still complex, sharp tools - despite their simple appearance and proteststions of both biggest fans and haters alike, the dominating factor for effectiveness of an LLM tool on a problem is still whether or not you're holding it wrong.
LLMs definitely write more robust code than most. They don't take shortcuts or resort to ugly hacks. They have no problem writing tedious guards against edge cases that humans brush off. They also keep comments up to date and obsess over tests.
> They don't take shortcuts or resort to ugly hacks.
That hasn't, universally, been my experience. Sometimes the code is fine. Sometimes it is functional, but organized poorly, or does things in a very unusual way that is hard to understand. And sometimes it produces code that might work sometimes but misses important edge cases and isn't robust at all, or does things in an incredibly slow way.
> They have no problem writing tedious guards against edge cases that humans brush off.
The flip side of that is that instead of coming up with a good design that doesn't have as many edge cases, it will write verbose code that handles many different cases in similar, but not quite the same ways.
> They also keep comments up to date and obsess over tests.
Sure but they will often make comments or tests that aren't actually useful, or modify tests to succeed instead of fixing the code.
One significant danger of LLMs is that the quality of the output is higly variable and unpredictable.
That's ok, if you have someone knowledgeable reviewing and correcting it. But if you blindly trust it, because it produced decent results a few times, you'll probably be sorry.
> Sure but they will often make comments or tests that aren't actually useful, or modify tests to succeed instead of fixing the code.
I've been deeply concerned that there's been a rise of TDD. I thought we already went through this and saw its failure. But we're back to we're people cannot differentiate "tests aren't enough" from "tests are useless". The amount of faith people put into tests is astounding. Especially when they aren't spending much time analyzing the tests and understanding their coverage.
> They don't take shortcuts or resort to ugly hacks.
My experience is quite different
> They have no problem writing tedious guards against edge cases that humans brush off.
Ditto.
I have a hard time getting them to write small and flexible functions. Even with explicit instructions about how a specific routine should be done. (Really easy to produce in bash scripts as they seem to avoid using functions, but so do people, but most people suck at bash) IME they're fixated on the end goal and do not grasp the larger context (which is often implicit though I still find difficulty when I'm highly explicit. Which at that point it's usually faster to write myself)
It also makes me question context. Are humans not doing this because they don't think about it or because we've been training people to ignore things? How often do we hear "I just care that it works?" I've only heard that phrase from those that also love to talk about minimum viable products because... frankly, who is not concerned if it works? That's always been a disagreement about what is sufficient. Only very junior people believe in perfection. It's why we have sayings like "there's no solution more permanent than a temporary fix that works". It's the same people who believe tests are proof of correctness rather than a bound on correctness. The same people who read that last sentence and think I'm suggesting to not write tests or believe tests are useless.
I'd be concerned with the LLM operator quite a bit because of this. Subtle things are important when instructing LLMs. Subtle things in the prompts can wildly change the output
> They don't take shortcuts or resort to ugly hacks.
In my experience that is all they do, and you constantly have to fight them to get the quality up, and then fight again to prevent regressions on every change.
What? Yes they do take shortcuts and hacks. They change the tests case to make it pass. As the context gets longer it is less reliable at following earlier instructions. I literally had Claude hallucinate nonexistent APIs and then admitted “You caught me! I didn’t actually know, let me do a web search” and then after the web search it still mixes deprecated patterns and APIs against instructions.
I’m much more worried about the reliability of software produced by LLMs.
I had 5.3-Codex take two tries to satisfy a linter on Typescript type definitions.
It gave up, removed the code it had written directly accessing the correct property, and replaced it with a new function that did a BFS to walk through every single field in the API response object while applying a regex "looksLikeHttpsUrl" and hoping the first valid URL that had https:// would be the correct key to use.
On the contrary, the shift from pretraining driving most gains to RL driving most gains is pressuring these models resort to new hacks and shortcuts that are increasingly novel and disturbing!
The discourse around LLMs has created this notion that humans are not lazy and write perfect code. They get compared to an ideal programmer instead of real devs.
LLM's at best asymptotically approach a human doing the same task. They are trained on the best and the worst. Nothing they output deserves faith other than what can be proven beyond a shadow of a doubt with your own eyes and tooling. I'll say the same thing to anyone vibe coding that I'd say to programmatically illiterate. Trust this only insofar as you can prove it works, and you can stay ahead of the machine. Dabble if you want, but to use something safely enough to rely on, you need to be 10% smarter than it is.
> LLMs definitely write more robust code than most.
I’ve been using Opus 4.6 and GPT-Codex-5.3 daily and I see plenty of hacks and problems all day long.
I think this is missing the point. The code in this product might be robust in the sense that it follows documentation and does things without hacks, but the things it’s doing are a mismatch for what is needed in the situation.
It might be perfectly structured code, but it uses hardcoded shared credentials.
A skilled operator could have directed it to do the right things and implement something secure, but an unskilled operator doesn’t even know how to specify the right requirements.
I would love to see the prompt history. Always curious how much human intervention/guidance is necessary for this type of work because when I read the article I come away thinking I prompt Claude and it comes out with all these results. For example, "So Claude went after the app instead. Grabbed the Android APK, decompiled it with jadx." All by itself or the author had to suggest and fiddle with bits?
Why would anyone watch a live stream of someone else poking a computer into completing a task? It’s barely more interesting than having someone tell you about a dream they had.
One of my earlier experiences with codex was actually reverse engineering, far before it was good at actual coding.
It was able to decompile a react native app (Tesla Android app), and fully trace from a "How does X UI display?" down to a network call with a payload for me to intercept.
Granted it did it by splitting the binary into a billion txt files with each one being a single function and then rging through it, but it worked.
I heard about this and tried quite a bit to reverse engineer a decompiled binary from a big game to find struct/schema information but could never get anything useful.
I love that it shows you the thought process that to a Senior or Staff level person would be expected to know in their approach to a reverse engineering problem with no documentation
I wholeheartedly disagree. Running strings and a decompiler explicitly written for that language is kinda the first thing that comes to mind. Trying hundreds of random ways to talk to it before even doing any real reverse engineering is just a waste of compute. You're never going to guess the JSON to send to it or the random bytes. But it's not my tokens getting spent on it so meh
How about complaining that brain waves get sent to a server?
I'm a neuroscientist, so I'm not going to say that the EEG data is mind reading or anything, but as a precedent, non privacy of brain data is very bad.
This is the easiest signal though, on basically any account. You can see the time that communication happens, and the times when it doesn't.
For example a while back I wanted to map out my sleep cycle and I found a tool that charts your browser history over a 24 hour period, and it mapped almost perfectly to my sleep / wake periods.
But keep in mind that other less obvious data sources can often lead to similar issues. For example phone accelerometer data can be used to precisely locate someone driving in a car in a city by comparing it with a street map.
In the context of the military even just inferring a comprehensive map of which people are on which shift and when they change might be considered a threat.
How useful could something like this be for research? I'm not a neuroscientist so I have no clue, but it seems like the only justification I can think of..
The general idea of an EEG system that posts data to a network?
Very, but there are already tons of them at lots of different price, quality, openness levels. A lot of manufacturers have their own protocols; there are also quasi/standards like Lab Streaming Layer for connecting to a hodgepodge of devices.
This particular data?
Probably not so useful. While it’s easy to get something out of an EEG set, it takes some work to get good quality data that’s not riddled with noise (mains hum, muscle artifacts, blinks, etc). Plus, brain waves on their own aren’t particularly interesting—-it’s seeing how they change in response to some external or internal event that tells us about the brain.
Not a neuroscientist either but I would imagine that raw data without personal information would not be useful for much. I can imagine that it would be quite valuable if accompanied with personal data plus user reports about how they slept each night, what they dreamed about if anything, whether it was positive dreams or nightmares etc. And I think quite a few people wouldn’t mind sharing all of that in the name of science, but in this case they don’t seem to have even tried to ask.
Millions of people voluntarily use Gmail which gives a lot more useful data than EEG output to DHS et al without a warrant under FAA702. What makes you think people who “have nothing to hide” would care about publishing their EEG data?
Can someone explain the other iot devices using the same broker? I tried cross referencing the feature list, information about the user base, kickstarter origin and flutter app with some search results and I’m pretty sure that I found the company and product in question. But they don’t (publicly) produce iot devices? Sooo I’m wondering if different companies are streaming their data into a shared sink and why they would do that?
This guy bought an internet connected sleep mask so it's not surprising that it was collecting all kinds of data, or that it was doing it insecurely (everyone should expect IoT anything to be a security nightmare) so to me the surprising thing about this is that the company actually bothered to worry about saving bandwidth/power and went through the trouble of using MQTT. Probably not the best choice, and they didn't bother to do it securely, but I'm genuinely impressed that they even tried to be efficient while sucking up people's personal data.
Ok, obviously unethical to do it, but this sounds like you've got the power to create some sci-fi shared dreaming device, where you can read people's brainwaves and send signals to other people's masks based on those signals. Or send signals to everyone at the same time and suddenly people all across the world experience some change in their dream simultaneously.
Like, don't actually do it, but I feel like there's inspiration for a sci-fi novel or short story there.
Ok so obviously this is a security disaster. But also ... is there a hackable consumer EEG device that gets useful data and is as comfortable as a sleep mask (and presumably you're not slathering electrode every time you put on your sleep mask)?
Cuz once the thing can't phone home, that sounds pretty cool.
This feels like a reason to buy the device to me? I would want to block all of the data going to the cloud and would only want operations happening locally. But the MQTT broadcast then allows me to create a local only integration in Home Assistant with all of the data.
What's the real risk profile? Robbers can see you are asleep instead of waiting until you aren't home?
I have not implemented MQTT automations myself, but it's there a way to encrypt them? That could be a nice to have
Sounds like you cannot control which MQTT endpoint it is headed to? It just goes to the server of the device. Assuming you could modify the firmware, you could program it to send to a local MQTT.
I'm the founder of neurotech/sleeptech company https://affectablesleep.com, and this post shows the major issue with current wellness device regulation.
I believe there was some good that came from last months decision to be more open to what apps and data can say without going through huge regulatory processes (though because we apply auditory stimulation, this doesn't apply to us), however, there should be at least regulatory requirements for data security.
We've developed all of our algorithms and processing to happen on device, which is required anyway due to the latency which would result from bluetooth connections, but even the data sent to the server is all encrypted. I'd think that would be the basics. How do you trust a company with monitoring, and apparently providing stimulation, if they don't take these simple steps?
Author doesn’t spell out why they are not naming them, but my guess is they are trying to not promote the product to malicious actors who would be interested in the sleep data of others.
I guess that’s not a huge problem, though, since all users are presumably at least anonymous.
Agents are excellent for reverse engineering. I was also recently working on a BLE reverse engineering exercise and followed a similar path. I ran into lots of headaches with BLE on my Mac and tabled it.
Author or others who know, did you perform this on Linux? I imagine it lacks the tooling challenges I had with BLE on MacOS.
What sort of tools did it use? I suppose the path mine took may have been a dead end. The Tuya app (I was also using decompiled APK) downloads the BLE definitions on-demand and weren't embedded in the app. It wanted me to capture traffic on a device with the app. I punted but plan to resume with an emulator setup or real device connected with adb.
"The ZZZ mask is an intelligent sleep mask — it allows you to sleep less while sleeping deeper. That’s the premise — but really it is a paradigm breaking computer that allows full automation and control over the sleep process, including access to dreamtime."
or if this is another scifi variation of the same theme, with some dev like embellishments.
the shared MQTT credentials pattern is unfortunately super common in budget IoT. seen the exact same thing in smart plugs and air quality sensors. the frustrating part is per-device auth is not even hard to set up, mosquitto supports client certs and topic ACLs with minimal config. manufacturers skip it because per-device key provisioning adds a step to the assembly line and nobody wants to think about key management. so they hardcode one set of creds and hope nobody runs strings on the binary.
Why is it that almost all ODB-II dongles you buy have the same MAC address? If you buy two, one for each car, your app can never tell which car you're connected to.
They all come with Bluetooth certified logos, as well.
The ones that don't reuse everything cost like $120, not $15.
The narrator in the article acts as a third person observer and identifies "Claude" as the active hacker. So assuming the (unidentified) company that sells/manages the product wants to prosecute a CFAA violation, who do they go after? Was Claude the one responsible for all of the hacking?
What do you mean? IANAL, but Claude doesn't just "wake up" (whatever that means) and decide to reverse engineering/hack stuff, so if this is a CFAA violation the person who prompted Claude is indeed responsible. At best, one could argue that the company producing Claude is partially responsible because it didn't prevent people from using it to reverse engineer stuff, but there's no way Claude is "responsible for all of the hacking", regardless of how many times the blog posts says "Claude did X".
> I recently got a smart sleep mask from Kickstarter. I was not expecting to end up with the ability to read strangers' brainwaves and send them electric impulses in their sleep. But here we are.
One of the best opening paragraphs in a SF novel that I’ve ever read.
I have deployed open MQTT to the world for quick prototypes on non personal (and healthcare) data. Once my cloud provider told me to stop because they didn’t like it, that could be used for relay DDOS attacks.
I would not trust the sleep mask company even if they somehow manage to have some authentication and authorisation on their MQTT.
It did get me thinking - maybe there should be IoTS devices, where the S stands for Security. A commitment to updates for a certain amount of time, the source code in escrow to be released when updates/support ceases, probably other things I'm not thinking of.
As an aside, it seems cool that the bar to reverse engineering has lowered from all the LLMs. Maybe we'll get to take full control of many of these "smart" devices that require proprietary/spyware apps and use them in a fully private way. There's no excuse that any such apps solely to interact with devices locally need to connect to the internet, like dishwasher.
I discovered a very similar vulnerability in Mysa smart thermostats a year ago, also involving MQTT, and also allowing me to view and control anyone's thermostat anywhere in the world:
https://news.ycombinator.com/item?id=43392991
Also discovered during reverse-engineering of the devices’ communications protocols.
Yes. An excerpt from my initial email to Mysa's security contact…
> I stumbled upon these vulnerabilities on one of the coldest days of this winter in Vancouver. An attacker using them could have disabled all Mysa-connected heaters in the America/Vancouver timezone in the middle of the night. That would include the heat in the room where my 7-month-old son sleeps.
This smells like bullshit to me, although I am admittedly not experienced with Claude.
I find it difficult to believe that a sleep mask exists with the features listed: "EEG brain monitoring, electrical muscle stimulation around the eyes, vibration, heating, audio." while also being something you can strap to your face and comfortably sleep in, with battery capacity sufficient for several hours of sleep.
I also wonder how Claude probed bluetooth. Does Claude have access to bluetooth interface? Why? Perhaps it wrote a secondary program then ran that, but the article describes it as Claude probing directly.
I'm also skeptical of Claude's ability to make accurate reverse-engineered bluetooth protocol. This is at least a little more of an LLM-appropriate task, but I suspect that there was a lot of chaff also produced that the article writer separated from the wheat.
If any of this happened at all. No hardware mentioned, no company, no actual protocol description published, no library provided.
It makes a nice vague futuristic cyperpunk story, but there's no meat on those bones.
This isn't to the level of the OP, but I just asked Claude "Are there any interesting Bluetooth devices in my vicinity which aren't actually mine or ones I am connected to?" and it downloaded a tool called `blueutil` and identified a variety of things.
When I complained that the results were boring, it installed a Python package called 'bleak', found a set of LED lights (which I assumed are my daughter's) and tried to control them. It said the signal was too weak and got me to move around the house, whereupon it connected to them, figured out the protocol, and actually changed the lights while I was sat on her bed - where I am right now. Now I have a new party trick when she gets home! I had no idea they were Bluetooth controlled, nor clearly without any security at all.
A lot of BLE peripherals are very easy to probe. And there are libraries available for most popular languages that allow you to connect to a peripheral and poke at any exposed internals with little effort.
As for the reverse engineering, the author claims that all it took was dumping the strings from the Dart binary to see what was being sent to the bluetooth device. It's plausible, and I would give them the benefit of the doubt here.
Claude could access anything on your device, including system or third party commands for network or signal processing - it may even have their manuals/sites/man pages in the training set. It’s remarkably good at figuring things out, and you can watch the reasoning output. There are mcp tools for reverse engineering that can give it even higher level abilities (ghidra is a popular one).
Yesterday I watched it try and work around some filesystem permission restrictions, it tried a lot of things I would never have thought of, and it was eventually successful. I was kinda goading it though.
Found that in seconds. EEG, electrical stimulation, heat, audio, etc. Claims a 20 hour battery.
As to the Claude interactions, like others I am suspicious and it seems overly idealized and simplified. Claude can't search for BT devices, but you could hook it up with an MCP that does that. You can hook it up with a decompiler MCP. And on and on. But it's more involved than this story details.
That appears to be more than a centimeter thick, and not particularly flexible. It's more like ski goggles than a sleep mask.
So yeah, a product exists that claims to be a sleep mask with these features. Maybe someone could even sleep while wearing that thing, as long as they sleep on their back and don't move around too much. I remain skeptical that it actually does the things it claims and has the battery life it claims. This is kickstarter after all. Regardless, this would qualify as the device in question for the article. Or at least inspiration for it.
Without evidence such as wireshark logs, programs, protocol documentation, I'm not convinced that any of this actually _happened_.
Claude, or any good agent, doesn't need MCP to do things. As long as it has access to a shell it can craft any command that it needs to fulfill its prompt.
There are no shell commands to do what is described. I could get Claude to interact with BLE devices, but it did it by writing and running various helper applications, for instance using the Bleak library. So I guess not an MCP per se.
Not really? I did something similar for a different device recently. It can make files and has access to bash. It's perfectly capable of installing packages and writing small scripts basically entirely autonomously. No MCP needed.
I was originally going to ask something similar, but from a different angle.
These blog posts now making the rounds on HN are the usual reverse engineering stories, but made a lot more compelling simply because they involve using AI.
Never mind that the AI part isn't doing any heavy lifting and probably just as tedious as not using AI in the first place. I am confused why the author mentions it so prominently. Past authors would not have been so dramatic and just waved their hands that they had some trial and error before finding out how the app is built. The focus would have been on the lack of auth and the funny stuff they did before reporting it to the devs.
It's disappointing to see. It doesn't take much work to configure a MQTT server to require client certificates for all connections. It does require an extra step in provisioning to give each device a client certificate. But for a commercial product, it's inexcusably negligent.
Then there's hardening your peripheral and central device/app against the kinds of spoofing attacks that are described in this blog post.
If your peripheral and central device can securely [0] store key material, then (in addition to the standard security features that come with the Bluetooth protocol) one may implement mutual authentication between the central and peripheral devices and, optionally, encryption of the data that is transmitted across that connection.
Then, as long as your peripheral and central devices are programmed to only ever respond when presented with signatures that can be verified by a trusted public key, the spoofing and probing demonstrated here simply won't work (unless somebody reverse engineers the app running on the central device to change its behaviour after the signature verification has been performed).
To protect against that, you'd have to introduce server-mediated authorisation. On Android, that would require things like the Play Integrity API and app signatures. Then, if the server verifies that the instance of the app running on the central device is unmodified, it can issue a token that the central device can send to the peripheral for verification in addition to the signatures from the previous step.
Alternatively, you could also have the server generate the actual command frames that the central device sends to the peripheral. The server would provide the raw command frame and the command frame signed with its own key, which can be verified by the peripheral.
I guess I got a bit carried away here. Certainly, not every peripheral needs that level of security. But, into which category this device falls, I'm not sure. On the one hand, it's not a security device, like an electronic door lock. And on the other hand, it's a very personal peripheral with some unusual capabilities like the electrical muscle stimulation gizmo and the room occupancy sensor.
[0]: Like with the Android KeyStore and whichever HSMs are used in microcontrollers, so that keys can't be extracted by just dumping strings from a binary.
Interesting project. Here's a thought which I've always had in the back of my mind, ever since I saw something similar in an episode of Buck Rogers (70s-80s)! Many people struggle with falling asleep due to persistent beta waves; natural theta predominance is needed but often delayed. Imagine an "INEXPENSIVE" smart sleep mask that facilitates sleep onset by inducing brain wave transitions from beta (wakeful, high-frequency) to alpha (8-13 Hz, relaxed) and then theta (4-8 Hz, stage 1 light sleep) via non-invasive stimulation. A solution could be a comfortable eye mask with integrated headphones (unintrusive) and EEG sensors.
It could use binaural beats or similar audio stimulation to "inject" alpha/theta frequencies externally, guiding the brain to a tipping point for abrupt sleep onset. Sensors would detect current waves; app-controlled audio ramps from alpha-inducing beats to theta, ensuring natural predominance. If it could be designed, it could accelerate sleep transition, improve quality, non-pharmacological.
Yeah, I'm sure that technology has existed for decades. Common folks just not allowed to know about it. It's "for our own good!" sarcastically speaking :(
> Claude ran strings on the binary and this was the most productive step of the whole session.
After $150 in tokens, inflating GPU prices by 10%, spending $550 of VC money, and increasing the earth's temperature by 0.2 degC, claude did what a 16 year old that read two blog posts about reverse engineering would do.
> For obvious reasons, I am not naming the product/company here, but have reached out to inform them about the issue.
Coward. The only way to challenge this garbage is "Name and Shame". Light a fire under their asses. That fire can encourage them to do right, and as a warning to all other companies.
Doesn't disclosing this to the world at the same time as you disclose it to the company immediately send hundreds of black hats to their terminals to see how much chaos they can create before the company implements a fix?
Perhaps the author is not a coward, but is giving the company time to respond and commit to a fix for the benefit of other owners who could suffer harm.
So your plan is to let the blackhats in the know attack user devices, rather than send out a large warning to "Quit using immediately"?
If we applied this similar analogy to a e.coli infection of foods, your recommendation amounts to "If we say the company name, the company would be shamed and lose money and people might abuse the food".
People need to know this device is NOT SAFE on your network, paired to your phone, or anything. And that requires direct and public notification.
I did consider naming, but they were very responsive to the disclosure and I was not entirely familiar with potential legal implications of doing so. (For what it's worth, it is not Luuna)
It's good that they were responsive in the disclosure, but it's still a mark of sloppiness that this was done in the first place, and I'd like to know so I can avoid them.
Even if naming and shaming doesn't work, I sure want to know so I can always avoid them for myself and my family. Thanks for the call-out and the educated guess.
Is this some kind of joke? Claude hallucinated everything, including capacity of device to accurately measure EGG of brain waves and hallucinated the process of decoding APK to some paranoidal user who has posted his conspiracy level AI hallucinations “finds” to his blog post and everyone is like “Yeah, Claude can do this”. Is everyone here insane? I am insane?
You have no evidence of that, and it seems very unlikely unless you're intentionally wildly assuming the craziest possible scenario, as if you're paranoid or insane.
You do realize the user can see the tool calls running and check their real, actual output, during this process, right?
You do realize that there are several sleep masks on Kickstarter that actually have these features, right?
Well, in the end user agreement there are usually clauses that forbids it. It's tolerated in some geographies for interoperability, research and infosec, but you agreed on ToS already.
Kickstarter is full of projects like this where every possible shortcut is taken to get to market. I’ve had some good success with a few Kickstarter projects but I’ve been very selective about which projects I support. More often than not I can identify when a team is in over their heads or think they’re just going to figure out the details later, after the money arrives.
For a period of time it was popular for the industrial designers I knew to try to launch their own Kickstarters. Their belief was that engineering was a commodity that they could hire out to the lowest bidder after they got the money. The product design and marketing (their specialty) was the real value. All of their projects either failed or cost them more money than they brought in because engineering was harder than they thought.
I think we’re in for another round of this now that LLMs give the impression that the software and firmware parts are basically free. All of those project ideas people had previously that were shelved because software is hard are getting another look from people who think they’re just going to prompt Claude until the product looks like it works.
I think you're right. And it's going to be loads of fun to watch.
Not go say there haven't also been very good coders who weren't outsourcing anything, who still got out over their skis with stuff they promised on Kickstarter. I worked on Star Citizen and saw the lure of inflating project scope, responding to the vox populi, go to someone's head in realtime. Where they could still at some point conceivably have done what they had promised if they could just resist promising more stuff.
I find it odd that industrial designers wouldn't have a firmer grasp on what was involved in shipping a product than coders do, since code seems much more prone to mission creep than a physical product would be. But I totally agree that if you're used to outsourcing the build phase of whatever you do, AI is going to be the ultimate mirage.
At this point, I trust LLMs to come up with something more secure than the cheapest engineering firm for hire.
"Anyone else out there vibe circuit-building?"
https://xcancel.com/beneater/status/2012988790709928305
Is there more context to this? I'm assuming Ben is experimenting and demonstrating the danger of vibe circuit designing? Mostly because I know he has a ton of experience and I'd expect him to not make this mistake (also seems like he told the AI why it was wrong)
I'm not sure, it was posted on HN a couple weeks ago with the same title as the text in his tweet. I'd guess he was experimenting and trying to show the dangers, like you suggested.
In https://quesma.com/blog/nano-banana-pro-intelligence-with-to... (Nov 2025) we had an illustrative diagram of using Nano Banana Pro to create a circuit diagram.
People make these mistakes too. Several times in my high school shop class kids shorted out 9V batteries trying to build circuits because they didn't understand how electronics work. At no point did our teacher stop them from doing so - on at least one occasion I unplugged one from a breadboard before it got too toasty to handle (and I was/am an electronics nublet). Similarly there was also a lot of hand-wringing about the Gemini pizza glue in a world where people do wacky stuff like cook fish in a dishwasher or defrost chicken overnight on the counter or put cooked steak on the same plate it was on when raw just a few minutes prior.
LLMs are just surfacing the fact that assessing and managing risk is an acquired, difficult-to-learn skill. Most people don't know what they don't know and fail to think about what might happen if they do something (correctly or otherwise) before they do it, let alone what they'd do if it goes wrong.
> Several times in my high school shop class kids shorted out 9V batteries trying to build circuits because they didn't understand how electronics work. At no point did our teacher stop them from doing so
Yes, and that's okay because the classroom is a learning environment. However, LLMs don't learn; a model that releases the magic smoke in this session will be happy to release it all over again next time.
> LLMs are just surfacing the fact that assessing and managing risk is an acquired, difficult-to-learn skill.
Which makes the problem worse, not better. If risk management is a difficult skill, then that means we can't extrapolate from 'easy' demonstrations of said skill to argue that an LLM is generally safe for more sensitive tasks.
Overall, it seems like LLMs have a long tail of failures. Even while their mean or median performance is good, they seem exponentially more likely than a similarly-competent human to advise something like `rm -rf /`. This is a deeply unintuitive behaviour, precisely because our 'human-like' intuition is engaged with resepct to the average/median skill.
Well said, but I'd add that LLMs are also surfacing the fact that there's a swathe of people out there who will treat the machines as more trustworthy than humans by default, and don't believe they need to do any assessment or risk management in the first place.
People are just lazy. It’s got nothing to do with LLMs having more trust because they’re a machine because most people would happily trust their friend over an expert. They’d trust the first blog post they find online over an expert. Most people are just too lazy and not skilled enough to perform independent review.
And to be fair to those people, coming to topics with a research mindset is genuinely hard and time consuming. So I can’t actually blame people for being lazy.
All LLMs do is provide an even easier way to “research”. But it’s not like people were disbelieving random Facebook posts, online scams, and word-of-mouth before LLMs.
As right as this may be, it elides the crucial difference between asking LLMs and all the other methods of asking questions you enumerated. The difference is not between the quality of information you might get from a friend or a blog versus an LLM. The difference is the centralization and feeding of the same poor quality information to massive numbers of people at scale. At least whatever bonkers theory someone "researches" on their own is going to be a heterodox set of ideas, with a limited blast radius. Even a major search engine up-ranking a site devoted to, like, how horse dewormers can cure covid, doesn't present it as if that link is the answer to how to cure covid, right? LLMs have a pernicious combination of sounding authoritative while speaking gibberish. Their real skill is not in surfacing the truth from a mass of data, it's in presenting a set of assertions as truth in a way that might satisfy the maximum number of people with limited curiosity, and in establishing an artificial sense of trust. That's why LLMs are likely the most demonic thing ever made by man. They are machines built to lie, tell half-truths, obfuscate and flatter at the same time. Doesn't that sound enough like every religion's warning about the devil?
But nothing has changed there. People have been posting intelligent-sounding gibberish on social media and blogs for years before LLMs.
The problem with centralisation isn’t that it gobbles up data. It’s that it allows those weights to be dictated by a small few who might choose to skew the model more favourably to the messaging they’ve want to promote.
And this is a genuine concern. But it’s also not a new problem either. We already have that problem with new broadcasters, newspaper publications, social media ethics teams, and so on and so forth.
The new problem LLMs bring to human interaction isn’t any of the issues described above. It’s with LLMs replacing human contact in situations where you need something with a conscience to step in.
For example, conversations leading to AI promoting negative thoughts from people with mental health problems because the chat history starts to overwhelm the context window, resulting in the system prompt doing a poorer job of weighting the conversation away from dangerous topics like suicide.
This isn’t to say that the points which you’ve addressed aren’t real problems that exist. They definitely do exist. But they’ve also always existed, even before GPT was invented. We’ve just never properly addressed those problems because:
either there’s no incentive to. If you are powerful enough to control the narrative then why would you use that power to turn the narrative against you?
…or there simply isn’t a good way of solving that problem. eg I might hate stupid conspiracy theories, but censoring research is a much worse alternative. So we just have to allow nutters to share their dumb ideas in the hope that enough legitimate research is published, and enough people are sensible enough to read it, that the nutters don’t have any meaningful impact on society.
What's your point?
The AI is being sold as an expert, not a student. These are categorically different things.
The mistake in the post is one that can be avoided by taking a single class at a community college. No PhD required, not even a B.S., not even an electricians certificate.
So I don't get your point. You're comparing a person in a learning environment to the equivalent of a person claiming to have a PhD in electrical engineering. A student letting the magic smoke escape from a basic circuit is a learnable experience (a memorable one that has high impact), especially when done in a learning environment where an expert can ensure more dangerous mistakes are less likely or non existent. But the same action from a PhD educated engineer would make you reasonably question their qualifications. Yes, humans make mistakes but if you follow the AI's instructions and light things on fire you get sued. If you follow the engineer's instructions and set things on fire then that engineer gets fired likely loses their license.
So what is your point?
No one thinks their breadboard wont catch on fire because an AI agent told them it wouldn’t. Its never been easier to learn because of these agents.
Lawyers are getting in trouble because they use AI and submit fabricated citations about fabricated cases as precedent. A bunch of charges were recently thrown out in Wisconsin because of this, and it's not the first time such behavior has made the news.
https://www.wpr.org/news/judge-sanctions-kenosha-county-da-a...
AI is indeed being understood to be an expert that replaces human judgement, and people are being hurt because of it.
The real analog here would be an electronics teacher leading his students to create a circuit that caught fire. If you’re confidently giving faulty information to people that don’t know any better, you’re not teaching them.
In my experience people don’t use LLMs to learn but to circumvent learning.
I am sure this is true. On the flip side, as someone who is addicted to learning, I've been finding LLMs to be amazing at feeding my addiction. :)
Some recent examples:
* foreign languages ("explain the difference between these two words that have the same English translation", "here's a photo of a mock German exam paper and here is my written answer - mark it & show how I could have done better")
* domains that I'm familiar with but might not know the exact commands off the top of my head (troubleshooting some ARP weirdness across a bunch of OSX/Linux/Windows boxes on an Omada network)
* learning basic skills in a new domain ("I'm building this thing out of 4mm mild steel - how do I go about choosing the right type of threading tap?", "what's the difference between Type B and Type F RCCB?")
Many of these can be easily answered with a web search, but the ability to ask follow-up questions has been a game changer.
I'd love to hear from other addicts - are there areas where LLMs have really accelerated your learning?
Hah, yesterday I was discussing solar panels and moving shadows. I would have wasted money buying a commercial solar panel if I didn’t have this chat.
Learned a lot on how it works, to the point I’m confident that I can go the DIY route and spend my money in AliExpress buying components instead.
Why not ask a pro solar panel installer instead? I live in an apartment, of course they would say it’s not possible to place a solar panel on my terrace. I don’t believe in things not being possible.
But I had two semesters of electronics/robotics in my CS undergrad and I know to not to trust the LLM blindly and verify.
I'm of a similar mind but I think you also need to be careful. I find that people are more willing to believe a chatbot than a search result simply due to the way the information is presented. But if you're thinking "but search results can be wrong too!" then that's exactly my point. The problem is quite similar to people "doing their own research". I'm sure conspiracy theorists do a lot of reading, a lot of searching, and all that cargo cult research stuff. But I say cargo cult because it has all the form of research but none of the substance. That doesn't mean using LLMs is exclusive cargo cult learning but it is also easy to fall into a trap of that, and I'd argue easier than it is to fall into cargo cult learning by searching, which is easier to fall into cargo cult learning than by reading books, which is easier than being in a university lecture. Doesn't mean the tools are bad, but that it's easy to fool ourselves.
Basically if you can't differentiate how your typical conspiracy theorist isn't researching then you're at greater risk. It's worth thinking about that question, as they do do a lot of reading, thinking, and looking things up. It's more subtle, right?
FWIW, a thing I find LLMs really useful for is learning the vernacular of fields I'm unfamiliar or less familiar with. It is especially helpful when searches fail due to overloaded words (and let's be honest, Google's self elected lobotomy), but it is more a launching point. Though this still has the conspiracy problem as it is easy to self-reinforce a belief and not considering the alternatives. Follow-up questions are nice and can really help sifting through large amounts of information, but they certainly have a preference to narrow the view. I think this makes learning feel faster and more direct but have also taught (at the university level) I think it is important to learn all the boring stuff too. That stuff may not be important "now" but a well organized course means that that stuff is going to be important "soon" and "now" is the best time to learn it. No different than how musicians need to practice boring scales and patterns, athletes need to do drills and not just learn by competing (or "simulated" computations), or how children learn to write by boringly writing shapes over and over. I find the LLMs like to avoid the boring parts.
I agree, I always ask to know more if I don’t get it or it’s a new subject. But I think we’re in the minority, it’s easier to just accept the answer and move on, it requires very little effort compared to trying to understand and retain.
Just because a calculator will only ever be used by a subset of the population to type 80085 and giggle, doesn't mean it can't also be used for complex calculations.
AI is a tool that can accelerate learning, or severely inhibit it. I do think the tooling is going to continue to make it easier and easier to get good output without knowing what you're doing, though.
I'll give an example. I tell people I tip by: round the decimal, divide by 10, multiply by 2. Nearly every time I say that people tell me it is too difficult. This includes people with PhD STEM educations...
Exactly. I like to say that learning feels like frustration. If I'm right, then LLM's eliminate precisely the thing that is learning.
That's a very strong claim. I don't think people expect their circuits to ignite, LLM instruction or not. But I'd expect learning from a book or dedicated website would be less likely for that to occur. (Even accounting for bad manufacturing)
You're biased because you're not considering that by definition the student is inexperienced. Unknown unknowns. Tons of people don't know very basic things (why would they?) like circuits with capacitors bring dangerous when the power is off.
Why are you defending there LLM? Would you be as nice to a person? I'd expect not because these threads tend to point out a person's idiocy. I'm not sure why we give greater leeway to the machine. I'm not sure why we forgive them as if they are a student learning but someone posting similar instructions on a blog gets (rightfully) thrashed. That blog writer is almost never claiming PhD expertise
I agree that LLMs can greatly aid in learning. But I also think they can greatly hinder learning. I'm not sure why anyone thinks it's any different than when people got access to the internet. We gave people access to all the information in the world and people "do their own research" and end up making egregious errors because they don't know how to research (naively think it's "searching for information"), what questions to ask, or how to interrogate data (and much more). Instead we've ended up with lots of conspiratorial thinking. Now a sycophantic search engine is going to fix that? I'm unconvinced. Mostly because we can observe the result.
> We gave people access to all the information in the world and people "do their own research" and end up making egregious errors because they don't know how to research (naively think it's "searching for information"), what questions to ask, or how to interrogate data (and much more).
You pin pointed a major problem with education, indeed. Personally, I think 3 crucial courses should be taught in school to mitigate that: 1) rational thinking 2) learning how to learn 3) learning how to do a research.
I think so too, but I also think this is part of the failure of math and science education. That is exactly what those topics are. But many courses will focus on the facts and not the substance. Instead of linking Feynman's Cargo Cult Science, which also broaches this I'll link this one instead[0] as I think it better illustrates what I'm saying.
[0] https://enlightenedidiot.net/random/feynman-on-brazilian-edu...
The result of more people getting into electronics because it’s easier now?
When reading I suggest trying to interpret what the person wrote rather than just ignore it. I'd probably start by taking the advice of your username
What’s wrong with dishwasher salmon?
It doesn't get hot enough to be a safe cooking method
https://youtu.be/dSwzau2_KF8?t=1108
In Norway we eat plenty of salmon which is quite raw or even raw (in sushi). It has to be frozen and thawed first, to kill parasites.
A friend that studied fish production did recommend not eating salmon though and eating trout instead (ørret in Norwegian). Based on scientific evidence difference is pretty small (15% fish not surviving for salmon vs 12% for trout). But rainbow trout does have more DHA per kg.
The difference is that LLMs pretend to be experts on all things. The high school shop kids aren’t under the impression they can build a smart toaster or whatever.
Ha ha, I said this before when Ben's post came up earlier, but, yes I am. And so far it has been a positive experience.
The cheapest engineering firms you hire are also using LLMs.
The operator is still a factor.
Yeah, but they’ll add another layer of complexity over doing it yourself
The people doing these kickstarters are outsourcing the work because they can’t do it themselves. If they use an LLM, they don’t know what to look for or even ask for, which is how they get these problems where the production backend uses shared credentials and has no access control.
The LLM got it to “working” state, but the people operating it didn’t understand what it was doing. They just prompt until it looks like it works and then ship it.
You're still not following.
The parents are saying they'd rather vibe code themselves than trust an unproven engineering firm that does(n't) vibe code.
This took me a while(I'm slow), but I think GP is saying: "I've seen enough of (expressions) thinking ideas is the key when engineering was; with everyone snorting LLMs, we'll see that replicating in software world" but nicely.
THAT makes sense. Engineering was never cheap nor non-differentiating if normalized by man-hours, only when it was USD normalized. If a large enough number of people were to get the same FALSE impression that software and firmware parts are now basically free and non-differentiating commodities, then there will be tons of spectacular failures in software world in coming years. There has already been early previews of those here.
I’m following exactly, but the parent commenter is off on a tangent unrelated to the topic.
We’re not taking about the parent commenter, we’re talking about unskilled Kickstarter operators making decisions. Not a skilled programmer using an LLM.
> they'd rather vibe code themselves than trust an unproven engineering firm
You could cut the statement short here, and it would still be a reasonable position to take these days.
LLMs are still complex, sharp tools - despite their simple appearance and proteststions of both biggest fans and haters alike, the dominating factor for effectiveness of an LLM tool on a problem is still whether or not you're holding it wrong.
I forgot about that Jobs/Apple reference!
Paraphasing, LLMs are great (bad) tools for the right (wrong) job...
in the right hands,
at the right time,
in the right place...
Well I certainly hope that’s true to some degree or I’m out of a job
I don’t know, you can get a lot of nice engineering done in a Shenzhen dark alley.
LLMs definitely write more robust code than most. They don't take shortcuts or resort to ugly hacks. They have no problem writing tedious guards against edge cases that humans brush off. They also keep comments up to date and obsess over tests.
> They don't take shortcuts or resort to ugly hacks.
That hasn't, universally, been my experience. Sometimes the code is fine. Sometimes it is functional, but organized poorly, or does things in a very unusual way that is hard to understand. And sometimes it produces code that might work sometimes but misses important edge cases and isn't robust at all, or does things in an incredibly slow way.
> They have no problem writing tedious guards against edge cases that humans brush off.
The flip side of that is that instead of coming up with a good design that doesn't have as many edge cases, it will write verbose code that handles many different cases in similar, but not quite the same ways.
> They also keep comments up to date and obsess over tests.
Sure but they will often make comments or tests that aren't actually useful, or modify tests to succeed instead of fixing the code.
One significant danger of LLMs is that the quality of the output is higly variable and unpredictable.
That's ok, if you have someone knowledgeable reviewing and correcting it. But if you blindly trust it, because it produced decent results a few times, you'll probably be sorry.
I have a hard time getting them to write small and flexible functions. Even with explicit instructions about how a specific routine should be done. (Really easy to produce in bash scripts as they seem to avoid using functions, but so do people, but most people suck at bash) IME they're fixated on the end goal and do not grasp the larger context (which is often implicit though I still find difficulty when I'm highly explicit. Which at that point it's usually faster to write myself)
It also makes me question context. Are humans not doing this because they don't think about it or because we've been training people to ignore things? How often do we hear "I just care that it works?" I've only heard that phrase from those that also love to talk about minimum viable products because... frankly, who is not concerned if it works? That's always been a disagreement about what is sufficient. Only very junior people believe in perfection. It's why we have sayings like "there's no solution more permanent than a temporary fix that works". It's the same people who believe tests are proof of correctness rather than a bound on correctness. The same people who read that last sentence and think I'm suggesting to not write tests or believe tests are useless.
I'd be concerned with the LLM operator quite a bit because of this. Subtle things are important when instructing LLMs. Subtle things in the prompts can wildly change the output
> They don't take shortcuts or resort to ugly hacks.
In my experience that is all they do, and you constantly have to fight them to get the quality up, and then fight again to prevent regressions on every change.
They absolutely take shortcuts and resort to ugly hacks.
My AGENTS.md is filled with specific lines to counter all of them that come up.
What? Yes they do take shortcuts and hacks. They change the tests case to make it pass. As the context gets longer it is less reliable at following earlier instructions. I literally had Claude hallucinate nonexistent APIs and then admitted “You caught me! I didn’t actually know, let me do a web search” and then after the web search it still mixes deprecated patterns and APIs against instructions.
I’m much more worried about the reliability of software produced by LLMs.
I had 5.3-Codex take two tries to satisfy a linter on Typescript type definitions.
It gave up, removed the code it had written directly accessing the correct property, and replaced it with a new function that did a BFS to walk through every single field in the API response object while applying a regex "looksLikeHttpsUrl" and hoping the first valid URL that had https:// would be the correct key to use.
On the contrary, the shift from pretraining driving most gains to RL driving most gains is pressuring these models resort to new hacks and shortcuts that are increasingly novel and disturbing!
Interesting and completely wrong statement, what gave you this impression?
I know right. I kept waiting for a sarcasm tag at the end
right and wrong don't exist when evaluating subjective quantifiers
The discourse around LLMs has created this notion that humans are not lazy and write perfect code. They get compared to an ideal programmer instead of real devs.
This. The hacks, shortcuts and bugs I saw in our product code after i got hired, were stuff every LLM would tell you not to do.
LLM's at best asymptotically approach a human doing the same task. They are trained on the best and the worst. Nothing they output deserves faith other than what can be proven beyond a shadow of a doubt with your own eyes and tooling. I'll say the same thing to anyone vibe coding that I'd say to programmatically illiterate. Trust this only insofar as you can prove it works, and you can stay ahead of the machine. Dabble if you want, but to use something safely enough to rely on, you need to be 10% smarter than it is.
Amen. On top of that, especially now, with good prompting you can get closer to that better than you think.
> LLMs definitely write more robust code than most.
I’ve been using Opus 4.6 and GPT-Codex-5.3 daily and I see plenty of hacks and problems all day long.
I think this is missing the point. The code in this product might be robust in the sense that it follows documentation and does things without hacks, but the things it’s doing are a mismatch for what is needed in the situation.
It might be perfectly structured code, but it uses hardcoded shared credentials.
A skilled operator could have directed it to do the right things and implement something secure, but an unskilled operator doesn’t even know how to specify the right requirements.
And the cheapest engineering firm won't use LLMs as well, wherever possible?
The cheapest engineering firm will turn out to be headed up by an openclaw instance.
fun fact, LLMs come in cheapest and useless and expensive but actually does what's being asked, too.
So, will they? Probably. Can you trust the kind of LLM that you would use to do a better job than the cheapest firm? Absolutely.
this.
Oh gosh anyone who thinks LLMs make firmware free haven’t seriously tried to use it for firmware engineering then.
I would love to see the prompt history. Always curious how much human intervention/guidance is necessary for this type of work because when I read the article I come away thinking I prompt Claude and it comes out with all these results. For example, "So Claude went after the app instead. Grabbed the Android APK, decompiled it with jadx." All by itself or the author had to suggest and fiddle with bits?
Very little intervention tbh. I will try to retrieve it and post.
That's great to hear. I'd be interested to see the session. Yes, Claude Code keeps sessions in ~/.claude/projects/ by default. Thank you!
here you go: https://gist.github.com/aimihat/a206289b356cac88e2810654adf0...
By default, Claude code keeps session history (as jsonl files in ~/.claude).
It’s wasteful not to save and learn from those.
Check this out: https://github.com/kulesh/catsyphon
Really is a derth of livestreams demostrating these things. Youd think if thetes so much Unaided AI work people would stream it.
Why would anyone watch a live stream of someone else poking a computer into completing a task? It’s barely more interesting than having someone tell you about a dream they had.
Fantastic claims require fantastic Evidence.
While most comments are focused on the issue that they found, I’m more intrigued by the fact that Claude was able to reverse engineer so well.
Lowering the skills bar needed to reverse engineer at this level could have its own AI-related implications.
One of my earlier experiences with codex was actually reverse engineering, far before it was good at actual coding.
It was able to decompile a react native app (Tesla Android app), and fully trace from a "How does X UI display?" down to a network call with a payload for me to intercept.
Granted it did it by splitting the binary into a billion txt files with each one being a single function and then rging through it, but it worked.
I heard about this and tried quite a bit to reverse engineer a decompiled binary from a big game to find struct/schema information but could never get anything useful.
I love that it shows you the thought process that to a Senior or Staff level person would be expected to know in their approach to a reverse engineering problem with no documentation
Levels up the way I think about things
I wholeheartedly disagree. Running strings and a decompiler explicitly written for that language is kinda the first thing that comes to mind. Trying hundreds of random ways to talk to it before even doing any real reverse engineering is just a waste of compute. You're never going to guess the JSON to send to it or the random bytes. But it's not my tokens getting spent on it so meh
> and send them electric impulses in their sleep. So, it's like Lovense, but for dreams?
Sorry, I know it's horrible, but I couldn't resist.
> I was not expecting to end up with the ability to read strangers' brainwaves and send them electric impulses in their sleep. But here we are.
Almost out of a Phillip K Dick novel
Just what I was thinking.
China has a recent history of spying on personal data. https://www.telegraph.co.uk/news/2026/01/26/china-hacked-dow...
How about complaining that brain waves get sent to a server? I'm a neuroscientist, so I'm not going to say that the EEG data is mind reading or anything, but as a precedent, non privacy of brain data is very bad.
Non-privacy of this person is currently sleeping data is very bad as well, for different reasons.
You know, now that I'm thinking about it, I'm beginning to wonder if poor data privacy could have some negative effects.
It sounds like there was "presence in room" data as well, which could be very bad
This is the easiest signal though, on basically any account. You can see the time that communication happens, and the times when it doesn't.
For example a while back I wanted to map out my sleep cycle and I found a tool that charts your browser history over a 24 hour period, and it mapped almost perfectly to my sleep / wake periods.
Unsecured fitness monitor data revealed military guard post (IIRC) activity a while back.
Yawp. T’was Strava. https://www.theguardian.com/world/2018/jan/28/fitness-tracki...
not because you knew how much someone worked out. But because it had GPS.
True.
But keep in mind that other less obvious data sources can often lead to similar issues. For example phone accelerometer data can be used to precisely locate someone driving in a car in a city by comparing it with a street map.
In the context of the military even just inferring a comprehensive map of which people are on which shift and when they change might be considered a threat.
People will be lining up to have their brainwaves harvested because it'll be mildly easier to send emails or something similarly inane.
Corporations will be lining up to require their employees have their brainwaves harvested, so they can fire employees who aren't alert enough.
Will someone invent the equivalent of a mouse jiggler to get around this?
Porn?
They'll update the required brain state to "alert but not enjoying yourself".
You could read the alertness level from an EEG, which could be helpful to a burglar. The device with slow-wave status seems ideal.
How useful could something like this be for research? I'm not a neuroscientist so I have no clue, but it seems like the only justification I can think of..
The general idea of an EEG system that posts data to a network?
Very, but there are already tons of them at lots of different price, quality, openness levels. A lot of manufacturers have their own protocols; there are also quasi/standards like Lab Streaming Layer for connecting to a hodgepodge of devices.
This particular data?
Probably not so useful. While it’s easy to get something out of an EEG set, it takes some work to get good quality data that’s not riddled with noise (mains hum, muscle artifacts, blinks, etc). Plus, brain waves on their own aren’t particularly interesting—-it’s seeing how they change in response to some external or internal event that tells us about the brain.
Not a neuroscientist either but I would imagine that raw data without personal information would not be useful for much. I can imagine that it would be quite valuable if accompanied with personal data plus user reports about how they slept each night, what they dreamed about if anything, whether it was positive dreams or nightmares etc. And I think quite a few people wouldn’t mind sharing all of that in the name of science, but in this case they don’t seem to have even tried to ask.
What if you gonna think about your social security number 30000 times in your dreams, and someone knows the pattern? See the danger? That's evil.
I believe they use it for sleep tracking
If they're taking patient data for research without permission, they are not ethical researchers.
Is it really “without permission” if it’s from a server for which the access credentials have been deliberately published to the entire internet?
From the perspective of research ethics: it is very much without permission in that situation
If it's without the patient's permission, then yes, it is without the only permission that matters for medical ethics.
I would presume data privacy laws already have good precedent for health data?
> I would presume data privacy laws already have good precedent for health data?
Google for a list of all the exceptions to HIPPA. There are a lot of things that _seem_ like they should be covered by HIPPA but are not...
Interesting...
Only for "covered entities" under HIPAA (at least in the US)
"Broker" is right there in the title of the post.
Baby's gotta get some cash somewhere.
An MQTT Broker just mean server, that's MQTT terminology.
Dark humor is like food.
Not everybody gets it.
Here it's more Poe's law.
Maybe?
Poe's law is always described as being about parodies of extreme viewpoints, not about perceived misunderstandings of what is being commented on.
I don't think that the viewpoint that these people will sell your data in a heartbeat is extreme.
Millions of people voluntarily use Gmail which gives a lot more useful data than EEG output to DHS et al without a warrant under FAA702. What makes you think people who “have nothing to hide” would care about publishing their EEG data?
Can someone explain the other iot devices using the same broker? I tried cross referencing the feature list, information about the user base, kickstarter origin and flutter app with some search results and I’m pretty sure that I found the company and product in question. But they don’t (publicly) produce iot devices? Sooo I’m wondering if different companies are streaming their data into a shared sink and why they would do that?
They were scanning BLE so any device using that protocol in range would be picked up. Similar to seeing your neighbor's Wi-Fi router from your couch.
You should financialize this by creating a prediction market around it.
This guy bought an internet connected sleep mask so it's not surprising that it was collecting all kinds of data, or that it was doing it insecurely (everyone should expect IoT anything to be a security nightmare) so to me the surprising thing about this is that the company actually bothered to worry about saving bandwidth/power and went through the trouble of using MQTT. Probably not the best choice, and they didn't bother to do it securely, but I'm genuinely impressed that they even tried to be efficient while sucking up people's personal data.
Meanwhile streaming everyone's data, negating any benefit.
Ok, obviously unethical to do it, but this sounds like you've got the power to create some sci-fi shared dreaming device, where you can read people's brainwaves and send signals to other people's masks based on those signals. Or send signals to everyone at the same time and suddenly people all across the world experience some change in their dream simultaneously.
Like, don't actually do it, but I feel like there's inspiration for a sci-fi novel or short story there.
I feel if you're doing something that will require a Hans Zimmer soundtrack you might be the bad guy.
That’s the plot of Paprika.
Inception
The Cell
Brainstorm.
Dreamscape, 1984
Ok so obviously this is a security disaster. But also ... is there a hackable consumer EEG device that gets useful data and is as comfortable as a sleep mask (and presumably you're not slathering electrode every time you put on your sleep mask)? Cuz once the thing can't phone home, that sounds pretty cool.
This feels like a reason to buy the device to me? I would want to block all of the data going to the cloud and would only want operations happening locally. But the MQTT broadcast then allows me to create a local only integration in Home Assistant with all of the data.
What's the real risk profile? Robbers can see you are asleep instead of waiting until you aren't home?
I have not implemented MQTT automations myself, but it's there a way to encrypt them? That could be a nice to have
Sounds like you cannot control which MQTT endpoint it is headed to? It just goes to the server of the device. Assuming you could modify the firmware, you could program it to send to a local MQTT.
Simpler just update your local network dns so whatevercompany.brain.com redirect to your local 10.0.0.3 mqtt
With no encryption, this isn’t a problem.
I thought the author was going to change the hardcoded server (or override DNS) and set up his own.
I'm the founder of neurotech/sleeptech company https://affectablesleep.com, and this post shows the major issue with current wellness device regulation.
I believe there was some good that came from last months decision to be more open to what apps and data can say without going through huge regulatory processes (though because we apply auditory stimulation, this doesn't apply to us), however, there should be at least regulatory requirements for data security.
We've developed all of our algorithms and processing to happen on device, which is required anyway due to the latency which would result from bluetooth connections, but even the data sent to the server is all encrypted. I'd think that would be the basics. How do you trust a company with monitoring, and apparently providing stimulation, if they don't take these simple steps?
Name the company, hiding it is irresponsible
Author doesn’t spell out why they are not naming them, but my guess is they are trying to not promote the product to malicious actors who would be interested in the sleep data of others.
I guess that’s not a huge problem, though, since all users are presumably at least anonymous.
less sleep data, i imagine, and more the whole “send remote electrical impulses” thing
It’s probably safe to assume they are all like that.
Agents are excellent for reverse engineering. I was also recently working on a BLE reverse engineering exercise and followed a similar path. I ran into lots of headaches with BLE on my Mac and tabled it.
Author or others who know, did you perform this on Linux? I imagine it lacks the tooling challenges I had with BLE on MacOS.
It was on a MBP, didn’t run into any issues
What sort of tools did it use? I suppose the path mine took may have been a dead end. The Tuya app (I was also using decompiled APK) downloads the BLE definitions on-demand and weren't embedded in the app. It wanted me to capture traffic on a device with the app. I punted but plan to resume with an emulator setup or real device connected with adb.
huh, not sure if life imitates snark and bull https://medium.com/luminasticity/great-products-of-illuminat...
"The ZZZ mask is an intelligent sleep mask — it allows you to sleep less while sleeping deeper. That’s the premise — but really it is a paradigm breaking computer that allows full automation and control over the sleep process, including access to dreamtime."
or if this is another scifi variation of the same theme, with some dev like embellishments.
That is the premise of HypnoSpace Outlaw, a neat game about 90s internet nostalgia and scifi.
the shared MQTT credentials pattern is unfortunately super common in budget IoT. seen the exact same thing in smart plugs and air quality sensors. the frustrating part is per-device auth is not even hard to set up, mosquitto supports client certs and topic ACLs with minimal config. manufacturers skip it because per-device key provisioning adds a step to the assembly line and nobody wants to think about key management. so they hardcode one set of creds and hope nobody runs strings on the binary.
Why is it that almost all ODB-II dongles you buy have the same MAC address? If you buy two, one for each car, your app can never tell which car you're connected to.
They all come with Bluetooth certified logos, as well.
The ones that don't reuse everything cost like $120, not $15.
Well that’s a brand new sentence.
But not a beautiful sentence.
damn, this would make a cool midi controller
Really interesting read. This feels less like a security bug and more like a missing execution boundary.
I asked ChatGPT which product this could be and it came up with
https://www.kickstarter.com/projects/selepu/dreampilot-ai-gu...
Claude could not tell which one
> For obvious reasons, I am not naming the product/company here, but have reached out to inform them about the issue.
It's working as intended
The narrator in the article acts as a third person observer and identifies "Claude" as the active hacker. So assuming the (unidentified) company that sells/manages the product wants to prosecute a CFAA violation, who do they go after? Was Claude the one responsible for all of the hacking?
What do you mean? IANAL, but Claude doesn't just "wake up" (whatever that means) and decide to reverse engineering/hack stuff, so if this is a CFAA violation the person who prompted Claude is indeed responsible. At best, one could argue that the company producing Claude is partially responsible because it didn't prevent people from using it to reverse engineer stuff, but there's no way Claude is "responsible for all of the hacking", regardless of how many times the blog posts says "Claude did X".
The narrator. It doesn’t matter to the law the kind of intimate relationship you have with your tool.
> I recently got a smart sleep mask from Kickstarter. I was not expecting to end up with the ability to read strangers' brainwaves and send them electric impulses in their sleep. But here we are.
One of the best opening paragraphs in a SF novel that I’ve ever read.
Oh, wait.
Remember that the S in IoT stands for Security.
I have deployed open MQTT to the world for quick prototypes on non personal (and healthcare) data. Once my cloud provider told me to stop because they didn’t like it, that could be used for relay DDOS attacks.
I would not trust the sleep mask company even if they somehow manage to have some authentication and authorisation on their MQTT.
I don't think there is an S in IoT?..
Right - the saying indicates that IoT stuff is well known for ignoring security.
Went right over my head :)
It did get me thinking - maybe there should be IoTS devices, where the S stands for Security. A commitment to updates for a certain amount of time, the source code in escrow to be released when updates/support ceases, probably other things I'm not thinking of.
Where I work, the saying is, "The H in ABC stands for Happiness."
(Also, "We're not happy until you're not happy.")
It does work a lot better with verbal inflection.
Thank you for your astute observation. :)
Exactly
And the P in IoT stands Privacy, and the Q for quality.
The K, of course, stands for Ka-ching!
The L stands for longevity.
That's exactly what I need. A radio transmitter as close as possible to my brain when I sleep.
As an aside, it seems cool that the bar to reverse engineering has lowered from all the LLMs. Maybe we'll get to take full control of many of these "smart" devices that require proprietary/spyware apps and use them in a fully private way. There's no excuse that any such apps solely to interact with devices locally need to connect to the internet, like dishwasher.
https://www.jeffgeerling.com/blog/2025/i-wont-connect-my-dis...
>Since every device shares the same credentials and the same broker, if you can read someone's brainwaves you can also send them electric impulses.
Amazing.
OK, but can we get a teledildonics device that records all thrusts onto the Blockchain?
How is the smart sleep mask called?
Amazing to see claude's reasoning and process through reversing this
I discovered a very similar vulnerability in Mysa smart thermostats a year ago, also involving MQTT, and also allowing me to view and control anyone's thermostat anywhere in the world: https://news.ycombinator.com/item?id=43392991
Also discovered during reverse-engineering of the devices’ communications protocols.
IoT device security is an utterly shambolic mess.
That is terrifying. Messing with thermostats could be enough to kill vulnerable people.
Yes. An excerpt from my initial email to Mysa's security contact…
> I stumbled upon these vulnerabilities on one of the coldest days of this winter in Vancouver. An attacker using them could have disabled all Mysa-connected heaters in the America/Vancouver timezone in the middle of the night. That would include the heat in the room where my 7-month-old son sleeps.
I’m not super familiar with MQTT. I wonder how common this is..
MQTT is a very simple pub/sub messaging protocol.
It's used in a enormous number of IoT devices.
The "IoT gateway" service from AWS supports MQTT and a whole lot of IoT devices are tethered to this service specifically.
This smells like bullshit to me, although I am admittedly not experienced with Claude.
I find it difficult to believe that a sleep mask exists with the features listed: "EEG brain monitoring, electrical muscle stimulation around the eyes, vibration, heating, audio." while also being something you can strap to your face and comfortably sleep in, with battery capacity sufficient for several hours of sleep.
I also wonder how Claude probed bluetooth. Does Claude have access to bluetooth interface? Why? Perhaps it wrote a secondary program then ran that, but the article describes it as Claude probing directly.
I'm also skeptical of Claude's ability to make accurate reverse-engineered bluetooth protocol. This is at least a little more of an LLM-appropriate task, but I suspect that there was a lot of chaff also produced that the article writer separated from the wheat.
If any of this happened at all. No hardware mentioned, no company, no actual protocol description published, no library provided.
It makes a nice vague futuristic cyperpunk story, but there's no meat on those bones.
This isn't to the level of the OP, but I just asked Claude "Are there any interesting Bluetooth devices in my vicinity which aren't actually mine or ones I am connected to?" and it downloaded a tool called `blueutil` and identified a variety of things.
When I complained that the results were boring, it installed a Python package called 'bleak', found a set of LED lights (which I assumed are my daughter's) and tried to control them. It said the signal was too weak and got me to move around the house, whereupon it connected to them, figured out the protocol, and actually changed the lights while I was sat on her bed - where I am right now. Now I have a new party trick when she gets home! I had no idea they were Bluetooth controlled, nor clearly without any security at all.
thread with claude: https://gist.github.com/aimihat/a206289b356cac88e2810654adf0...
A lot of BLE peripherals are very easy to probe. And there are libraries available for most popular languages that allow you to connect to a peripheral and poke at any exposed internals with little effort.
As for the reverse engineering, the author claims that all it took was dumping the strings from the Dart binary to see what was being sent to the bluetooth device. It's plausible, and I would give them the benefit of the doubt here.
Claude could access anything on your device, including system or third party commands for network or signal processing - it may even have their manuals/sites/man pages in the training set. It’s remarkably good at figuring things out, and you can watch the reasoning output. There are mcp tools for reverse engineering that can give it even higher level abilities (ghidra is a popular one).
Yesterday I watched it try and work around some filesystem permission restrictions, it tried a lot of things I would never have thought of, and it was eventually successful. I was kinda goading it though.
https://www.kickstarter.com/projects/selepu/dreampilot-ai-gu...
Found that in seconds. EEG, electrical stimulation, heat, audio, etc. Claims a 20 hour battery.
As to the Claude interactions, like others I am suspicious and it seems overly idealized and simplified. Claude can't search for BT devices, but you could hook it up with an MCP that does that. You can hook it up with a decompiler MCP. And on and on. But it's more involved than this story details.
That appears to be more than a centimeter thick, and not particularly flexible. It's more like ski goggles than a sleep mask.
So yeah, a product exists that claims to be a sleep mask with these features. Maybe someone could even sleep while wearing that thing, as long as they sleep on their back and don't move around too much. I remain skeptical that it actually does the things it claims and has the battery life it claims. This is kickstarter after all. Regardless, this would qualify as the device in question for the article. Or at least inspiration for it.
Without evidence such as wireshark logs, programs, protocol documentation, I'm not convinced that any of this actually _happened_.
Claude, or any good agent, doesn't need MCP to do things. As long as it has access to a shell it can craft any command that it needs to fulfill its prompt.
There are no shell commands to do what is described. I could get Claude to interact with BLE devices, but it did it by writing and running various helper applications, for instance using the Bleak library. So I guess not an MCP per se.
Not really? I did something similar for a different device recently. It can make files and has access to bash. It's perfectly capable of installing packages and writing small scripts basically entirely autonomously. No MCP needed.
Yes, it is very lacking in details. The Claude output would have been interesting, or a few logs or protocol dumps.
The lack of detail makes me suspect the truth of most of the story.
https://news.ycombinator.com/item?id=47020069
wow! Thanks for that.
I was originally going to ask something similar, but from a different angle.
These blog posts now making the rounds on HN are the usual reverse engineering stories, but made a lot more compelling simply because they involve using AI.
Never mind that the AI part isn't doing any heavy lifting and probably just as tedious as not using AI in the first place. I am confused why the author mentions it so prominently. Past authors would not have been so dramatic and just waved their hands that they had some trial and error before finding out how the app is built. The focus would have been on the lack of auth and the funny stuff they did before reporting it to the devs.
A lot of so called "smart" devices have little or no concept of privacy or personal boundaries built into them.
Reading a blog post where Claude did all the actual work is kinda sad.
It's disappointing to see. It doesn't take much work to configure a MQTT server to require client certificates for all connections. It does require an extra step in provisioning to give each device a client certificate. But for a commercial product, it's inexcusably negligent.
Then there's hardening your peripheral and central device/app against the kinds of spoofing attacks that are described in this blog post.
If your peripheral and central device can securely [0] store key material, then (in addition to the standard security features that come with the Bluetooth protocol) one may implement mutual authentication between the central and peripheral devices and, optionally, encryption of the data that is transmitted across that connection.
Then, as long as your peripheral and central devices are programmed to only ever respond when presented with signatures that can be verified by a trusted public key, the spoofing and probing demonstrated here simply won't work (unless somebody reverse engineers the app running on the central device to change its behaviour after the signature verification has been performed).
To protect against that, you'd have to introduce server-mediated authorisation. On Android, that would require things like the Play Integrity API and app signatures. Then, if the server verifies that the instance of the app running on the central device is unmodified, it can issue a token that the central device can send to the peripheral for verification in addition to the signatures from the previous step.
Alternatively, you could also have the server generate the actual command frames that the central device sends to the peripheral. The server would provide the raw command frame and the command frame signed with its own key, which can be verified by the peripheral.
I guess I got a bit carried away here. Certainly, not every peripheral needs that level of security. But, into which category this device falls, I'm not sure. On the one hand, it's not a security device, like an electronic door lock. And on the other hand, it's a very personal peripheral with some unusual capabilities like the electrical muscle stimulation gizmo and the room occupancy sensor.
[0]: Like with the Android KeyStore and whichever HSMs are used in microcontrollers, so that keys can't be extracted by just dumping strings from a binary.
Interesting project. Here's a thought which I've always had in the back of my mind, ever since I saw something similar in an episode of Buck Rogers (70s-80s)! Many people struggle with falling asleep due to persistent beta waves; natural theta predominance is needed but often delayed. Imagine an "INEXPENSIVE" smart sleep mask that facilitates sleep onset by inducing brain wave transitions from beta (wakeful, high-frequency) to alpha (8-13 Hz, relaxed) and then theta (4-8 Hz, stage 1 light sleep) via non-invasive stimulation. A solution could be a comfortable eye mask with integrated headphones (unintrusive) and EEG sensors. It could use binaural beats or similar audio stimulation to "inject" alpha/theta frequencies externally, guiding the brain to a tipping point for abrupt sleep onset. Sensors would detect current waves; app-controlled audio ramps from alpha-inducing beats to theta, ensuring natural predominance. If it could be designed, it could accelerate sleep transition, improve quality, non-pharmacological.
So are the brain waves the cause or the effect?
Are beta waves a sign that my mind is racing and wide awake, or are they the reason?
Don't know but as AI advances, questions like that may get easier to answer.
What’s your proposed mechanism for how audio waves would induce brain waves?
No idea about audio frequencies close to hearing, but I'm pretty sure it's common to manipulate the brain with ultrasonic frequencies these days.
Yeah, I'm sure that technology has existed for decades. Common folks just not allowed to know about it. It's "for our own good!" sarcastically speaking :(
That's a toughie, but if it were me and I had the energy, I'd start by looking at the following patents:
- US20030171688A1: Mind controller - Induces alpha/theta brainwaves via audio messages. - US20070084473A1: Brain wave entrainment in sound - Modulates music for desired brain states. - US11309858: Inducing brainwaves by sound - Adjusts volume gains for specific frequencies. - US5036858A: Changing brain wave frequency - Generates binaural beats to alter waves. - US3951134: Remotely altering brain waves - Monitors and modifies via RF/EM waves. - US5306228A: Brain wave synchronizer - Uses light/sound for entrainment. - US6587729: RF hearing effect - Transmits speech via microwaves to brain. - US6488617: Desired brain state - Electromagnetic pulses for mind states. - US4858612: Microwave hearing simulation - Induces sounds in auditory cortex. - US6930235B2: EM to sound waves - Relates waves for brain influence. - EP0747080A1: Brain wave inducing - Sine waves via speaker for alpha waves. - US5954629A: Brain wave system - Feedback light stimulation. - US5954630A: FM theta sound - Superposes low frequencies for theta induction. - US5159703A: Silent subliminal - Ultrasonic carriers for brain inducement. - US6017302A: Acoustic manipulation - Subaudio pulses for nervous system control.
Who cares. I'm so tired.
the headlines these days
> Claude ran strings on the binary and this was the most productive step of the whole session.
After $150 in tokens, inflating GPU prices by 10%, spending $550 of VC money, and increasing the earth's temperature by 0.2 degC, claude did what a 16 year old that read two blog posts about reverse engineering would do.
I think the number of people who could do this in half an hour is low.
Article is saying it was the most productive step and crediting it to Claude. However it is indeed what anyone would do pretty much as a first step.
The impact of AI on environment is overblown.
cyberpunk
> For obvious reasons, I am not naming the product/company here, but have reached out to inform them about the issue.
Coward. The only way to challenge this garbage is "Name and Shame". Light a fire under their asses. That fire can encourage them to do right, and as a warning to all other companies.
My guess is this is Luuna https://www.kickstarter.com/projects/flowtimebraintag/luuna
Doesn't disclosing this to the world at the same time as you disclose it to the company immediately send hundreds of black hats to their terminals to see how much chaos they can create before the company implements a fix?
Perhaps the author is not a coward, but is giving the company time to respond and commit to a fix for the benefit of other owners who could suffer harm.
but is giving the company time to respond and commit to a fix for the benefit of other owners who could suffer harm.
If that's the case then they should have deferred this whole blog post.
It took me 30 seconds with ChatGPT by saying:
Identify the kickstarter product talked around in this blog post: (link)
To think some blackhat hasn't already did that is frankly laughable. What I did was like the lowest of low-bars these days.
Put the product name in the title & maybe it sends thousands instead of hundreds of blackhats…
We often treat doxxing the same way, prohibiting posting of easily discovered information.
So your plan is to let the blackhats in the know attack user devices, rather than send out a large warning to "Quit using immediately"?
If we applied this similar analogy to a e.coli infection of foods, your recommendation amounts to "If we say the company name, the company would be shamed and lose money and people might abuse the food".
People need to know this device is NOT SAFE on your network, paired to your phone, or anything. And that requires direct and public notification.
And ChatGPT hallucinated a misleading answer that you are confidently regurgitating.
their original message said "my guess", not ChatGPT's, talk about responsible disclosure...
I did consider naming, but they were very responsive to the disclosure and I was not entirely familiar with potential legal implications of doing so. (For what it's worth, it is not Luuna)
Please name 50 other companies it's not.
It's good that they were responsive in the disclosure, but it's still a mark of sloppiness that this was done in the first place, and I'd like to know so I can avoid them.
I don't see estim mentioned on that website, but I do see a comparison chart with 4 other competitors with similar capabilities to the one you linked.
What makes you think this is the one?
https://meta.wikimedia.org/wiki/Cunningham%27s_Law
I said a guess, not absolute.
Even if naming and shaming doesn't work, I sure want to know so I can always avoid them for myself and my family. Thanks for the call-out and the educated guess.
EEG devices can cost a lot to own personally as well.
The other side of owning equipment like this is it still could be useful for some for personal and private use.
EEG is very useful for accurate sleep tracking.
Presumably they’ll be named and shamed after they’ve been given a chance to fix things.
Is this some kind of joke? Claude hallucinated everything, including capacity of device to accurately measure EGG of brain waves and hallucinated the process of decoding APK to some paranoidal user who has posted his conspiracy level AI hallucinations “finds” to his blog post and everyone is like “Yeah, Claude can do this”. Is everyone here insane? I am insane?
Why do you think it's all hallucinated?
You have no evidence of that, and it seems very unlikely unless you're intentionally wildly assuming the craziest possible scenario, as if you're paranoid or insane.
You do realize the user can see the tool calls running and check their real, actual output, during this process, right?
You do realize that there are several sleep masks on Kickstarter that actually have these features, right?
The user has also shared the Claude transcript:
https://gist.github.com/aimihat/a206289b356cac88e2810654adf0...
Won't they sue for the reverse engineering?
On what grounds could they sue?
Well, in the end user agreement there are usually clauses that forbids it. It's tolerated in some geographies for interoperability, research and infosec, but you agreed on ToS already.
Without a brand name, how can we verify this is real?
Without any skin in the game with your username, why should we take anything you say seriously?
Interesting position in a thread about the dangers of exposing yourself to the internet.
“Ask an LLM to hack your app” should be a production-readiness step from now on.