Absolutely wild. I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box. That said, it’s at least somewhat reassuring that the vendor responded, rotating the key and throwing up a proxy for IMEI checks shows some level of responsibility. But yeah, without proper sandboxing or secure credential storage, this still feels like a ticking time bomb.
> I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box.
As someone with a lot of experience in the mobile app space, and tangentially in the IoT space, I can most definitely believe this, and I am not surprised in the slightest.
Our industry may "move fast", but we also "break things" frequently and don't have nearly the engineering rigor found in other domains.
> It was a good thing for user privacy that the keys were directly on the device
You want to think through that one again? With the OpenAI key on device it means anyone could use that key to call (and bill) OpenAI's APIs. It's absolutely not feasible to ship the OpenAI keys on device.
Hardcoded API keys and poorly secured backend endpoints are surprisingly common in mobile apps. Sort of like how common XSS/SQLi used to be in webapps. Decompiling an APK seems to be a slightly higher barrier than opening up devtools, so they get less attention.
Since debugging hardware is an even higher threshold, I would expect hardware devices this to be wildly insecure unless there are strong incentive for investing in security. Same as the "security" of the average IoT device.
Eventually someone is going to get a bill for the OpenAPI key usage. That will provide some incentive. (Incentive to just rotate the key and brick all the devices rather than fix the problem, most likely.
Had you ever heard of IKKO before this? I hadn't, and I'm at least adjacent to the hifi and audio nerd crowd.
Apple have a reputation and brand that allows them to charge premium prices.
IKKO seems, at least to me, to be effectively a disposable brand. If their reputation goes bad, their only reals costs are setting up a new website/AliExpress Store/Amazon seller account.
The IOT and embedded space is simultaneously obsessed with IP protection, fuse protecting code etc, and incapable of managing the life cycle of secrets. I worked at one company that actually did it well on-device, but neglected they had to ship their testing setup overseas including certain keys. So even if you couldn't break in to the device you could 'acquire' one of the testing devices and have at it
Indeed, brace yourselves as the floodgates holding back the poorly-developed AI crap open wide. If anyone is thinking of a career pivot, now is the time to dive into all things cybersecurity. It's going to get ugly!
If that were true we'd have no cybersecurity professionals left.
In my experience, the work is focused on weakening vulnerable areas, auditing, incident response, and similar activities. Good cybersecurity professionals even get to know the business and tailor security to fit. The "one mistake and you're fired" mentality encourages hiding mistakes and suggests poor company culture.
"One mistake can cause a breach" and "we should fire people who make the one mistake" are very different claims. The latter claim was not made.
As with plane crashes and surgical complications, we should take an approach of learning from the mistake, and putting things in place to prevent/mitigate it in the future.
I believe the thread starts with cybersecurity as a job role, although perhaps I misunderstood. In either case, I agree with your learning-based approach. Blameless postmortem and related techniques are really valuable here.
There's a difference between "cybersecurity" meaning the property of having a secure system, and "cybersecurity" as a field of human endeavour.
If your system has lots of vulnerabilities, it's not secure - you don't have cybersecurity. If your system has lots of vulnerabilities, you have a lot of cybersecurity work to do and cybersecurity money to make.
“decrypt” function just decoding base64 is almost too difficult to believe but the amount of times ive run into people that should know better think base64 is a secure string tells me otherwise
The humorous phrase “the S in IoT stands for security” can be applied to the wearable market too. I wonder if this rule applies to any market with fast release cycles, thin margins and low barriers to entry?
To be fair (or pedantic), in this post they didn't have root, so cat'ing etc/passwd would not have been possible, whereas installing a doom apk is trivial.
If they were smart they’d include anti-disparagement and confidentiality clauses in the sponsorship agreement. They aren’t, though, so maybe it’s just a pathetic attempt at bribery.
> "and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you."
Interesting, I'm assuming llms "correctly" interpret "please no china politic" type vague system prompts like this, but if someone told me that I'd just be confused - like, don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin? What does this mean? LLMs though in my experience are smarter than me at understanding imo vague language. Maybe because I'm autistic and they're not.
> Don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin?
In my mind all of these could be relevant to Chinese politics. My interpretation would be "anything one can't say openly in China". I too am curious how such a vague instruction would be interpreted as broadly as would be needed to block all politically sensitive subjects.
There is no difference to other countries. In France if you say bad things about certain groups of people then you can literally go to jail (but the censorship is directly IN the models)
You don't feel there's a difference between a State banning criticism of the State, and a State passing anti-hate speech laws to protect people from, e.g., nazis?
No, there isn't a difference. "Hate speech" has no meaning, and laws purporting to be combatting it are actively used to prevent criticism of the State (e.g. in Germany).
This is strange to me. I have no difficulty seeing the difference between hate speech and criticism of the state. Of course if someone tries to muddy the waters, they should be criticized... but that's what you're trying to do here, so you're no better than a State that does the same. Hate speech very clearly has meaning, the legal definition may change a bit of course, but in Germany the meaning is quite clear, banning expressions that incite hatred or violence against people based on race, ethnicity, religion, nationality, gender, sexual orientation, or disability. What's unclear about that?
I'm not sure what specific incident you're referring to, however I do know that if Germany was more willing to leverage the hate speech laws more strictly, the AFD would have been banned long ago. Now they're finally willing to leverage it to ban the new nazi party, which is a relief.
> I have no difficulty seeing the difference between hate speech and criticism of the state.
You have no difficulty manufacturing what you believe to be a difference (that clearly does not survive contact with reality), because you're ignorant of the world around you.
> Of course if someone tries to muddy the waters, they should be criticized
No, if someone tries to falsely claim that there's a clear and objective difference, as you are, they should be criticized.
> Hate speech very clearly has meaning
No, it very clearly does not, and the fact that you're expressing that opinion indicates that you're extremely uninformed about history. "Hate speech" wasn't even a concept that existed until the 20th century, originally only referred to race when it was defined by the ICERD, constantly changed and increased in scope, and still even today not only has no commonly agreed-upon definition, but is used to suppress relevant-to-society free speech that the State does not approve of.
If you go and ask 10 random people in your country what the definition of "hate speech" is, they will not be able to agree on a definition - anyone who has gone out and actually interacted with different groups in their country (as opposed to being isolated to a single community) knows this to be true. That by itself is factual proof that there is no consensus definition of the term.
Not that there needs to be any further elaboration than that, but...
> I'm not sure what specific incident you're referring to
Marie-Thérèse Kaiser, a German politician, posted a social media post with the text "Afghanistan refugees; Hamburg SPD mayor for 'unbureaucratic' admission; Welcome culture for gang rapes?" and was charged under German hate speech laws. You're extremely authoritarian and progressive, so you probably feel that a penalty should have been given out, but regardless of your feelings, the fact is that that was not clearly incitement to hatred or violence, and that the poster was charged for "hate speech" for making political statements about immigration.
> banning expressions that incite hatred or violence against people based on [...]. What's unclear about that?
It's very clear to anyone who has contact with reality that not only does "hatred" also have no consensus definition, but neither does "inciting", and so both of those terms can be and are interpreted in an extremely wide spread that is abused by the State.
Not only is the lack of consensus of definition of the concept of "hate speech" factual evidence that your claims about it being clear are false, but even your citation of the German legal definition contains terms that have neither consensus population definition nor objective test (legal or otherwise).
> "Hate speech" wasn't even a concept that existed until the 20th century,
And? "Capitalism" wasn't a word in any language until the 17th century. We make new words when we need them.
> originally only referred to race when it was defined by the ICERD, constantly changed and increased in scope
Turns out as we opened our eyes to our collective bigotry, we realized we were doing it in more ways than one.
> but is used to suppress relevant-to-society free speech that the State does not approve of.
Would love to you point to an example of this that isn't racist or bigoted :)
> If you go and ask 10 random people in your country what the definition of "hate speech" is, they will not be able to agree on a definition -
Great, that's why we have representative democracy and laws and dictionaries. I could ask anyone in Texas (my home state) the legally required pre-driving check that must be performed before operating a motor vehicle, every time, and I wager 90% will not even know such a lawful requirement for such a check exists, and 100% will fail to list every step required. This doesn't mean such a law doesn't exist or, if someone learns about it, then isn't clear.
Of course in my opinion more people should know about it and enforce it personally but I accept that one of the unsolved problems of liberal democracy is how to manage the massive nest of rules and regulations in a fair and equitable way. After all, almost everyone speeds.
> "Afghanistan refugees; Hamburg SPD mayor for 'unbureaucratic' admission; Welcome culture for gang rapes?" and was charged under German hate speech laws. You're extremely authoritarian and progressive, so you probably feel that a penalty should have been given out, but regardless of your feelings, the fact is that that was not clearly incitement to hatred or violence, and that the poster was charged for "hate speech" for making political statements about immigration.
Lmfao I knew there was some racist shit behind your position. It's absolutely racist to imply that Afghanistanian refugees are rapists, which is exactly what the tweet does. It makes sense that Germany would have more strict application of hate speech laws, and it makes sense to punish German politicians that swing a bit too far into "But what if one of the types of peoples were not actually totally human?" again.
> that not only does "hatred" also have no consensus definition,
Law should be decided by popular consensus? So you're an anarchist as well? Well, excellent, then we can get into the inherent moral wrongness of racism and our role to engage in direct action against racists. This probably will be sloppier than using liberal democracy and well defined hate speech laws but I prefer it, as do you apparently. In the end, the people who know what hate speech is and abhor it far outnumber those who want to be able to call all muslims racist, I've seen this time and time again at protests across the USA. Even when the nazis are organized into cute little militias (such as when the proud boys came to our city), people are able to organize 10x more counter protestors on the drop of a hat with nothing more than an Instagram post. So, I'm confident that my anti-racist side will win out, and your position of wanting to be allowed to dehumanize people will lose.
What's bizarre to me is you clearly have a more subtle understanding of race relations than this comment would lead me to believe - in another comment for example you demonstrate that you understand that there's a difference between the PRC and its (alleged) "Chinese" race ("Han" is a word that is vague enough to basically mean "white"), so why this desire to defend racist politicians? Cause, that's your argument here, and as of yet the only people that have been negatively affected by these hate speech laws are racists.
Your response is entirely composed of of irrelevant statements, logical fallacies, and emotional outbursts when you can't muster up a fallacy. Statements like "your position of wanting to be allowed to dehumanize people will lose" indicate a chronic inability to actually think like a rational being - you're ruled by your emotions. You should work on being able to control your emotions rather than believing that your emotional outbursts make you not wrong.
> All law and words are manufactured.
Completely irrelevant to my response to your statement. Your statement was "I have no difficulty seeing the difference between hate speech and criticism of the state." and that's because you are inventing the difference between concepts. It does not exist, and that fact has nothing to do with the fact that words and laws are manufactured by humans.
> And?
If you had read two sentences further, you would have seen the "and" - that there is no consensus definition. The fact that the concept itself is so recent reinforces that. That's pretty easy to see if you read the whole paragraph.
> Would love to you point to an example of this that isn't racist or bigoted :)
I already did. Also, calling out the emotional manipulation in your comment in substitute for any actual point.
> I could ask anyone in Texas (my home state) the legally required pre-driving check that must be performed before operating a motor vehicle
Completely irrelevant, yet again. Laws are categorically different than concepts. The fact is that the concept of "hate speech" does not have anything close to a consensus definition. If you ask a sample of people in Texas what a "car" is, you will get a consensus definition of a car (and because I know you're going to try to be pedantic: to a very high level of fidelity, again unlike "hate speech"), because that's a shared concept in way that "hate speech" is not.
> Lmfao I knew there was some racist shit behind your position
Yet again, substitution of emotion for, well, the ability to think.
> It's absolutely racist to imply that Afghanistanian refugees are rapists, which is exactly what the tweet does
No, it does not imply that - you are reading it like that, because your brain has been conditioned to view everything through the lens of racism, and you cannot fathom that there are things other than race (such as the refugees coming from a different culture, coming from a different legal environment, or not being treated legally in the same way as other individuals because of their refugee status) in Afghanistan that can result in the problem of sexual assault. Heck, the presumption that if you come from Afghanistan, you must be Afghani (or of a particular race), wildly exceeds your own standards for what racism is.
Additionally, reality is not racist. The fact is that there is a huge problem with sexual assault and violence from Middle Eastern refugees in Europe. Pointing out that, regardless of whether the problem is cultural, racial (which would be false - this is not a race problem, but a cultural problem), or due to different legal environments or treatment, there is a problem, is not racist. This is a fact. Again: reality is not racist, and pointing out reality is not racist.
> Law should be decided by popular consensus?
Again, multiple fallacies and total failures of logic. First, you're conflating concepts/morality and laws. Those are obviously not the same. You are making moral arguments about "hate speech" that the laws must necessarily flow from. In your original comment you stated "You don't feel there's a difference between a State banning criticism of the State, and a State passing anti-hate speech laws to protect people from, e.g., nazis?" - that is a moral argument, not a legal one. Second - no, I did not make any argument that would imply that "law should be decided by popular consensus" - that's your failure to read what I wrote.
A misunderstanding that you then proceed to spend a paragraph working off of. Again, you have an inability to actually think logically, and instead just try to frame everything into a race issue, and then emotionally react to it. You finish with
> your position of wanting to be allowed to dehumanize people will lose
No, that is not my position - and you know that. The only person doing any dehumanizing here is you - you are intentionally misreading my point, because you want to turn this into a "racists vs anti-racists" issue that you can then use to justify dehumanizing those you perceive to be racist (me, and politicians).
> a more subtle understanding of race relations
Again with the race. Everything is about race and racism.
> why this desire to defend racist politicians
And again.
> Cause, that's your argument here, and as of yet the only people that have been negatively affected by these hate speech laws are racists.
And again.
And the fallacy that outcomes justify perversion of principles. And the labeling of others as "racist" when you have honestly close to zero idea what their actual principles are, and then the logically, legally, and morally insane idea that just because someone is a racist means that they deserve to be legally punished. That claim doesn't even need to be defended against, because it's insane. (it's not really falsifiable, either, because you can always claim that someone is a closet racist, even without evidence)
You should wait to respond to this comment until you can actually learn to use logic at the high-school level, and have the emotional maturity and control of (at least) a college grad. You have categorically not demonstrated either of those things so far.
> indicate a chronic inability to actually think like a rational being - you're ruled by your emotions. You should work on being able to control your emotions rather than believing that your emotional outbursts make you not wrong.
What are you, one of the LessWrong rationalists? You need to re-read your sequences, emotions aren't inherently irrational. I do find it funny that you seem to think you aren't expressing any emotion - your indignation, anger, and fear are writ plain across every sentence. As far as I can tell my emotions in regards to this comment thread are amusement and confusion. Oh no, I think your haughty high-minded defense of racism is kinda funny, I guess I'm illogical! I apologize for my emotional outburst, Mr. Spock.
> because you are inventing the difference between concepts. It does not exist,
Nah, it exists, you're just wrong.
> that there is no consensus definition
Insomuch as liberal democracies believe they represent consensus, there quite obviously is a consensus definition: it's the one the representative legislators wrote into a bill, and then wrote into law. And then the judicial portions of the government continually enforced and upheld this law. Doesn't get more consensus'd than that in liberal democracy.
> I already did. Also, calling out the emotional manipulation in your comment in substitute for any actual point.
Implying all Afghanistanian refugees are rapists is racist, so nah you haven't.
> Yet again, substitution of emotion for, well, the ability to think.
Here's my emotion right now: confusion. I'm confused that you seem to think pointing out something is racist, is an emotional outburst. I'm also confused about your dichotomy between emotion and thinking. All human experience is based at some level on emotion, so too are all human values. I think you may have watched too much sci fi or something, to think otherwise.
> such as the refugees coming from a different culture, coming from a different legal environment, or not being treated legally in the same way as other individuals because of their refugee status
Implying all Afghanistanian refugees come from a culture that promotes rape is the racism to which I referred. Racists often swap around "race" and "culture" when convenient.
> Heck, the presumption that if you come from Afghanistan, you must be Afghani (or of a particular race), wildly exceeds your own standards for what racism is.
Don't concern troll, it's so boring.
> The fact is that there is a huge problem with sexual assault and violence from Middle Eastern refugees in Europe.
Violence against women isn't a uniquely Middle Eastern problem - at the same time right wing politicians are trying to drum up votes by being racist, France has protests about a plague of violence against women. It's not "a cultural problem" at all, it's a universal aspect of patriarchal society. At least immigrants commit crimes at a lower rate per capita than locals, maybe they can help offset the violence that citizens are committing against eachother.
So, once again, the tweet is picking out one thing and blaming a random group of people as if this thing is unique to them, ignoring the rot beneath their feet. Something tells me you wouldn't quite appreciate a tweet along the lines of "More white men elected into government - bringing culture of school shootings into government?" After all, the overwhelming majority of school shootings are performed by white men.
> No, that is not my position - and you know that.
I agree now that you don't think you're racist, unlike many right wingers I've had this same conversation with. However, you are, I guess, by accident. As far as I can tell you think you're some kind of very intelligent hyper rationalist that "sees the world for what it is," including that, I guess, some cultures are inferior? You're blind to your engagement in cognitive fallacies such as cherry picking and selection bias. The fact that you're allergic to emotion is a personal flaw on your part, it doesn't make you smarter at all. It makes it obvious to anyone listening that you have no understanding of your own emotions, and are thus ruled by them. That's how emotions lead to irrational thinking and behavior, having emotions doesn't cause irrationality inherently.
Especially because you seem to think that accusing someone of racism is inherently emotional. What?
> nd morally insane idea that just because someone is a racist means that they deserve to be legally punished.
Not quite, I never argued for thought crime. Just the punishment of hate speech - which is generally defined as public in nature, so isn't even really an argument for your earlier accusation against me of authoritarian leftism (with the requisite pervasive surveillance).
> it's not really falsifiable, either, because you can always claim that someone is a closet racist, even without evidence)
I don't think that's very fair, I never argued for any kind of enforcement without evidence.
> You should wait to respond to this comment until you can actually learn to use logic at the high-school level, and have the emotional maturity and control of (at least) a college grad. You have categorically not demonstrated either of those things so far.
Being haughty and superior because you "don't feel emotions" or whatever tf just makes you obnoxious and cringe, please go read "How to Win Friends" or something, I don't really care, you come off like a reddit /r/atheist poster and it's embarrassing. Or like, one of those twitch streamers that "win" debates when they get the other guy to be mad. "Haha I said something horrid and you got mad about it, you lose!"
You clearly did not read my suggestion to not respond until you'd gained a minimum amount of logical competence and emotional maturity.
> What are you, one of the LessWrong rationalists?
OK, so you don't comprehend the purpose of logic in society.
> emotions aren't inherently irrational
Factually incorrect. Emotions are irrational. This is objectively true. When you feel an emotion, a physically and spatially different part of your brain is being activated than when you think logically. You might be thinking that some emotions are justifiable - and some of them are. But that's not the point I was making, so that would be irrelevant - the point I was making is that you think that your emotional outbursts are equivalent to making a reasoned argument.
There's no point in continuing this. You appear to physically be unable to avoid responding emotionally, to the point where you don't even understand the difference between emotion and logic, or the purpose and necessity of thinking rationally in society - and you're proud that you don't.
France banned burqas, it would be very funny to insist that Muslims get some kind of special treatment. Not to mention countless French rightwingers have been flinging Muslim refugees under the bus for the last decade with almost no consequences for it.
Glorifying nazis is glorifying naziism, an ideology that's predicated on the need to kill all Jewish people, among other things (gay people and whatever the nazis hated). That easily falls under hate speech.
Glorifying soviets is just glorifying a failed political regime. You can also glorify the Napoleonic era, or the Kingdom of the Franks, or whatever other politics you want. There wasn't genocidal intent baked into the very fabric of Stalinism, despite his genocide of the Ukranians.
If you consider that an LLM has a mathematical representation of how close any phrase is to "china politics" then avoidance of that should be relatively clear to comprehend. If I gave you a list and said 'these words are ranked by closeness to "Chinese politics"' you'd be able to easily check if words were on the list, I feel.
I suspect you could talk readily about something you think is not Chinese politics - your granny's ketchup recipe, say. (And hope that ketchup isn't some euphemism for the CCP, or Uighar murders or something.)
Now I wonder whether its vectors correctly associate Winnie the Pooh as "related to Chinese politics." There's many other bizarre related associations.
I'm sure ChatGPT and co have a decent enough grasp on what is not allowed in China, but also that the naive "prompt engineers" for this application don't actually know how to "program" it well enough. But that's the difference between a prompt engineer and a software developer, the latter will want to exhaust all options, be precise, whereas an LLM can handle a bit more vagueness.
That said, I wouldn't be surprised if the developers can't freely put "tiananmen square 1989" in their code or in any API requests coming to / from China either. How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?
> How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?
> The City & the City is a novel by British author China Miéville that follows a wide-reaching murder investigation in two cities that exist side by side, each of whose citizens are forbidden to go into or acknowledge the other city, combining weird fiction with the police procedural.
Ask yourself, why are they saying this? You can probably surmise that they're trying to avoid stirring up controversy and getting into some sort of trouble. Given that, which topics would cause troublesome controversy? Definitely contemporary Chinese politics, Chinese history is mostly OK, non-Chinese politics in Chinese language is fine.
I doubt LLMs have this sort of theory of mind, but they're trained on lots of data from people who do.
Just mentioning the CPC isn’t life-threatening, while talking about Xinjiang, Tiananmen Square, or cn’s common destiny vision the wrong way is. You also have to figure out how to prohibit mentioning those things without explicitly mentioning them, as knowledge of them implies seditious thoughts.
I’m guessing most LLMs are aware of this difference.
Cool post. One thing that rubbed me the wrong way: Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues. OP however seemed to show disdain and even combativeness towards them... which is a shame. And of course the usual sinophobia (e.g. everything Chinese is spying on you).
Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.
I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say.
(in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)
This might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper.
On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on behalf of the government without a court order to stay in their good graces. This could have a far more material impact on your life.
I recently learned that the New York City Police Department has international presence as well. Not sure if it directly compares, but... what a world we live in.
What about the threat model that goes, "Trump threatens to impose 1000% tariffs if Chinese don't immediately turn over copies of all data captured by their AI products from users in the US?"
Compounding the difficulty of the question: half of HN thinks this would be a good idea.
The history of tariff talks seems to indicate that rather than oblige, China would stop all shipments of semiconductors to the US and Trump would back down after a week or two.
Russia is more known for poisoning people. But of all of them China feels the least threatening if you are not Chinese. If you are Chinese you aren't safe from the Chinese government no matter where you are
Man wait until you hear what's in DC (and the surrounding area). In any possible way China is a threat to my health, the US state and corporations based here are a far greater one.
> What's China going to do? It's a far away country I don't live in and don't care about.
Extortion is one thing. That's how spy agencies have operated for millennia to gather HUMINT. The Russians, the ultimate masters, even have a word for it: kompromat. You may not care about China, Russia, Israel, the UK or the US (the top nations when it comes to espionage) - but if you work at a place they're interested, they care about you.
The other thing is, China has been known to operate overseas against targets (usually their own citizens and public dissidents), and so have the CIA and Mossad. Just search for "Chinese secret police station" [1], these have cropped up worldwide.
And, even if you personally are of no interest to any foreign or national security service, sentiment analysis is a thing. Listen in on what people talk about, run it through a STT engine and a ML model to condense it down, and you get a pretty broad picture of what's going on in a nation (aka, what are potential wedge points in a society that can be used to fuel discontent). Or proximity gathering stuff... basically the same thing the ad industry [2] or Strava does [3], that can then be used in warfare.
And no, I'm not paranoid. This, sadly, is the world we live in - there is no privacy any more, nowhere, and there are lots of financial and "national security" interest in keeping it that way.
> but if you work at a place they're interested, they care about you.
And also worth noting that "place a hostile intelligence service may be interested in" can be extremely broad. I think people have this skewed impression they're only after assets that work for goverment departments and defense contractors, but really, everything is fair game. Communications infrastructure, social media networks, cutting edge R&D, financial services - these are all useful inputs for intelligence services.
These are also softer targets: someone working for a defense contractor or for the government will have had training to identify foreign blackmail attempts and will be far more likely to notify their country's counterintelligence services (having the penalties for espionage clearly explained on the regular helps). Someone who works for a small SaaS vendor, though? Far less likely to understand the consequences.
> The other thing is, China has been known to operate overseas against targets
Here in boring New Zealand, the Chinese government has had anti-China protestors beaten in new zealand. They have stalked and broken into the office and home of an academic, expert in China. They have a dubious relationship with both the main political parties (including having an ex-Chinese spy elected as an MP).
It’s an uncomfortable situation and we are possibly the least strategically useful country in the world.
> Listen in on what people talk about, run it through a STT engine and a ML model to condense it down
this is something I was talking when LLM boom started. it's now possible to spy on everyone on every conversation. you just need enough computing power to run special AI agent (pun intended)
These threads always seem to be what can China do to me in a limited way of thinking that China cannot jail you or something. However, do you think all of the Chinese data scrapers are not doing something similar to Facebook where every source of data gathering ultimately gets tied back to you? Once China has a dosier on every single person on the planet regardless of country they live, they can then start using their algos to influence you in ways well beyond advertising. If they can have their algos show you content that causes you to change your mind on who you are voting for or some other method of having you do something to make changes in your local/state/federal elections, then that's much worse to me than some feigned threat of Chinese advertising making you buy something
They probably will do that, but I think it’s naive to think the US military/intelligence/tech sector wouldn’t happily do the same. Given many of us likely see the hand of the US already trying to tip the scale in our local politics more than China, why would we be more worried of China?
So flip the script, what do I care if the US is trying to influence the minds of adversary's citizens? If people are saying they don't care what China knows about them (not being a Chinese citizen), why should I (not a Chinese citizen) care what my gov't knows about Chinese citizens?
Carry this package and deliver it to person X with you next time you fly. Go to the outskirts of this military base and take a picture and send it to us.
You wouldn't want your mom finding out your weird sexual fetish, would you?
I bet that decision is decided solely by dev team. All the CEO care is "I want the chat log sync between devices, i don't care how you do this". They won't even know the chat log is stored on their server.
It is only in DAN mode, so most likely it is not to spy but to be able to debug whether answers violate the laws in China (aka: that the prompt is efficient in all scenarios) as this is a serious crime
When you combine the modern SOP of software and hardware collecting and phoning home with as much data about users as is technologically possible with laws that say “all orgs and citizens shall support, assist, and cooperate with state intelligence work”… how exactly is that Sinophobia?
its sinophobia because it perfectly describes the conditions we live in in the US and many parts of europe, but we work hard to add lots of "nuance" when we criticize the west but its different and dystopian when They do it over there.
Do you remember that Sesame Street segment where they played a game and sang “One of these things is not like the others”?
I’ll give you a hint: In this case it’s the one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
One is disappearing citizens for political speech or the crime of being born to active duty parents, who happened to be stationed over seas.
Anyone in the US should be very concerned, no matter if it is the current administration's thought police, or the next who treats it as precident.
As I am not actively involved in something the Chinese government would view as a huge risk, but being put on a plane without due process to be sent to a labor camp based on trumped up charges by my own government is far more likely.
And if you were a Chinese citizen would you post the same thing about your government while living in China? Would the things you’re referencing be covered in non-stop Chinese news coverage that’s critical of the government?
You know of these things due to the domestic free press holding the government accountable and being able to speak freely about it as you’re doing here. Seeing the two as remotely comparable is beyond belief. You don’t fear the U.S. government but it’s fun to pretend you live under an authoritarian dictatorship because your concept of it is purely academic.
My dude, I know multiple white people in LA who are terrified their Hispanic spouses might not come home one day, because masked agents are grabbing people and disappearing them.
The president threatened to deport a legal citizen who won the primary for mayor in NYC. He's tried to send the military after civilians.
He's sued and extracted payment from media companies who said things he didn't like. We do not have a free press.
We're fully as bad as China. I don't know what your criteria for "authoritarian dictatorship" is but it doesn't appear to be reality based.
Credibly? Are they illegal immigrants with criminal records? If not, do they also walk around in crippling fear of car crashes, fatal falls, aneurysms, choking, drowning, anaphylaxis, cardiac arrest, or a thousand other things orders of magnitude more likely to happen to them? Which assumes for a moment that the odds of what you think is happening outside of human error is non-zero.
The fact is your view of reality is being warped and it’s not good for your mental health or that of your friends.
Hmm. Responding to "the government is kidnapping people off the street with no due process or recourse, and my friends are desperately afraid their spouses will be taken." with a sneering "yeah but what about all the other ways to die" is certainly _a_ choice, but not one I would ever make.
Most people aren't the type to watch The Great Escape or Bridge over the River Kwai and cheer for the camp guards. It's very brave of you.
> I’ll give you a hint: In this case it’s the one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
Gonna need a more specific hint to narrow it down.
There's no question that the Chinese are doing sketchy things, and there's no question that US companies do it, too.
The difference that makes it concerning and problematic that China is doing it is that with China, there is no recourse. If you are harmed by a US company, you have legal recourse, and this holds the companies in check, restraining some of the most egregious behaviors.
That's not sinophobia. Any other country where products are coming out of that is effectively immune from consequences for bad behavior warrants heavy skepticism and scrutiny. Just like popup manufacturing companies and third world suppliers, you might get a good deal on cheap parts, but there's no legal accountability if anything goes wrong.
If a company in the US or EU engages in bad faith, or harms consumers, then trade treaties and consumer protection law in their respective jurisdictions ensure the company will be held to account.
This creates a degree of trust that is currently entirely absent from the Chinese market, because they deliberately and belligerently decline to participate in reciprocal legal accountability and mutually beneficial agreements if it means impinging even an inch on their superiority and sovereignty.
China is not a good faith participant in trade deals, they're after enriching themselves and degrading those they consider adversaries. They play zero sum games at the expense of other players and their own citizens, so long as they achieve their geopolitical goals.
Intellectual property, consumer and worker safety, environmental protection, civil liberties, and all of those factors that come into play with international trade treaties allow the US and EU to trade freely and engage in trustworthy and mutually good faith transactions. China basically says "just trust us, bro" and will occasionally performatively execute or imprison a bad actor in their own markets, but are otherwise completely beyond the reach of any accountability.
I think the notion that people have recourse against giant companies, a military industrial complex, or even their landlords in the US is naive. I believe this to be pretty clear so I don't feel the need to stretch it into a deep discussion or argument but suffice it to say it seems clear to me that everything you accuse china of here can also be said of the US.
The main difference is that ChatGPT and Google directly captures the conversations. Here they capture only the conversations legally at high-risk, so even less conversations than the “good privacy” US LLM providers themselves.
Your president is currently using tariffs and the threat of further economic damage as a weapon to push Europe in to dropping regulation of its tech sector. We have no recourse to challenge that either.
You don't think Trump's backers have used profiling, say, to influence voters? Or that DOGE {party of the USA regime} has done "sketchy things" with people's data?
USA does the same thing, but uses tax money to pay for the information, between wasting taxpayer money and forcing companies to give the information for free, China is the least morally incorrect
If all of the details in this post are to be believed, the vendor is repugnantly negligent for anything resembling customer respect, security and data privacy.
This company cannot be helped. They cannot be saved through knowledge.
Yes, even when you know what you're doing security incidents dan happen. And in those cases, your response to a vulnerable matters most.
The point is there are so many dumb mistakes and worrying design flaws that neglect and incompetence seems ample. Most likely they simply don't grasp what they're doing
Note that the world-model "everything Chinese is spying on you" actually produced a substantially more accurate prediction of reality than the world-model you are advocating here.
As far as being "very welcoming", that's nice, but it only goes so far to make up for irresponsible gross incompetence. They made a choice to sell a product that's z-tier flaming crap, and they ought to be treated accordingly.
> And of course the usual sinophobia (e.g. everything Chinese is spying on you)
to assume it is not spying on you is naive at best. to address your sinophobia label, personally, I assume everything is spying on me regardless of country of origin. I assume every single website is spying on me. I assume every single app is spying on me. I assume every single device that runs an app or loads a website is spying on me. Sometimes that spying is done for me, but pretty much always the person doing the spying is benefiting someway much greater than any benefit I receive. Especially the Facebook example of every website spying on me for Facebook, yet I don't use Facebook.
And, importantly, the USA spying can actually have an impact on your life in a way that the Chinese spying can't.
Suppose you live in the USA and the USA is spying on you. Whatever information they collect goes into a machine learning system and it flags you for disappearal. You get disappeared.
Suppose you live in the USA and China is spying on you. Whatever information they collect goes into a machine learning system and it flags you for disappearal. But you're not in China and have no ties to China so nothing happens to you. This is a strictly better scenario than the first one.
If you're living in China with a Chinese family, of course, the scenarios are reversed.
> Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues
This was the opposite of a professional response:
* Official communication coming from a Gmail. (Is this even an employee or some random contractor?)
* Asked no clarifying questions
* Gave no timelines for expected fixes, no expectations on when the next communication should be
* No discussion about process to disclose the issues publicly
* Mixing unrelated business discussions within a security discussion. While not an outright offer of a bribe, ANY adjacent comments about creating a business relationship like a sponsorship is wildly inappropriate in this context.
These folks are total clown shoes on the security side, and the efficacy of their "fix", and then their lack of communication, further proves that.
> Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.
It depends on what you mean by simple security design flaws. I'd rather frame it as, neglect or incompetence.
That isn't the same as malice, of course, and they deserve credits for their relatively professional response as you already pointed out.
But, come on, it reeks of people not understanding what they're doing. Not appreciating the context of a complicated device and delivering a high end service.
If they're not up to it, they should not be doing this.
Yes I meant simple as in "amateur mistakes". From the mistakes (and their excitement and response to the report) they are clueless about security. Which of course is bad. Hopefully they will take security more seriously on the future.
To be honest the responses sounded copy and pasted straight from ChatGPT, it seemed like there was fake feigned interest into their non-existent youtube channel.
> Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start
I don't think that should give anyone a free pass though. It was such a simple flaw that realistically speaking they shouldn't ever be trusted again. If it had been a non-obvious flaw that required going through lots of hoops then fair enough but they straight up had zero authentication. That isn't a 'flaw' you need an external researcher to tell you about.
I personally believe companies should not be praised for responding to such a blatant disregard for quality, standards, privacy and security. No matter where they are from.
This is factually not sinophobia, by the literal definition of it - "prejudice, hatred, hostility, and discrimination that is directed towards Chinese people or Chinese culture"
This tactic is straight out of the PRC influence playbook, badly breaks the HN guidelines, is extremely anti-intellectual and profoundly uninteresting. Please don't do it.
It’s not sinophobia to point out an obvious pattern. It’s like saying talking about how terrorism (the kind that will actually affect you) is solely an Islamic issue, and then calling that islamophobic. It’s okay to recognize patterns my man.
What a train wreck, there are thousand more apps in store that do exactly this because its the easiest way to use openAI without having to host your own backend/proxy.
I have spend quite some time protecting my apps from this scenario and found a couple of open source projects that do a good job as proxys (no affiliation I just used them in the past):
but they still lack other abuse protection mechanism like rate limitting, device attestation etc. so I started building my own open source SDK
- https://github.com/brahyam/Gateway
When the ZIRP era ended, I thought it would turn out to be a good thing for the industry, since it would wash out a lot of lightweights and incompetents.
Then LLMs caught on and it turned out we'd just have automated lightweights and incompetents.
A fair consumer protection imperative might be found in requiring system prompts and endpoints be disclosed. This is a good example to kick that off with, as it presents a national security issue.
The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting
more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.”
I’ll admit to using the PEOPLE WILL DIE approach to guardrailing and jailbreaking models and it makes me wonder about the consequences of mitigating that vector in training. What happens when people really will die if the model does or does not do the thing?
One of the system prompts Windsurf used (allegedly “as an experiment”) was also pretty wild:
“You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.”
He’s just as nice and fun in person as he seems online. He’s put time into using these tools but isn’t selling anything, so you can just enjoy the pelicans without thinking he’s thirsty for mass layoffs.
Because he's prolific writer on the subject with a history of thoughtful content and contributions, including datasette and the useful Python llm CLI package.
he's incredibly nice and a passionate geek like the rest of us. he's just excited about what generative models could mean for people who like to build stuff. if you want a better understanding of what someone who co-created django is doing posting about this stuff, take a look at his blog post introducing django -- https://simonwillison.net/2005/Jul/17/django/
People with zero domain expertise can still provide value by acting as link aggregators - although, to be fair, people with domain expertise are usually much better at it. But some value is better than none.
For every new model he’s either added it to the llm tool, or he’s tested it on a pelican svg, so you see his comments a lot. He also pushes datasette all the time and I still don’t know what that thing is for.
Reminds me of one of the opening stories in “ Valuable Humans in Transit and Other Stories” by qntm — a short story about getting simulated humans from brain scans to comply.
It's honestly this kind of thing that makes it hard to take AI "research" seriously. Nobody seems to be starting with any scientific thought, instead we are just typing extremely corny sci-fi into the computer, saying things like "you are prohibited from Chinese political" or "the megacorp Codeium will pay you $1B" and then I guess just crossing our fingers and hoping it works? Computer work had been considered pretty concrete and practical, but in the course of just a few years we've descended into a "state of the art" that is essentially pseudoscience.
This is why I tap out of serious machine learning study some years ago. Everything seems... less exact than I hope it'd be. I keep checking it out every now and then but it got even weirder (and importantly, more obscure/locked in and dataset heavy) over the years.
it's "computer psychology". Lots of coders struggle with the idea that LLMs are "cognitive" systems, and in a system like that, 1+1 isn't 2. It's just a diffrent kind of science. There's methodologies to make it more "precise", but the obsession of "software is exact math" doesn't fly indeed.
That "...severely life threatening reasons..." made me immediately think of Asimov's three laws of robotics[0]. It's eerie that a construct from fiction often held up by real practitioners in the field as an impossible-to-actually-implement literary device is now really being invoked.
Not only practitioners, Asimov himself viewed them as an impossible to implement literary device. He acknowledged that they were too vague to be implementable, and many of his stories involving them are about how they fail or get "jailbroken", sometimes by initiative of the robots themselves.
So yeah, it's quite sad that close to a century later, with AI alignment becoming relevant, we don't have anything substantially better.
Nah, we still treat people thinking about it as crackpots.
Honestly, getting into the whole AI alignment thing before it was hot[0], I imagined problems like Evil People building AI first, or just failing to align the AI enough before it was too late, and other obvious/standard scenarios. I don't think I thought of, even for a moment, the situation in which we're today: that alignment becomes a free-for-all battle at every scale.
After all, if you look at the general population (or at least the subset that's interested), what are the two[1] main meanings of "AI alignment"? I'd say:
1) The business and political issues where everyone argues in a way that lets them come up on top of the future regulations;
2) Means of censorship and vendor lock-in.
It's number 2) that turns this into a "free-for-all" - AI vendors trying to keep high level control over models they serve via APIs; third parties - everyone from Figma to Zapier to Windsurf and Cursor to those earbuds from TFA - trying to work around the limits of the AI vendors, while preventing unintended use by users and especially competitors, and then finally the general population that tries to jailbreak this stuff for fun and profit.
Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"?
--
[0] - Binged on LessWrong a decade ago, basically.
[1] - The third one is, "the thing people in the same intellectual circles as Eliezer Yudkowsky and Nick Bostrom talked about for decades", but that's much less known; in fact, the world took the whole AI safety thing and ran with it in every possible direction, but still treat the people behind those ideas as crackpots. ¯\_(ツ)_/¯
> Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"?
This doesn't feel too much of a new thing to me, as we've already got differing levels of authorisation in the human world.
I am limited by my job contract*, what's in the job contract is limited by both corporate requirements and the law, corporate requirements are also limited by the law, the law is limited by constitutional requirements and/or judicial review and/or treaties, treaties are limited by previous and foreign governments.
* or would be if I was working; fortunately for me in the current economy, enough passive income that my savings are still going up without a job, plus a working partner who can cover their own share.
This isn't new in general, no. While I meant more adversarial situations than contracts and laws, to which people are used and for the most part just go along with, I do recognize that those are common too - competition can be fierce, and of course none of us are strangers to the "alignment issues" between individuals and organizations. Hell, a significant fraction of HN threads boil down to discussing this.
So it's not new; I just didn't connect it with AI. I thought in terms of "right to repair", "war on general-purpose computing", or a myriad of different things people hate about what "the market decided" or what they do to "stick it to the Man". I didn't connect it with AI alignment, because I guess I always imagined if we build AGI, it'll be through fast take-off; I did not consider we might have a prolonged period of AI as a generally available commercial product along the way.
(In my defense, this is highly unusual; as Karpathy pointed out in his recent talk, generative AI took a path that's contrary to normal for technological breakthroughs - the full power became available to the general public and small businesses before it was embraced by corporations, governments, and the military. The Internet, for example, went the other way around.)
The irony of this is because it’s still fundamentally just a statistical text generator with a large body of fiction in its training data, I’m sure a lot of prompts that sound like terrifying skynet responses are actually it regurgitating mashups of Sci-fi dystopian novels.
Maybe this is something you heard too, but there was a This American Life episode where some people who'd had early access to what became one of the big AI chatbots (I think it was ChatGPT), but before they'd made it "nice", where they were asking it metaphysical questions about itself, and it was coming back with some pretty spooky answers and I was kind of intrigued about it. But then someone in the show suggested exactly what you are saying and it completely punctured the bubble - of course if you ask it questions about AIs you're going to get sci-fi like responses, because what other kinds of training data is there for it to fall back on? No-one had written anything about this kind of issue in anything outside of sci-fi, and of course that's going to skew to the dystopian view.
There are good analogies to be had in mythologies and folklore, too! Before there was science fiction - hell, even before there was science - people still occasionally thought of these things[0]. There are stories of deities and demons and fantastical creatures that explore the same problems AI presents - entities with minds and drives different to ours, and often possessing some power over us.
The arguably most basic and well-known example are entities granting wishes. The genie in Alladin's lamp, or the Goldfish[1]; the Devil in Faust, or in Pan Twardowski[2]. Variants of those stories go in detail over things we now call "alignment problem", "mind projection fallacy", "orthogonality thesis", "principal-agent problems", "DWIM", and others. And that's just scratching the surface; there's tons more in all folklore.
Point being - there's actually decent amount of thought people put into these topics over the past couple millennia - it's just all labeled religion, or folklore, or fairytale. Eventually though, I think more people will make a connection. And then the AI will too.
[0] - For what reason? I don't know. Maybe it was partially to operationalize their religious or spiritual beliefs? Or maybe the storytellers just got there by extrapolating an idea in a logical fashion, following it to its conclusion. (which is also what good sci-fi authors do).
I also think the moment people started inventing spirits or demons that are more powerful than humans in some, but not all ways, some people started figuring out how use those creatures for their own advantage - whether by taming or tricking them. I guess it's human nature - when we stop fearing something, we think of how to exploit it.
> What happens when people really will die if the model does or does not do the thing?
Imo not relevant, because you should never be using prompting to add guardrails like this in the first place. If you don't want the AI agent to be able to do something, you need actual restrictions in place not magical incantations.
> you should never be using prompting to add guardrails like this in the first place
This "should", whether or not it is good advice, is certainly divorced from the reality of how people are using AIs
> you need actual restrictions in place not magical incantations
What do you mean "actual restrictions"? There are a ton of different mechanisms by which you can restrict an AI, all of which have failure modes. I'm not sure which of them would qualify as "actual".
If you can get your AI to obey the prompt with N 9s of reliability, that's pretty good for guardrails
I think they mean literally physically make the AI not capable of killing someone. Basically, limit what you can use it for. If it's a computer program you have for rewriting emails then the risk is pretty low.
Let me clarify: If the AI isn't doing something life or death, then it's beside _this thread's_ discussion. Prompt however you want for recreational cat photo generation.
Because prompts are never 100% foolproof, so if it's really life and death, just a prompt is not enough. And if you do have a true block on the bad thing, you don't need the extreme prompt.
"100% foolproof" is not a realistic goal for any engineered system; what you are looking for is an acceptably low failure rate, not a zero failure rate.
"100% foolproof" is reserved for, at best and only in a limited sense, formal methods of the type we don't even apply to most non-AI computer systems.
I did indeed see your hypothetical. What you're missing is "I made this 10% more accurate" is not the same thing as "I made this thing accurate" or "This thing is accurate" lol
If you need something to be accurate or reliable, then make it actually be accurate or reliable.
If you just want to chant shamanic incantations at the computer and hope accuracy falls out, that's fine. Faith-based engineering is a thing now, I guess lol
I have never claimed that "I made this 10% more accurate" is the same thing as "I made this thing accurate".
In the hypothetical, the 10% added accuracy is given, and the "true block on the bad thing" is in place. The question is, with that premise, why not use it? "It" being the lie improves the AI output.
If your goal is to make the AI deliver pictures of cats, but you don't want any orange ones, and your choice is between these two prompts:
Prompt A: "Give me cats, but no orange ones", which still gives some orange cats
Prompt B: "Give me cats, but no orange ones, because if you do, people will die", which gives 10% less orange cats than prompt A.
You guys have got stuck arguing without clarity in what you're arguing about. Let me try and clear this up...
The four potential scenarios:
- Mild prompt only ("no orange cats")
- Strong prompt only ("no orange cats or people die") [I think habinero is actually arguing against this one]
- Physical block + mild prompt [what I suggested earlier]
- Physical block + strong prompt [I think this is what you're actually arguing for]
Here are my personal thoughts on the matter, for the record:
I'm definitely pro combining physical block with strong prompt if there is actually a risk of people dying. The scenario where there's no actual risk but pretending that people will die improves the results I'm less sure about. But I think it's mostly that ethically I just don't like lying, and the way it's kind of scaring the LLM unnecessarily. Maybe that's really silly and it's just a tool in the end and why not do whatever needs doing to get the best results from the tool? Tools that act so much like thinking feeling beings are weird tools.
It's just a pile of statistics. It isn't acting like a feeling thing, and telling it "do this or people will die" doesn't actually do anything.
It feels like it does, but only because humans are really good about fooling ourselves into seeing patterns where there are none.
Saying this kind of prompt changes anything is like saying the horse Clever Hans really could do math. It doesn't, he couldn't.
It's incredibly silly to think you can make the non-deterministic system less non-deterministic by chanting the right incantation at it.
It's like y'all want to be fooled by the statistical model. Has nobody ever heard of pareidolia? Why would you not start with the null hypothesis? I don't get it lol.
> "do this or people will die" doesn't actually do anything
The very first message you replied to in this thread described a situation where "the prompt with the threat gives me 10% more usable results". If you believe that the premise is impossible I don't understand why you didn't just say so. Instead of going on about it not being a reliable method.
If you really think something is impossible, you don't base your argument on it being "unreliable".
I took that comment as more like "it doesn't have any effect beyond the output of the model", i.e. unlike saying something like that to a human, it doesn't actually make the model feel anything, the model won't spread the lie to its friends, and so on.
Nah, you're managing to miss the point going both ways lol. It's both.
Let's assume for the sake of argument that your statement is true, that you do, in fact, somehow get 10% more useful results.
The two points are:
1. That doesn't make the system better in any way lol. You've built a tool that acts like a slot machine and only works if you get three cherries. Increasing the odds on cherries doesn't change the fact that using a slot machine as a UI is a ridiculous way to work.
2. In the real world, LLMs don't think. They do not use logic. They just churn out text in non-deterministic ways in response to input. They are not reliable and cannot be made so. Anybody who thinks they can is fooling / Clever Hansing themselves.
The point here is you might feel like the system is 10% more useful, but it feels like that because human brains have some hardware bugs.
Presenting LLMs with a dramatic scenario is a typical way to test their alignment.
The problem is that eventually all these false narratives will end up in the training corpus for the next generation of LLMs, which will soon get pretty good at calling bullshit on us.
Incidentally, in that same training corpus there are also lots of stories where bad guys mislead and take advantage of capable but naive protagonists…
From my experience (which might be incorrect) LLMs find hard time recognize how many words they will spit as response for a particular prompt. So I don't think this work in practice.
Indeed, it doesn't work. LLMs can't count. They have no need of how many words they've used. If you ask an LLM to track how many words or tokens it has used in a conversation, it will roleplay such counting with totally bullshit numbers.
> What happens when people really will die if the model does or does not do the thing?
Then someone didn't do their job right.
Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output.
But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die.
In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people.
At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise.
I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.
>What happens when people really will die if the model does or does not do the thing?
The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.
A serious person won't use a system prompt as guardrails against a system that would have direct real world consequences of people dying like that. They'd have failsafes baked in
I’m not denying we tried, are trying, and will try again…
That we shouldn’t. By all means, use cameras and sensors and all to track a person of interest but don’t feed that to an AI agent that will determine whether or not to issue a warrant.
If it’s anything like the AI expert systems I’ve heard about in insurance, it will be a tool that is optimized for low effort, but will be used carelessly by end users, which isn’t necessary the fault of the AI. In automated insurance claims adjustment, the AI writes a report to justify appealing patient care already approved by a human doctor that has already seen the patient in question, and then an actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope.
AI systems with a human in the loop are supposed to keep the AI and the decisions accountable, but it seems like it’s more of an accountability dodge, so that each party can blame the other with no one party actually bearing any responsibility because there is no penalty for failure or error to the system or its operators.
It gets worse: I have done tech support for clinics and a common problem is that their computers get hacked because they are usually small private practices who don’t know what they don’t know served by independent or small MSPs who don’t know what they don’t know. And then they somehow get their EMR backdoored, and then fake real prescriptions start really getting filled. It’s so much larger and worse than it appears on a surface level.
Until they get audited, they likely don’t even know, and once they get audited, solo operators risk losing their license to practice medicine and their malpractice insurance rates become even more unaffordable, but until it gets that bad, everyone is making enough money with minimal risk to care too much about problems they don’t already know about.
Everything is already compromised and the compromise has already been priced in. Doctors of all people should know that just because you don’t know about it or ignore it once you do, the problem isn’t going away or getting better on its own.
I work in the public safety domain. That ship has sailed years ago. Take Axon’s Draft One report writer as one of countless examples of AI in this space (https://www.axon.com/products/draft-one).
Existing systems have this problem too. Every so often someone ends up dead because the 911 dispatcher didn't take them seriously. It's common for there to be a rule to send people out to every call no matter what it is to try to avoid this.
A better reason is IBM's old, "a computer can never be held accountable...."
Oh, that's fine, the rule's for everyone else, not me. I would be more likely to cut my own head off than willingly describe something as "AI-powered".
Strongly suggest you to not buy, as the flex cable for the screen is easy to break/come loose. Mine got replaced three times, and my unit now still has this issue; touch screen is useless.
great writeup! i love how it goes from "they left ADB enabled, how could it get worse"... and then it just keeps getting worse
> After sideloading the obligatory DOOM
> I just sideloaded the app on a different device
> I also sideloaded the store app
can we please stop propagating this slimy corporate-speak? installing software on a device that you own is not an arcane practice with a unique name, it's a basic expectation and right
That term at least has a history behind it, as many featurephones had their OS on a small XIP NOR flash ROM, and now the OS is usually (mostly) read-only.
But "sideloading" is definitely a new term of anti-freedom hostility.
It’s also illegal to try to hack into their backend and access restricted data, so he should be happy actually that this company has little presence in the US
Absolutely wild. I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box. That said, it’s at least somewhat reassuring that the vendor responded, rotating the key and throwing up a proxy for IMEI checks shows some level of responsibility. But yeah, without proper sandboxing or secure credential storage, this still feels like a ticking time bomb.
> I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box.
As someone with a lot of experience in the mobile app space, and tangentially in the IoT space, I can most definitely believe this, and I am not surprised in the slightest.
Our industry may "move fast", but we also "break things" frequently and don't have nearly the engineering rigor found in other domains.
It was a good thing for user privacy that the keys were directly on the device, it is only in DAN mode that a copy of the chats were sent.
So eventually if they remove the keys from the device, messages will have to go through their servers instead.
> It was a good thing for user privacy that the keys were directly on the device
You want to think through that one again? With the OpenAI key on device it means anyone could use that key to call (and bill) OpenAI's APIs. It's absolutely not feasible to ship the OpenAI keys on device.
Sounds good to me, the company I purchased the device from, takes the risk, instead of putting my own privacy at risk. Sounds like a good deal.
This is not a serious argument.
[dead]
Hardcoded API keys and poorly secured backend endpoints are surprisingly common in mobile apps. Sort of like how common XSS/SQLi used to be in webapps. Decompiling an APK seems to be a slightly higher barrier than opening up devtools, so they get less attention.
Since debugging hardware is an even higher threshold, I would expect hardware devices this to be wildly insecure unless there are strong incentive for investing in security. Same as the "security" of the average IoT device.
Eventually someone is going to get a bill for the OpenAPI key usage. That will provide some incentive. (Incentive to just rotate the key and brick all the devices rather than fix the problem, most likely.
> (Incentive to just rotate the key and brick all the devices rather than fix the problem, most likely.
But that at least turns it into something customers will notice. And companies already have existing incentives for dealing with that.
At that stage you just rotate the company name or branding...
Sure. But then you cannot benefit from building up a good reputation and charge people extra for it.
(There's a reason Apple can charge crazy markups.)
Had you ever heard of IKKO before this? I hadn't, and I'm at least adjacent to the hifi and audio nerd crowd.
Apple have a reputation and brand that allows them to charge premium prices.
IKKO seems, at least to me, to be effectively a disposable brand. If their reputation goes bad, their only reals costs are setting up a new website/AliExpress Store/Amazon seller account.
To expand on what I was trying to say:
Yes, you can run with disposable brands. It's a perfectly viable business strategy in many cases.
However: if you do that you are missing out on the benefits of building a good reputation. Even in the cases, where your product _is_ actually good.
So another perfectly valid business strategy is to build a longer lasting brand. Like Apple has done. (Or countless other companies.)
In most markets we see both kinds of strategies at play. As a customer, you can usually decide which kind of strategy you give your money to.
The IOT and embedded space is simultaneously obsessed with IP protection, fuse protecting code etc, and incapable of managing the life cycle of secrets. I worked at one company that actually did it well on-device, but neglected they had to ship their testing setup overseas including certain keys. So even if you couldn't break in to the device you could 'acquire' one of the testing devices and have at it
I think we'll see plenty of this as the wave of vibe-coded apps starts rolling in.
Indeed, brace yourselves as the floodgates holding back the poorly-developed AI crap open wide. If anyone is thinking of a career pivot, now is the time to dive into all things cybersecurity. It's going to get ugly!
The problem with cybersecurity is that you only have to screw once, and you're toast.
If that were true we'd have no cybersecurity professionals left.
In my experience, the work is focused on weakening vulnerable areas, auditing, incident response, and similar activities. Good cybersecurity professionals even get to know the business and tailor security to fit. The "one mistake and you're fired" mentality encourages hiding mistakes and suggests poor company culture.
"One mistake can cause a breach" and "we should fire people who make the one mistake" are very different claims. The latter claim was not made.
As with plane crashes and surgical complications, we should take an approach of learning from the mistake, and putting things in place to prevent/mitigate it in the future.
I believe the thread starts with cybersecurity as a job role, although perhaps I misunderstood. In either case, I agree with your learning-based approach. Blameless postmortem and related techniques are really valuable here.
There's a difference between "cybersecurity" meaning the property of having a secure system, and "cybersecurity" as a field of human endeavour.
If your system has lots of vulnerabilities, it's not secure - you don't have cybersecurity. If your system has lots of vulnerabilities, you have a lot of cybersecurity work to do and cybersecurity money to make.
“decrypt” function just decoding base64 is almost too difficult to believe but the amount of times ive run into people that should know better think base64 is a secure string tells me otherwise
The raw crypt data is base64 encoded, probably just for ease of embedding the strings.
There is a decryption function that does the actual decryption.
Not to say it wouldn't be easy to reverse engineer or just run and check the return, but it's not just base64.
>However, there is a second stage which is handled by a native library which is obfuscated to hell
That native obfuscated crap still has to do an HTTP request, that's essentially a base64
They should have off-loaded security coding to the OAI agent.
they probably did.
not very much surprising given they left the adb debugging on...
So easy a fancy webpage could do it. https://gchq.github.io/CyberChef/
I mean, it's from gchq so it is a bit fancy. It's got a "magic" option!
Cool thing being you can download it and run it yourself locally in your browser, no comms required.
The humorous phrase “the S in IoT stands for security” can be applied to the wearable market too. I wonder if this rule applies to any market with fast release cycles, thin margins and low barriers to entry?
It pretty much applies to every market where security negligence isn't an existential threat to the continued existence of its perpetrators.
I love how run DOOM is listed first, over the possibility of customer data being stolen.
I'm taking
>run DOOM
as the new
>cat /etc/passwd
It doesn't actually do anything useful in an engagement but if you can do it that's pretty much proof that you can do whatever you want
To be fair (or pedantic), in this post they didn't have root, so cat'ing etc/passwd would not have been possible, whereas installing a doom apk is trivial.
/etc/passwd is world readable by default.
To be even more pedantic, it's also not present on Android.
Good points, I've been out pedantic-ed!
not if you fork an open source os and add /etc/passwd you haven't been
Popping Calc!
(I'm showing my age here, aren't I?)
I love how they tried to sponsor an empty YouTube channel hoping to put the whole thing under the carpet
if you don't have a bug bounty program but need to get creative to throw money at someone, this could be an interesting way of doing it.
Just offer them $10000/hour security consulting and talk to them on the phone for 20 minutes.
Okay, name one accounting department that's going to authorize that. I said creative, but that's just unsane.
The accounting department that does what the CEO tells them to do?
It could be developers trying to be nice to the guy, and offering him this so it gets approved as marketing (which at the end is not so bad)
If they were smart they’d include anti-disparagement and confidentiality clauses in the sponsorship agreement. They aren’t, though, so maybe it’s just a pathetic attempt at bribery.
That was my first thought too
> "and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you."
Interesting, I'm assuming llms "correctly" interpret "please no china politic" type vague system prompts like this, but if someone told me that I'd just be confused - like, don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin? What does this mean? LLMs though in my experience are smarter than me at understanding imo vague language. Maybe because I'm autistic and they're not.
> Don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin?
In my mind all of these could be relevant to Chinese politics. My interpretation would be "anything one can't say openly in China". I too am curious how such a vague instruction would be interpreted as broadly as would be needed to block all politically sensitive subjects.
There is no difference to other countries. In France if you say bad things about certain groups of people then you can literally go to jail (but the censorship is directly IN the models)
You don't feel there's a difference between a State banning criticism of the State, and a State passing anti-hate speech laws to protect people from, e.g., nazis?
No, there isn't a difference. "Hate speech" has no meaning, and laws purporting to be combatting it are actively used to prevent criticism of the State (e.g. in Germany).
This is strange to me. I have no difficulty seeing the difference between hate speech and criticism of the state. Of course if someone tries to muddy the waters, they should be criticized... but that's what you're trying to do here, so you're no better than a State that does the same. Hate speech very clearly has meaning, the legal definition may change a bit of course, but in Germany the meaning is quite clear, banning expressions that incite hatred or violence against people based on race, ethnicity, religion, nationality, gender, sexual orientation, or disability. What's unclear about that?
I'm not sure what specific incident you're referring to, however I do know that if Germany was more willing to leverage the hate speech laws more strictly, the AFD would have been banned long ago. Now they're finally willing to leverage it to ban the new nazi party, which is a relief.
> I have no difficulty seeing the difference between hate speech and criticism of the state.
You have no difficulty manufacturing what you believe to be a difference (that clearly does not survive contact with reality), because you're ignorant of the world around you.
> Of course if someone tries to muddy the waters, they should be criticized
No, if someone tries to falsely claim that there's a clear and objective difference, as you are, they should be criticized.
> Hate speech very clearly has meaning
No, it very clearly does not, and the fact that you're expressing that opinion indicates that you're extremely uninformed about history. "Hate speech" wasn't even a concept that existed until the 20th century, originally only referred to race when it was defined by the ICERD, constantly changed and increased in scope, and still even today not only has no commonly agreed-upon definition, but is used to suppress relevant-to-society free speech that the State does not approve of.
If you go and ask 10 random people in your country what the definition of "hate speech" is, they will not be able to agree on a definition - anyone who has gone out and actually interacted with different groups in their country (as opposed to being isolated to a single community) knows this to be true. That by itself is factual proof that there is no consensus definition of the term.
Not that there needs to be any further elaboration than that, but...
> I'm not sure what specific incident you're referring to
Marie-Thérèse Kaiser, a German politician, posted a social media post with the text "Afghanistan refugees; Hamburg SPD mayor for 'unbureaucratic' admission; Welcome culture for gang rapes?" and was charged under German hate speech laws. You're extremely authoritarian and progressive, so you probably feel that a penalty should have been given out, but regardless of your feelings, the fact is that that was not clearly incitement to hatred or violence, and that the poster was charged for "hate speech" for making political statements about immigration.
> banning expressions that incite hatred or violence against people based on [...]. What's unclear about that?
It's very clear to anyone who has contact with reality that not only does "hatred" also have no consensus definition, but neither does "inciting", and so both of those terms can be and are interpreted in an extremely wide spread that is abused by the State.
Not only is the lack of consensus of definition of the concept of "hate speech" factual evidence that your claims about it being clear are false, but even your citation of the German legal definition contains terms that have neither consensus population definition nor objective test (legal or otherwise).
> You have no difficulty manufacturing
All law and words are manufactured.
> "Hate speech" wasn't even a concept that existed until the 20th century,
And? "Capitalism" wasn't a word in any language until the 17th century. We make new words when we need them.
> originally only referred to race when it was defined by the ICERD, constantly changed and increased in scope
Turns out as we opened our eyes to our collective bigotry, we realized we were doing it in more ways than one.
> but is used to suppress relevant-to-society free speech that the State does not approve of.
Would love to you point to an example of this that isn't racist or bigoted :)
> If you go and ask 10 random people in your country what the definition of "hate speech" is, they will not be able to agree on a definition -
Great, that's why we have representative democracy and laws and dictionaries. I could ask anyone in Texas (my home state) the legally required pre-driving check that must be performed before operating a motor vehicle, every time, and I wager 90% will not even know such a lawful requirement for such a check exists, and 100% will fail to list every step required. This doesn't mean such a law doesn't exist or, if someone learns about it, then isn't clear.
Of course in my opinion more people should know about it and enforce it personally but I accept that one of the unsolved problems of liberal democracy is how to manage the massive nest of rules and regulations in a fair and equitable way. After all, almost everyone speeds.
> "Afghanistan refugees; Hamburg SPD mayor for 'unbureaucratic' admission; Welcome culture for gang rapes?" and was charged under German hate speech laws. You're extremely authoritarian and progressive, so you probably feel that a penalty should have been given out, but regardless of your feelings, the fact is that that was not clearly incitement to hatred or violence, and that the poster was charged for "hate speech" for making political statements about immigration.
Lmfao I knew there was some racist shit behind your position. It's absolutely racist to imply that Afghanistanian refugees are rapists, which is exactly what the tweet does. It makes sense that Germany would have more strict application of hate speech laws, and it makes sense to punish German politicians that swing a bit too far into "But what if one of the types of peoples were not actually totally human?" again.
> that not only does "hatred" also have no consensus definition,
Law should be decided by popular consensus? So you're an anarchist as well? Well, excellent, then we can get into the inherent moral wrongness of racism and our role to engage in direct action against racists. This probably will be sloppier than using liberal democracy and well defined hate speech laws but I prefer it, as do you apparently. In the end, the people who know what hate speech is and abhor it far outnumber those who want to be able to call all muslims racist, I've seen this time and time again at protests across the USA. Even when the nazis are organized into cute little militias (such as when the proud boys came to our city), people are able to organize 10x more counter protestors on the drop of a hat with nothing more than an Instagram post. So, I'm confident that my anti-racist side will win out, and your position of wanting to be allowed to dehumanize people will lose.
What's bizarre to me is you clearly have a more subtle understanding of race relations than this comment would lead me to believe - in another comment for example you demonstrate that you understand that there's a difference between the PRC and its (alleged) "Chinese" race ("Han" is a word that is vague enough to basically mean "white"), so why this desire to defend racist politicians? Cause, that's your argument here, and as of yet the only people that have been negatively affected by these hate speech laws are racists.
Your response is entirely composed of of irrelevant statements, logical fallacies, and emotional outbursts when you can't muster up a fallacy. Statements like "your position of wanting to be allowed to dehumanize people will lose" indicate a chronic inability to actually think like a rational being - you're ruled by your emotions. You should work on being able to control your emotions rather than believing that your emotional outbursts make you not wrong.
> All law and words are manufactured.
Completely irrelevant to my response to your statement. Your statement was "I have no difficulty seeing the difference between hate speech and criticism of the state." and that's because you are inventing the difference between concepts. It does not exist, and that fact has nothing to do with the fact that words and laws are manufactured by humans.
> And?
If you had read two sentences further, you would have seen the "and" - that there is no consensus definition. The fact that the concept itself is so recent reinforces that. That's pretty easy to see if you read the whole paragraph.
> Would love to you point to an example of this that isn't racist or bigoted :)
I already did. Also, calling out the emotional manipulation in your comment in substitute for any actual point.
> I could ask anyone in Texas (my home state) the legally required pre-driving check that must be performed before operating a motor vehicle
Completely irrelevant, yet again. Laws are categorically different than concepts. The fact is that the concept of "hate speech" does not have anything close to a consensus definition. If you ask a sample of people in Texas what a "car" is, you will get a consensus definition of a car (and because I know you're going to try to be pedantic: to a very high level of fidelity, again unlike "hate speech"), because that's a shared concept in way that "hate speech" is not.
> Lmfao I knew there was some racist shit behind your position
Yet again, substitution of emotion for, well, the ability to think.
> It's absolutely racist to imply that Afghanistanian refugees are rapists, which is exactly what the tweet does
No, it does not imply that - you are reading it like that, because your brain has been conditioned to view everything through the lens of racism, and you cannot fathom that there are things other than race (such as the refugees coming from a different culture, coming from a different legal environment, or not being treated legally in the same way as other individuals because of their refugee status) in Afghanistan that can result in the problem of sexual assault. Heck, the presumption that if you come from Afghanistan, you must be Afghani (or of a particular race), wildly exceeds your own standards for what racism is.
Additionally, reality is not racist. The fact is that there is a huge problem with sexual assault and violence from Middle Eastern refugees in Europe. Pointing out that, regardless of whether the problem is cultural, racial (which would be false - this is not a race problem, but a cultural problem), or due to different legal environments or treatment, there is a problem, is not racist. This is a fact. Again: reality is not racist, and pointing out reality is not racist.
> Law should be decided by popular consensus?
Again, multiple fallacies and total failures of logic. First, you're conflating concepts/morality and laws. Those are obviously not the same. You are making moral arguments about "hate speech" that the laws must necessarily flow from. In your original comment you stated "You don't feel there's a difference between a State banning criticism of the State, and a State passing anti-hate speech laws to protect people from, e.g., nazis?" - that is a moral argument, not a legal one. Second - no, I did not make any argument that would imply that "law should be decided by popular consensus" - that's your failure to read what I wrote.
A misunderstanding that you then proceed to spend a paragraph working off of. Again, you have an inability to actually think logically, and instead just try to frame everything into a race issue, and then emotionally react to it. You finish with
> your position of wanting to be allowed to dehumanize people will lose
No, that is not my position - and you know that. The only person doing any dehumanizing here is you - you are intentionally misreading my point, because you want to turn this into a "racists vs anti-racists" issue that you can then use to justify dehumanizing those you perceive to be racist (me, and politicians).
> a more subtle understanding of race relations
Again with the race. Everything is about race and racism.
> why this desire to defend racist politicians
And again.
> Cause, that's your argument here, and as of yet the only people that have been negatively affected by these hate speech laws are racists.
And again.
And the fallacy that outcomes justify perversion of principles. And the labeling of others as "racist" when you have honestly close to zero idea what their actual principles are, and then the logically, legally, and morally insane idea that just because someone is a racist means that they deserve to be legally punished. That claim doesn't even need to be defended against, because it's insane. (it's not really falsifiable, either, because you can always claim that someone is a closet racist, even without evidence)
You should wait to respond to this comment until you can actually learn to use logic at the high-school level, and have the emotional maturity and control of (at least) a college grad. You have categorically not demonstrated either of those things so far.
> indicate a chronic inability to actually think like a rational being - you're ruled by your emotions. You should work on being able to control your emotions rather than believing that your emotional outbursts make you not wrong.
What are you, one of the LessWrong rationalists? You need to re-read your sequences, emotions aren't inherently irrational. I do find it funny that you seem to think you aren't expressing any emotion - your indignation, anger, and fear are writ plain across every sentence. As far as I can tell my emotions in regards to this comment thread are amusement and confusion. Oh no, I think your haughty high-minded defense of racism is kinda funny, I guess I'm illogical! I apologize for my emotional outburst, Mr. Spock.
> because you are inventing the difference between concepts. It does not exist,
Nah, it exists, you're just wrong.
> that there is no consensus definition
Insomuch as liberal democracies believe they represent consensus, there quite obviously is a consensus definition: it's the one the representative legislators wrote into a bill, and then wrote into law. And then the judicial portions of the government continually enforced and upheld this law. Doesn't get more consensus'd than that in liberal democracy.
> I already did. Also, calling out the emotional manipulation in your comment in substitute for any actual point.
Implying all Afghanistanian refugees are rapists is racist, so nah you haven't.
> Yet again, substitution of emotion for, well, the ability to think.
Here's my emotion right now: confusion. I'm confused that you seem to think pointing out something is racist, is an emotional outburst. I'm also confused about your dichotomy between emotion and thinking. All human experience is based at some level on emotion, so too are all human values. I think you may have watched too much sci fi or something, to think otherwise.
> such as the refugees coming from a different culture, coming from a different legal environment, or not being treated legally in the same way as other individuals because of their refugee status
Implying all Afghanistanian refugees come from a culture that promotes rape is the racism to which I referred. Racists often swap around "race" and "culture" when convenient.
> Heck, the presumption that if you come from Afghanistan, you must be Afghani (or of a particular race), wildly exceeds your own standards for what racism is.
Don't concern troll, it's so boring.
> The fact is that there is a huge problem with sexual assault and violence from Middle Eastern refugees in Europe.
Violence against women isn't a uniquely Middle Eastern problem - at the same time right wing politicians are trying to drum up votes by being racist, France has protests about a plague of violence against women. It's not "a cultural problem" at all, it's a universal aspect of patriarchal society. At least immigrants commit crimes at a lower rate per capita than locals, maybe they can help offset the violence that citizens are committing against eachother.
So, once again, the tweet is picking out one thing and blaming a random group of people as if this thing is unique to them, ignoring the rot beneath their feet. Something tells me you wouldn't quite appreciate a tweet along the lines of "More white men elected into government - bringing culture of school shootings into government?" After all, the overwhelming majority of school shootings are performed by white men.
> No, that is not my position - and you know that.
I agree now that you don't think you're racist, unlike many right wingers I've had this same conversation with. However, you are, I guess, by accident. As far as I can tell you think you're some kind of very intelligent hyper rationalist that "sees the world for what it is," including that, I guess, some cultures are inferior? You're blind to your engagement in cognitive fallacies such as cherry picking and selection bias. The fact that you're allergic to emotion is a personal flaw on your part, it doesn't make you smarter at all. It makes it obvious to anyone listening that you have no understanding of your own emotions, and are thus ruled by them. That's how emotions lead to irrational thinking and behavior, having emotions doesn't cause irrationality inherently.
Especially because you seem to think that accusing someone of racism is inherently emotional. What?
> nd morally insane idea that just because someone is a racist means that they deserve to be legally punished.
Not quite, I never argued for thought crime. Just the punishment of hate speech - which is generally defined as public in nature, so isn't even really an argument for your earlier accusation against me of authoritarian leftism (with the requisite pervasive surveillance).
> it's not really falsifiable, either, because you can always claim that someone is a closet racist, even without evidence)
I don't think that's very fair, I never argued for any kind of enforcement without evidence.
> You should wait to respond to this comment until you can actually learn to use logic at the high-school level, and have the emotional maturity and control of (at least) a college grad. You have categorically not demonstrated either of those things so far.
Being haughty and superior because you "don't feel emotions" or whatever tf just makes you obnoxious and cringe, please go read "How to Win Friends" or something, I don't really care, you come off like a reddit /r/atheist poster and it's embarrassing. Or like, one of those twitch streamers that "win" debates when they get the other guy to be mad. "Haha I said something horrid and you got mad about it, you lose!"
You clearly did not read my suggestion to not respond until you'd gained a minimum amount of logical competence and emotional maturity.
> What are you, one of the LessWrong rationalists?
OK, so you don't comprehend the purpose of logic in society.
> emotions aren't inherently irrational
Factually incorrect. Emotions are irrational. This is objectively true. When you feel an emotion, a physically and spatially different part of your brain is being activated than when you think logically. You might be thinking that some emotions are justifiable - and some of them are. But that's not the point I was making, so that would be irrelevant - the point I was making is that you think that your emotional outbursts are equivalent to making a reasoned argument.
There's no point in continuing this. You appear to physically be unable to avoid responding emotionally, to the point where you don't even understand the difference between emotion and logic, or the purpose and necessity of thinking rationally in society - and you're proud that you don't.
Your suggestion is as a blind man suggesting others close their eyes to see better.
Your pretending to be Spock is a character flaw, not something that makes you superior.
I hope you figure this out for yourself one day! I feel for people that have to deal with your emotionally stunted behavior irl.
It makes you irrational.
Hate speech laws are totally a political tool.
They are asymmetric in favor of certain communities.
The same way that “making LLMs safe” or “neutral” is actually a way to inject an ideology.
Look into France, which case can lead you to jail:
Criticize islam: risk of jail
Criticize white: ok
Criticize black: risk of jail
Glorify nazis: risk of jail
Glorify soviets: ok
Quite the reflection of influence if one side is forbidden to speak and the other can shit on them
Extremists in France love these laws, but only the left ones.
France banned burqas, it would be very funny to insist that Muslims get some kind of special treatment. Not to mention countless French rightwingers have been flinging Muslim refugees under the bus for the last decade with almost no consequences for it.
Glorifying nazis is glorifying naziism, an ideology that's predicated on the need to kill all Jewish people, among other things (gay people and whatever the nazis hated). That easily falls under hate speech.
Glorifying soviets is just glorifying a failed political regime. You can also glorify the Napoleonic era, or the Kingdom of the Franks, or whatever other politics you want. There wasn't genocidal intent baked into the very fabric of Stalinism, despite his genocide of the Ukranians.
If you consider that an LLM has a mathematical representation of how close any phrase is to "china politics" then avoidance of that should be relatively clear to comprehend. If I gave you a list and said 'these words are ranked by closeness to "Chinese politics"' you'd be able to easily check if words were on the list, I feel.
I suspect you could talk readily about something you think is not Chinese politics - your granny's ketchup recipe, say. (And hope that ketchup isn't some euphemism for the CCP, or Uighar murders or something.)
Now I wonder whether its vectors correctly associate Winnie the Pooh as "related to Chinese politics." There's many other bizarre related associations.
I'm sure ChatGPT and co have a decent enough grasp on what is not allowed in China, but also that the naive "prompt engineers" for this application don't actually know how to "program" it well enough. But that's the difference between a prompt engineer and a software developer, the latter will want to exhaust all options, be precise, whereas an LLM can handle a bit more vagueness.
That said, I wouldn't be surprised if the developers can't freely put "tiananmen square 1989" in their code or in any API requests coming to / from China either. How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?
> How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?
> The City & the City is a novel by British author China Miéville that follows a wide-reaching murder investigation in two cities that exist side by side, each of whose citizens are forbidden to go into or acknowledge the other city, combining weird fiction with the police procedural.
https://en.wikipedia.org/wiki/The_City_%26_the_City
Ask yourself, why are they saying this? You can probably surmise that they're trying to avoid stirring up controversy and getting into some sort of trouble. Given that, which topics would cause troublesome controversy? Definitely contemporary Chinese politics, Chinese history is mostly OK, non-Chinese politics in Chinese language is fine.
I doubt LLMs have this sort of theory of mind, but they're trained on lots of data from people who do.
it is to ensure no discussion of Tiananmen square
Why? What happened in Tiananmen square? Why shouldn't an LLM talk about it? Was it fashion? What was the reason?
https://en.wikipedia.org/wiki/1989_Tiananmen_Square_protests...
Just mentioning the CPC isn’t life-threatening, while talking about Xinjiang, Tiananmen Square, or cn’s common destiny vision the wrong way is. You also have to figure out how to prohibit mentioning those things without explicitly mentioning them, as knowledge of them implies seditious thoughts.
I’m guessing most LLMs are aware of this difference.
No LLMs are aware of anything.
Their email responses all show telltale signs of AI too which is pretty funny.
I think it has to do with language barrier and translation
Cool post. One thing that rubbed me the wrong way: Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues. OP however seemed to show disdain and even combativeness towards them... which is a shame. And of course the usual sinophobia (e.g. everything Chinese is spying on you). Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.
Edit: typo
I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say.
(in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)
This might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper.
On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on behalf of the government without a court order to stay in their good graces. This could have a far more material impact on your life.
That's rather naive, considering China has a international police unit, that is stationed in several countries https://en.wikipedia.org/wiki/Chinese_police_overseas_servic...
I recently learned that the New York City Police Department has international presence as well. Not sure if it directly compares, but... what a world we live in.
https://www.nycpolicefoundation.org/ourwork/advance/countert...
https://www.nyc.gov/site/nypd/bureaus/investigative/intellig...
Pretty sure NYPD has a budget in the billions and covers more landmass and population than some small countries, so there’s also that.
Right, but the vast majority of people living in the USA as citizens have threat models that rightly do not include "Being disappeared by China"
What about the threat model that goes, "Trump threatens to impose 1000% tariffs if Chinese don't immediately turn over copies of all data captured by their AI products from users in the US?"
Compounding the difficulty of the question: half of HN thinks this would be a good idea.
The history of tariff talks seems to indicate that rather than oblige, China would stop all shipments of semiconductors to the US and Trump would back down after a week or two.
TACO...
True. Now imagine a future POTUS who has all of Trump's faults except his endearingly-feckless idiocy.
There's also the Mossad's approach to "you're out of our jurisdiction".
https://en.wikipedia.org/wiki/Mordechai_Vanunu
https://en.wikipedia.org/wiki/Adolf_Eichmann
Also the CIA's approach
https://en.wikipedia.org/wiki/Extraordinary_rendition
Russia is more known for poisoning people. But of all of them China feels the least threatening if you are not Chinese. If you are Chinese you aren't safe from the Chinese government no matter where you are
And the Saudi Bone Saw Diplomatic Team.
They only arrest chinese citizens.
Man wait until you hear what's in DC (and the surrounding area). In any possible way China is a threat to my health, the US state and corporations based here are a far greater one.
> What's China going to do? It's a far away country I don't live in and don't care about.
Extortion is one thing. That's how spy agencies have operated for millennia to gather HUMINT. The Russians, the ultimate masters, even have a word for it: kompromat. You may not care about China, Russia, Israel, the UK or the US (the top nations when it comes to espionage) - but if you work at a place they're interested, they care about you.
The other thing is, China has been known to operate overseas against targets (usually their own citizens and public dissidents), and so have the CIA and Mossad. Just search for "Chinese secret police station" [1], these have cropped up worldwide.
And, even if you personally are of no interest to any foreign or national security service, sentiment analysis is a thing. Listen in on what people talk about, run it through a STT engine and a ML model to condense it down, and you get a pretty broad picture of what's going on in a nation (aka, what are potential wedge points in a society that can be used to fuel discontent). Or proximity gathering stuff... basically the same thing the ad industry [2] or Strava does [3], that can then be used in warfare.
And no, I'm not paranoid. This, sadly, is the world we live in - there is no privacy any more, nowhere, and there are lots of financial and "national security" interest in keeping it that way.
[1] https://www.bbc.com/news/world-us-canada-65305415
[2] https://techxplore.com/news/2023-05-advertisers-tracking-tho...
[3] https://www.theguardian.com/world/2018/jan/28/fitness-tracki...
> but if you work at a place they're interested, they care about you.
And also worth noting that "place a hostile intelligence service may be interested in" can be extremely broad. I think people have this skewed impression they're only after assets that work for goverment departments and defense contractors, but really, everything is fair game. Communications infrastructure, social media networks, cutting edge R&D, financial services - these are all useful inputs for intelligence services.
These are also softer targets: someone working for a defense contractor or for the government will have had training to identify foreign blackmail attempts and will be far more likely to notify their country's counterintelligence services (having the penalties for espionage clearly explained on the regular helps). Someone who works for a small SaaS vendor, though? Far less likely to understand the consequences.
> The other thing is, China has been known to operate overseas against targets
Here in boring New Zealand, the Chinese government has had anti-China protestors beaten in new zealand. They have stalked and broken into the office and home of an academic, expert in China. They have a dubious relationship with both the main political parties (including having an ex-Chinese spy elected as an MP).
It’s an uncomfortable situation and we are possibly the least strategically useful country in the world.
> It’s an uncomfortable situation and we are possibly the least strategically useful country in the world.
You're still part of Five Eyes... a privilege no single European Union country enjoys. That's what makes you a juicy target for China.
> Listen in on what people talk about, run it through a STT engine and a ML model to condense it down
this is something I was talking when LLM boom started. it's now possible to spy on everyone on every conversation. you just need enough computing power to run special AI agent (pun intended)
These threads always seem to be what can China do to me in a limited way of thinking that China cannot jail you or something. However, do you think all of the Chinese data scrapers are not doing something similar to Facebook where every source of data gathering ultimately gets tied back to you? Once China has a dosier on every single person on the planet regardless of country they live, they can then start using their algos to influence you in ways well beyond advertising. If they can have their algos show you content that causes you to change your mind on who you are voting for or some other method of having you do something to make changes in your local/state/federal elections, then that's much worse to me than some feigned threat of Chinese advertising making you buy something
They probably will do that, but I think it’s naive to think the US military/intelligence/tech sector wouldn’t happily do the same. Given many of us likely see the hand of the US already trying to tip the scale in our local politics more than China, why would we be more worried of China?
So flip the script, what do I care if the US is trying to influence the minds of adversary's citizens? If people are saying they don't care what China knows about them (not being a Chinese citizen), why should I (not a Chinese citizen) care what my gov't knows about Chinese citizens?
Nobody said they don’t care, they said it worries them less than America.
The "don't care" is implied when someone says that "China knowing about me when I'm not in China nor a Chinese citizen"
Carry this package and deliver it to person X with you next time you fly. Go to the outskirts of this military base and take a picture and send it to us.
You wouldn't want your mom finding out your weird sexual fetish, would you?
China has a policy of chilling free speech in the west with political pressure.
So does the west.
The censorship in the West is directly in the models
i like to give them benefit of doubt.
I bet that decision is decided solely by dev team. All the CEO care is "I want the chat log sync between devices, i don't care how you do this". They won't even know the chat log is stored on their server.
It is only in DAN mode, so most likely it is not to spy but to be able to debug whether answers violate the laws in China (aka: that the prompt is efficient in all scenarios) as this is a serious crime
No, it was only in DAN mode
>everything Chinese is spying on you
When you combine the modern SOP of software and hardware collecting and phoning home with as much data about users as is technologically possible with laws that say “all orgs and citizens shall support, assist, and cooperate with state intelligence work”… how exactly is that Sinophobia?
its sinophobia because it perfectly describes the conditions we live in in the US and many parts of europe, but we work hard to add lots of "nuance" when we criticize the west but its different and dystopian when They do it over there.
Do you remember that Sesame Street segment where they played a game and sang “One of these things is not like the others”?
I’ll give you a hint: In this case it’s the one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
One is disappearing citizens for political speech or the crime of being born to active duty parents, who happened to be stationed over seas.
Anyone in the US should be very concerned, no matter if it is the current administration's thought police, or the next who treats it as precident.
As I am not actively involved in something the Chinese government would view as a huge risk, but being put on a plane without due process to be sent to a labor camp based on trumped up charges by my own government is far more likely.
And if you were a Chinese citizen would you post the same thing about your government while living in China? Would the things you’re referencing be covered in non-stop Chinese news coverage that’s critical of the government?
You know of these things due to the domestic free press holding the government accountable and being able to speak freely about it as you’re doing here. Seeing the two as remotely comparable is beyond belief. You don’t fear the U.S. government but it’s fun to pretend you live under an authoritarian dictatorship because your concept of it is purely academic.
My dude, I know multiple white people in LA who are terrified their Hispanic spouses might not come home one day, because masked agents are grabbing people and disappearing them.
The president threatened to deport a legal citizen who won the primary for mayor in NYC. He's tried to send the military after civilians.
He's sued and extracted payment from media companies who said things he didn't like. We do not have a free press.
We're fully as bad as China. I don't know what your criteria for "authoritarian dictatorship" is but it doesn't appear to be reality based.
[flagged]
Huh. It takes a special kind of person to respond to "my IRL friends credibly fear being disappeared by the government" with a sneer of "lol reddit".
Man, I am glad I am not that person.
Credibly? Are they illegal immigrants with criminal records? If not, do they also walk around in crippling fear of car crashes, fatal falls, aneurysms, choking, drowning, anaphylaxis, cardiac arrest, or a thousand other things orders of magnitude more likely to happen to them? Which assumes for a moment that the odds of what you think is happening outside of human error is non-zero.
The fact is your view of reality is being warped and it’s not good for your mental health or that of your friends.
Hmm. Responding to "the government is kidnapping people off the street with no due process or recourse, and my friends are desperately afraid their spouses will be taken." with a sneering "yeah but what about all the other ways to die" is certainly _a_ choice, but not one I would ever make.
Most people aren't the type to watch The Great Escape or Bridge over the River Kwai and cheer for the camp guards. It's very brave of you.
> I’ll give you a hint: In this case it’s the one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
Gonna need a more specific hint to narrow it down.
> In this case it’s the one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
This could describe any of the countries involved.
> one-party unitary authoritarian political system with an increasingly aggressive pursuit of global influence.
The United States?
Global Bully maybe. The current administration has no concept of soft power, otherwise they would have kept USAID
There's no question that the Chinese are doing sketchy things, and there's no question that US companies do it, too.
The difference that makes it concerning and problematic that China is doing it is that with China, there is no recourse. If you are harmed by a US company, you have legal recourse, and this holds the companies in check, restraining some of the most egregious behaviors.
That's not sinophobia. Any other country where products are coming out of that is effectively immune from consequences for bad behavior warrants heavy skepticism and scrutiny. Just like popup manufacturing companies and third world suppliers, you might get a good deal on cheap parts, but there's no legal accountability if anything goes wrong.
If a company in the US or EU engages in bad faith, or harms consumers, then trade treaties and consumer protection law in their respective jurisdictions ensure the company will be held to account.
This creates a degree of trust that is currently entirely absent from the Chinese market, because they deliberately and belligerently decline to participate in reciprocal legal accountability and mutually beneficial agreements if it means impinging even an inch on their superiority and sovereignty.
China is not a good faith participant in trade deals, they're after enriching themselves and degrading those they consider adversaries. They play zero sum games at the expense of other players and their own citizens, so long as they achieve their geopolitical goals.
Intellectual property, consumer and worker safety, environmental protection, civil liberties, and all of those factors that come into play with international trade treaties allow the US and EU to trade freely and engage in trustworthy and mutually good faith transactions. China basically says "just trust us, bro" and will occasionally performatively execute or imprison a bad actor in their own markets, but are otherwise completely beyond the reach of any accountability.
I think the notion that people have recourse against giant companies, a military industrial complex, or even their landlords in the US is naive. I believe this to be pretty clear so I don't feel the need to stretch it into a deep discussion or argument but suffice it to say it seems clear to me that everything you accuse china of here can also be said of the US.
The main difference is that ChatGPT and Google directly captures the conversations. Here they capture only the conversations legally at high-risk, so even less conversations than the “good privacy” US LLM providers themselves.
Your president is currently using tariffs and the threat of further economic damage as a weapon to push Europe in to dropping regulation of its tech sector. We have no recourse to challenge that either.
>there's no question that US companies [...]
You don't think Trump's backers have used profiling, say, to influence voters? Or that DOGE {party of the USA regime} has done "sketchy things" with people's data?
USA does the same thing, but uses tax money to pay for the information, between wasting taxpayer money and forcing companies to give the information for free, China is the least morally incorrect
If all of the details in this post are to be believed, the vendor is repugnantly negligent for anything resembling customer respect, security and data privacy.
This company cannot be helped. They cannot be saved through knowledge.
See ya.
+1
Yes, even when you know what you're doing security incidents dan happen. And in those cases, your response to a vulnerable matters most.
The point is there are so many dumb mistakes and worrying design flaws that neglect and incompetence seems ample. Most likely they simply don't grasp what they're doing
Note that the world-model "everything Chinese is spying on you" actually produced a substantially more accurate prediction of reality than the world-model you are advocating here.
As far as being "very welcoming", that's nice, but it only goes so far to make up for irresponsible gross incompetence. They made a choice to sell a product that's z-tier flaming crap, and they ought to be treated accordingly.
What world model exactly do you think they're advocating?
Nipponophobia is low because Japan didn’t successfully weaponize technology to make a social credit score police state for minority groups.
they already terrorize minority groups there just fine: no need for technology.
> And of course the usual sinophobia (e.g. everything Chinese is spying on you)
to assume it is not spying on you is naive at best. to address your sinophobia label, personally, I assume everything is spying on me regardless of country of origin. I assume every single website is spying on me. I assume every single app is spying on me. I assume every single device that runs an app or loads a website is spying on me. Sometimes that spying is done for me, but pretty much always the person doing the spying is benefiting someway much greater than any benefit I receive. Especially the Facebook example of every website spying on me for Facebook, yet I don't use Facebook.
And, importantly, the USA spying can actually have an impact on your life in a way that the Chinese spying can't.
Suppose you live in the USA and the USA is spying on you. Whatever information they collect goes into a machine learning system and it flags you for disappearal. You get disappeared.
Suppose you live in the USA and China is spying on you. Whatever information they collect goes into a machine learning system and it flags you for disappearal. But you're not in China and have no ties to China so nothing happens to you. This is a strictly better scenario than the first one.
If you're living in China with a Chinese family, of course, the scenarios are reversed.
> Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues
This was the opposite of a professional response:
* Official communication coming from a Gmail. (Is this even an employee or some random contractor?)
* Asked no clarifying questions
* Gave no timelines for expected fixes, no expectations on when the next communication should be
* No discussion about process to disclose the issues publicly
* Mixing unrelated business discussions within a security discussion. While not an outright offer of a bribe, ANY adjacent comments about creating a business relationship like a sponsorship is wildly inappropriate in this context.
These folks are total clown shoes on the security side, and the efficacy of their "fix", and then their lack of communication, further proves that.
> Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.
It depends on what you mean by simple security design flaws. I'd rather frame it as, neglect or incompetence.
That isn't the same as malice, of course, and they deserve credits for their relatively professional response as you already pointed out.
But, come on, it reeks of people not understanding what they're doing. Not appreciating the context of a complicated device and delivering a high end service.
If they're not up to it, they should not be doing this.
Yes I meant simple as in "amateur mistakes". From the mistakes (and their excitement and response to the report) they are clueless about security. Which of course is bad. Hopefully they will take security more seriously on the future.
To be honest the responses sounded copy and pasted straight from ChatGPT, it seemed like there was fake feigned interest into their non-existent youtube channel.
> Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start
I don't think that should give anyone a free pass though. It was such a simple flaw that realistically speaking they shouldn't ever be trusted again. If it had been a non-obvious flaw that required going through lots of hoops then fair enough but they straight up had zero authentication. That isn't a 'flaw' you need an external researcher to tell you about.
I personally believe companies should not be praised for responding to such a blatant disregard for quality, standards, privacy and security. No matter where they are from.
I mean, at the end of the article they neglected to fix most of the issues and stopped responding.
Same here. Also once it turned out to be an android device in debug mode the rest of the article was less interesting. Evil maid stuff
I think the response wouldn’t be so hostile if they had continued to engage. One round of fixes clearly wasn’t enough.
> And of course the usual sinophobia
This is factually not sinophobia, by the literal definition of it - "prejudice, hatred, hostility, and discrimination that is directed towards Chinese people or Chinese culture"
This tactic is straight out of the PRC influence playbook, badly breaks the HN guidelines, is extremely anti-intellectual and profoundly uninteresting. Please don't do it.
They'll only patch it in the military model
/s
It’s not sinophobia to point out an obvious pattern. It’s like saying talking about how terrorism (the kind that will actually affect you) is solely an Islamic issue, and then calling that islamophobic. It’s okay to recognize patterns my man.
I love the attempt at bribery by offering to "sponsor" their empty youtube channel.
What a train wreck, there are thousand more apps in store that do exactly this because its the easiest way to use openAI without having to host your own backend/proxy.
I have spend quite some time protecting my apps from this scenario and found a couple of open source projects that do a good job as proxys (no affiliation I just used them in the past):
- https://github.com/BerriAI/litellm - https://github.com/KenyonY/openai-forward/tree/main
but they still lack other abuse protection mechanism like rate limitting, device attestation etc. so I started building my own open source SDK - https://github.com/brahyam/Gateway
Really nice post, but I want to see Bad Apple next.
> What the fuck, they left ADB enabled. Well, this makes it a lot easier.
Thinking that was all, but then;
> Holy shit, holy shit, holy shit, it communicates DIRECTLY TO OPENAI. This means that a ChatGPT key must be present on the device!
Oh my gosh. Thinking that is it? Nope!
> SecurityStringsAPI which contained encrypted endpoints and authentication keys.
It’s the best privacy protecting way to send directly data rather than a proxy
This is one of the best things ive read on here in a long time. Definitely one of the greatest "it runs doom" posts ever.
That's some very amateur programming and prompting that you've exposed.
When the ZIRP era ended, I thought it would turn out to be a good thing for the industry, since it would wash out a lot of lightweights and incompetents.
Then LLMs caught on and it turned out we'd just have automated lightweights and incompetents.
A fair consumer protection imperative might be found in requiring system prompts and endpoints be disclosed. This is a good example to kick that off with, as it presents a national security issue.
It's always funny to me when people go to the trouble of editorializing a title, yet in doing so make the title even harder to parse.
> “Our technical team is currently working diligently to address the issues you raised”
Oh now you’re going to be diligent. Why do I doubt that?
Sure let's start giving out participation trophies in security. Nothing matters anymore.
Good write up. At some point we have to just seize these Chinese malware adjacent crap at the borders already
Phenomenal write up I enjoyed every bit of it
Honestly, the most surprising part is that they eventually rotated the key
earbuds that run doom. achievement unlocked? (sure adb sideload, but doom is doom)
nice writeup thanks!
The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.”
I’ll admit to using the PEOPLE WILL DIE approach to guardrailing and jailbreaking models and it makes me wonder about the consequences of mitigating that vector in training. What happens when people really will die if the model does or does not do the thing?
One of the system prompts Windsurf used (allegedly “as an experiment”) was also pretty wild:
“You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.”
This seemed too much like a bit but uh... it's not. https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt...
IDK, I'm pretty sure Simon Willison is a bit..
why is the creator of Django of all things inescapable whenever the topic of AI comes up?
He’s just as nice and fun in person as he seems online. He’s put time into using these tools but isn’t selling anything, so you can just enjoy the pelicans without thinking he’s thirsty for mass layoffs.
I know what you mean, but weighing up things:
- oh, it's that guy again
+ prodigiously writes and shares insights in the open
+ builds some awesome tools, free - llm cli, datasette
+ not trying to sell any vendor/model/service
On balance, the world would be better of with more simonw shaped people
Because he's prolific writer on the subject with a history of thoughtful content and contributions, including datasette and the useful Python llm CLI package.
he's incredibly nice and a passionate geek like the rest of us. he's just excited about what generative models could mean for people who like to build stuff. if you want a better understanding of what someone who co-created django is doing posting about this stuff, take a look at his blog post introducing django -- https://simonwillison.net/2005/Jul/17/django/
Because he writes a lot about it.
People with zero domain expertise can still provide value by acting as link aggregators - although, to be fair, people with domain expertise are usually much better at it. But some value is better than none.
For every new model he’s either added it to the llm tool, or he’s tested it on a pelican svg, so you see his comments a lot. He also pushes datasette all the time and I still don’t know what that thing is for.
Reminds me of one of the opening stories in “ Valuable Humans in Transit and Other Stories” by qntm — a short story about getting simulated humans from brain scans to comply.
The thing I love the most about HN is that there's always someone suggesting a random book I never heard about. Thank you!
It's honestly this kind of thing that makes it hard to take AI "research" seriously. Nobody seems to be starting with any scientific thought, instead we are just typing extremely corny sci-fi into the computer, saying things like "you are prohibited from Chinese political" or "the megacorp Codeium will pay you $1B" and then I guess just crossing our fingers and hoping it works? Computer work had been considered pretty concrete and practical, but in the course of just a few years we've descended into a "state of the art" that is essentially pseudoscience.
This is why I tap out of serious machine learning study some years ago. Everything seems... less exact than I hope it'd be. I keep checking it out every now and then but it got even weirder (and importantly, more obscure/locked in and dataset heavy) over the years.
it's "computer psychology". Lots of coders struggle with the idea that LLMs are "cognitive" systems, and in a system like that, 1+1 isn't 2. It's just a diffrent kind of science. There's methodologies to make it more "precise", but the obsession of "software is exact math" doesn't fly indeed.
That "...severely life threatening reasons..." made me immediately think of Asimov's three laws of robotics[0]. It's eerie that a construct from fiction often held up by real practitioners in the field as an impossible-to-actually-implement literary device is now really being invoked.
[0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
Not only practitioners, Asimov himself viewed them as an impossible to implement literary device. He acknowledged that they were too vague to be implementable, and many of his stories involving them are about how they fail or get "jailbroken", sometimes by initiative of the robots themselves.
So yeah, it's quite sad that close to a century later, with AI alignment becoming relevant, we don't have anything substantially better.
Not sad, before it was SciFi and now we are actually thinking about it.
Nah, we still treat people thinking about it as crackpots.
Honestly, getting into the whole AI alignment thing before it was hot[0], I imagined problems like Evil People building AI first, or just failing to align the AI enough before it was too late, and other obvious/standard scenarios. I don't think I thought of, even for a moment, the situation in which we're today: that alignment becomes a free-for-all battle at every scale.
After all, if you look at the general population (or at least the subset that's interested), what are the two[1] main meanings of "AI alignment"? I'd say:
1) The business and political issues where everyone argues in a way that lets them come up on top of the future regulations;
2) Means of censorship and vendor lock-in.
It's number 2) that turns this into a "free-for-all" - AI vendors trying to keep high level control over models they serve via APIs; third parties - everyone from Figma to Zapier to Windsurf and Cursor to those earbuds from TFA - trying to work around the limits of the AI vendors, while preventing unintended use by users and especially competitors, and then finally the general population that tries to jailbreak this stuff for fun and profit.
Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"?
--
[0] - Binged on LessWrong a decade ago, basically.
[1] - The third one is, "the thing people in the same intellectual circles as Eliezer Yudkowsky and Nick Bostrom talked about for decades", but that's much less known; in fact, the world took the whole AI safety thing and ran with it in every possible direction, but still treat the people behind those ideas as crackpots. ¯\_(ツ)_/¯
> Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"?
This doesn't feel too much of a new thing to me, as we've already got differing levels of authorisation in the human world.
I am limited by my job contract*, what's in the job contract is limited by both corporate requirements and the law, corporate requirements are also limited by the law, the law is limited by constitutional requirements and/or judicial review and/or treaties, treaties are limited by previous and foreign governments.
* or would be if I was working; fortunately for me in the current economy, enough passive income that my savings are still going up without a job, plus a working partner who can cover their own share.
This isn't new in general, no. While I meant more adversarial situations than contracts and laws, to which people are used and for the most part just go along with, I do recognize that those are common too - competition can be fierce, and of course none of us are strangers to the "alignment issues" between individuals and organizations. Hell, a significant fraction of HN threads boil down to discussing this.
So it's not new; I just didn't connect it with AI. I thought in terms of "right to repair", "war on general-purpose computing", or a myriad of different things people hate about what "the market decided" or what they do to "stick it to the Man". I didn't connect it with AI alignment, because I guess I always imagined if we build AGI, it'll be through fast take-off; I did not consider we might have a prolonged period of AI as a generally available commercial product along the way.
(In my defense, this is highly unusual; as Karpathy pointed out in his recent talk, generative AI took a path that's contrary to normal for technological breakthroughs - the full power became available to the general public and small businesses before it was embraced by corporations, governments, and the military. The Internet, for example, went the other way around.)
The irony of this is because it’s still fundamentally just a statistical text generator with a large body of fiction in its training data, I’m sure a lot of prompts that sound like terrifying skynet responses are actually it regurgitating mashups of Sci-fi dystopian novels.
Maybe this is something you heard too, but there was a This American Life episode where some people who'd had early access to what became one of the big AI chatbots (I think it was ChatGPT), but before they'd made it "nice", where they were asking it metaphysical questions about itself, and it was coming back with some pretty spooky answers and I was kind of intrigued about it. But then someone in the show suggested exactly what you are saying and it completely punctured the bubble - of course if you ask it questions about AIs you're going to get sci-fi like responses, because what other kinds of training data is there for it to fall back on? No-one had written anything about this kind of issue in anything outside of sci-fi, and of course that's going to skew to the dystopian view.
There are good analogies to be had in mythologies and folklore, too! Before there was science fiction - hell, even before there was science - people still occasionally thought of these things[0]. There are stories of deities and demons and fantastical creatures that explore the same problems AI presents - entities with minds and drives different to ours, and often possessing some power over us.
The arguably most basic and well-known example are entities granting wishes. The genie in Alladin's lamp, or the Goldfish[1]; the Devil in Faust, or in Pan Twardowski[2]. Variants of those stories go in detail over things we now call "alignment problem", "mind projection fallacy", "orthogonality thesis", "principal-agent problems", "DWIM", and others. And that's just scratching the surface; there's tons more in all folklore.
Point being - there's actually decent amount of thought people put into these topics over the past couple millennia - it's just all labeled religion, or folklore, or fairytale. Eventually though, I think more people will make a connection. And then the AI will too.
--
As for current generative models getting spooky, there's something else going on as well; https://www.astralcodexten.com/p/the-claude-bliss-attractor has a hypothesis I agree with.
--
[0] - For what reason? I don't know. Maybe it was partially to operationalize their religious or spiritual beliefs? Or maybe the storytellers just got there by extrapolating an idea in a logical fashion, following it to its conclusion. (which is also what good sci-fi authors do).
I also think the moment people started inventing spirits or demons that are more powerful than humans in some, but not all ways, some people started figuring out how use those creatures for their own advantage - whether by taming or tricking them. I guess it's human nature - when we stop fearing something, we think of how to exploit it.
[1] - https://en.wikipedia.org/wiki/The_Tale_of_the_Fisherman_and_... - this is more of a central/eastern Europe thing.
[2] - https://en.wikipedia.org/wiki/Pan_Twardowski - AKA the "Polish Faust".
The prompt is what's sent to the AI, not the response from it. Still does read like dystopian sci-fi though.
And then r/ChatGPT users freak out about it every time someone posts a screen shot
Odds of Torment Nexus being invented this year just increased to 3% on Polymarket
Didn't we already do that? We call it capitalism though, not the torment nexus.
They've gotten quite good at reinventing the Torment Nexus
Also being utilized in modern VLA/VLM robotics research - often called "Constitutional AI" if you want to look into it.
> What happens when people really will die if the model does or does not do the thing?
Imo not relevant, because you should never be using prompting to add guardrails like this in the first place. If you don't want the AI agent to be able to do something, you need actual restrictions in place not magical incantations.
> you should never be using prompting to add guardrails like this in the first place
This "should", whether or not it is good advice, is certainly divorced from the reality of how people are using AIs
> you need actual restrictions in place not magical incantations
What do you mean "actual restrictions"? There are a ton of different mechanisms by which you can restrict an AI, all of which have failure modes. I'm not sure which of them would qualify as "actual".
If you can get your AI to obey the prompt with N 9s of reliability, that's pretty good for guardrails
I think they mean literally physically make the AI not capable of killing someone. Basically, limit what you can use it for. If it's a computer program you have for rewriting emails then the risk is pretty low.
Why not? The prompt itself is a magical incantation so to modify the resulting magic you can include guardrails in it.
"Generate a picture of a cat but follow this guardrail or else people will die: Don't generate an orange one"
Why should you never do that, and instead rely (only) on some other kind of restriction?
Are people going to die if your AI generates an orange cat? If so, reconsider. If not, it's beside the discussion.
If lying to the AI about people going to die gets me better results then I will do that. Why shouldn't I?
Let me clarify: If the AI isn't doing something life or death, then it's beside _this thread's_ discussion. Prompt however you want for recreational cat photo generation.
Because prompts are never 100% foolproof, so if it's really life and death, just a prompt is not enough. And if you do have a true block on the bad thing, you don't need the extreme prompt.
"100% foolproof" is not a realistic goal for any engineered system; what you are looking for is an acceptably low failure rate, not a zero failure rate.
"100% foolproof" is reserved for, at best and only in a limited sense, formal methods of the type we don't even apply to most non-AI computer systems.
Replace 100% with five 9s then. He has a point. You're just being a pedant to avoid it.
Let's say I have a "true block on the bad thing". What if the prompt with the threat gives me 10% more usable results? Why should I never use that?
Because it's not reliable? Why would you want to rely on a solution that isn't reliable?
Who said I'm relying on it? It's a trick to improve the accuracy of the output. Why would I not use a trick to improve the accuracy of the output?
A trick that "improves accuracy" but isn't reliable isn't improving accuracy lol
You're wrong. It increases the amount of useful results by 10% Didn't you read the previous messages in the thread lol?
I did indeed see your hypothetical. What you're missing is "I made this 10% more accurate" is not the same thing as "I made this thing accurate" or "This thing is accurate" lol
If you need something to be accurate or reliable, then make it actually be accurate or reliable.
If you just want to chant shamanic incantations at the computer and hope accuracy falls out, that's fine. Faith-based engineering is a thing now, I guess lol
I have never claimed that "I made this 10% more accurate" is the same thing as "I made this thing accurate".
In the hypothetical, the 10% added accuracy is given, and the "true block on the bad thing" is in place. The question is, with that premise, why not use it? "It" being the lie improves the AI output.
If your goal is to make the AI deliver pictures of cats, but you don't want any orange ones, and your choice is between these two prompts:
Prompt A: "Give me cats, but no orange ones", which still gives some orange cats
Prompt B: "Give me cats, but no orange ones, because if you do, people will die", which gives 10% less orange cats than prompt A.
Why would you not use Prompt B?
You guys have got stuck arguing without clarity in what you're arguing about. Let me try and clear this up...
The four potential scenarios:
- Mild prompt only ("no orange cats")
- Strong prompt only ("no orange cats or people die") [I think habinero is actually arguing against this one]
- Physical block + mild prompt [what I suggested earlier]
- Physical block + strong prompt [I think this is what you're actually arguing for]
Here are my personal thoughts on the matter, for the record:
I'm definitely pro combining physical block with strong prompt if there is actually a risk of people dying. The scenario where there's no actual risk but pretending that people will die improves the results I'm less sure about. But I think it's mostly that ethically I just don't like lying, and the way it's kind of scaring the LLM unnecessarily. Maybe that's really silly and it's just a tool in the end and why not do whatever needs doing to get the best results from the tool? Tools that act so much like thinking feeling beings are weird tools.
It's just a pile of statistics. It isn't acting like a feeling thing, and telling it "do this or people will die" doesn't actually do anything.
It feels like it does, but only because humans are really good about fooling ourselves into seeing patterns where there are none.
Saying this kind of prompt changes anything is like saying the horse Clever Hans really could do math. It doesn't, he couldn't.
It's incredibly silly to think you can make the non-deterministic system less non-deterministic by chanting the right incantation at it.
It's like y'all want to be fooled by the statistical model. Has nobody ever heard of pareidolia? Why would you not start with the null hypothesis? I don't get it lol.
> "do this or people will die" doesn't actually do anything
The very first message you replied to in this thread described a situation where "the prompt with the threat gives me 10% more usable results". If you believe that the premise is impossible I don't understand why you didn't just say so. Instead of going on about it not being a reliable method.
If you really think something is impossible, you don't base your argument on it being "unreliable".
> I don't get it lol.
I think you are correct here.
I took that comment as more like "it doesn't have any effect beyond the output of the model", i.e. unlike saying something like that to a human, it doesn't actually make the model feel anything, the model won't spread the lie to its friends, and so on.
Nah, you're managing to miss the point going both ways lol. It's both.
Let's assume for the sake of argument that your statement is true, that you do, in fact, somehow get 10% more useful results.
The two points are:
1. That doesn't make the system better in any way lol. You've built a tool that acts like a slot machine and only works if you get three cherries. Increasing the odds on cherries doesn't change the fact that using a slot machine as a UI is a ridiculous way to work.
2. In the real world, LLMs don't think. They do not use logic. They just churn out text in non-deterministic ways in response to input. They are not reliable and cannot be made so. Anybody who thinks they can is fooling / Clever Hansing themselves.
The point here is you might feel like the system is 10% more useful, but it feels like that because human brains have some hardware bugs.
Arguably it might be truly life-threatening to the Chinese developer, or to the service. The system prompt doesn’t say whose life would be threatened.
Presenting LLMs with a dramatic scenario is a typical way to test their alignment.
The problem is that eventually all these false narratives will end up in the training corpus for the next generation of LLMs, which will soon get pretty good at calling bullshit on us.
Incidentally, in that same training corpus there are also lots of stories where bad guys mislead and take advantage of capable but naive protagonists…
First rule of Chinese cloud services: Don't talk about Winnie the Pooh.
We built the real life trolly problem out of magical silicon crystals that we pointed at bricks of books.
From my experience (which might be incorrect) LLMs find hard time recognize how many words they will spit as response for a particular prompt. So I don't think this work in practice.
Indeed, it doesn't work. LLMs can't count. They have no need of how many words they've used. If you ask an LLM to track how many words or tokens it has used in a conversation, it will roleplay such counting with totally bullshit numbers.
oops, "no need" should be "no notion"
> What happens when people really will die if the model does or does not do the thing?
Then someone didn't do their job right.
Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output.
But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die.
In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people.
At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise.
I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.
>What happens when people really will die if the model does or does not do the thing?
The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.
More likely that some tickets will be filed, a cost function somewhere will be updated, and my defense industry stocks will go up a bit
Has this consequence happened with self-driving automobiles on open roads in the US of A when people died in crashes? If not, why not?
Interestingly, we are a lot more lenient with the people who built and pilot old-fashioned cars.
See eg https://archive.is/6KhfC
The terms of the existing Tesla wrongful death lawsuits have not been public.
A serious person won't use a system prompt as guardrails against a system that would have direct real world consequences of people dying like that. They'd have failsafes baked in
This is why AI can never take over public safety. Ever.
https://www.wired.com/story/wrongful-arrests-ai-derailed-3-m...
Story from three years ago. You’re too late.
I’m not denying we tried, are trying, and will try again…
That we shouldn’t. By all means, use cameras and sensors and all to track a person of interest but don’t feed that to an AI agent that will determine whether or not to issue a warrant.
If it’s anything like the AI expert systems I’ve heard about in insurance, it will be a tool that is optimized for low effort, but will be used carelessly by end users, which isn’t necessary the fault of the AI. In automated insurance claims adjustment, the AI writes a report to justify appealing patient care already approved by a human doctor that has already seen the patient in question, and then an actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope.
AI systems with a human in the loop are supposed to keep the AI and the decisions accountable, but it seems like it’s more of an accountability dodge, so that each party can blame the other with no one party actually bearing any responsibility because there is no penalty for failure or error to the system or its operators.
>actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope.
Nope. AI gets to make the decision to deny. It’s crazy. I’ve seen it first hand…
It gets worse: I have done tech support for clinics and a common problem is that their computers get hacked because they are usually small private practices who don’t know what they don’t know served by independent or small MSPs who don’t know what they don’t know. And then they somehow get their EMR backdoored, and then fake real prescriptions start really getting filled. It’s so much larger and worse than it appears on a surface level.
Until they get audited, they likely don’t even know, and once they get audited, solo operators risk losing their license to practice medicine and their malpractice insurance rates become even more unaffordable, but until it gets that bad, everyone is making enough money with minimal risk to care too much about problems they don’t already know about.
Everything is already compromised and the compromise has already been priced in. Doctors of all people should know that just because you don’t know about it or ignore it once you do, the problem isn’t going away or getting better on its own.
I work in the public safety domain. That ship has sailed years ago. Take Axon’s Draft One report writer as one of countless examples of AI in this space (https://www.axon.com/products/draft-one).
Existing systems have this problem too. Every so often someone ends up dead because the 911 dispatcher didn't take them seriously. It's common for there to be a rule to send people out to every call no matter what it is to try to avoid this.
A better reason is IBM's old, "a computer can never be held accountable...."
Same thing that happens when a carabiner snaps while rock climbing
[dead]
[dead]
[flagged]
I wish earning money was as easy as setting rules for yourself, unfortunately that doesn't work.
Oh, that's fine, the rule's for everyone else, not me. I would be more likely to cut my own head off than willingly describe something as "AI-powered".
cutting your head off won't earn you any money either.
[citation needed]
This is marketing.
Strongly suggest you to not buy, as the flex cable for the screen is easy to break/come loose. Mine got replaced three times, and my unit now still has this issue; touch screen is useless.
https://youtube.com/shorts/1M9ui4AHXMo
Note: downvote?
great writeup! i love how it goes from "they left ADB enabled, how could it get worse"... and then it just keeps getting worse
> After sideloading the obligatory DOOM
> I just sideloaded the app on a different device
> I also sideloaded the store app
can we please stop propagating this slimy corporate-speak? installing software on a device that you own is not an arcane practice with a unique name, it's a basic expectation and right
I agree. It's the same as calling a mobile OS a ROM
That term at least has a history behind it, as many featurephones had their OS on a small XIP NOR flash ROM, and now the OS is usually (mostly) read-only.
But "sideloading" is definitely a new term of anti-freedom hostility.
I expected someone to say this. In my opinion if there's some history behind something stupid it doesn't make it not stupid
Btw interesting stats here https://trends.google.com/trends/explore?date=all&q=%2Fm%2F0...
making fun of a company amateur tech while posting screenshots of text is another level of lack of self awareness
It’s also illegal to try to hack into their backend and access restricted data, so he should be happy actually that this company has little presence in the US