> They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad
I've only seen John Carmack's public interactions, but they've all been professional and kind.
It's depressing to imagine HR getting involved because someone's feelings had been hurt by an objective discussion from a person like John Carmack.
I'm having flashbacks to the times in my career when coworkers tried to weaponize HR to push their agenda. Every effort was eventually dismissed by HR, but there is a chilling effect on everyone when you realize that someone at the company is trying to put your job at stake because they didn't like something you said. The next time around, the people targeted are much more hesitant to speak up.
I followed his posts internally before he left. He was strict about resource waste. Hand tracking would break constantly and he brought metrics to his posts. His whole point was that Apple has hardware nailed down and it’ll be efficient software that will be the differentiator. The bloat at Meta was the result of empire building.
I remember watching Carmack at a convention 15 years ago. He took a short sabbatical and came back with ID Tech 3 on an iPhone, and it still looks amazing well over a decade later.
This is a guy who figures that what he wants to do most with his 3 free weekends is to port his latest, greatest engine to a Cortex-A8. Leading corporate strategy? Maybe not. But Carmack on efficiency? Just do it.
I am sure Carmack himself encourages debates and discussions. Lionizing one person can't be expected of every employee (unless that person is also the founder or the company is tiny).
This is what got Lucovsky pushed out. He wanted to build OS from scratch and couldn't see past the technical argument and acknowledge the Product's team urgency to actually land something in the hands of customers. Meanwhile, he left a trail of toxicity that he doesn't even realize was there[0].
Interestingly, he was pulling the same bs at Google until reason prevailed and he got pushed out (but allowed to save face and claim he resigned willingly[1]).
Carmack is a legend and I admire his work, but he seems to believe his own legend these days (like a few others big-ego gamedevs) and that can lead to arbitrary preferences being sold as gospel.
I'm sure that's true but I've worked with a lot of engineers that are of this caliber and as long as you can form a coherent logical explanation they will bend they're way more open than you expect. But you got to put in the work to make that argument. They won't take it on faith
It is an entirely different thing, gifted, highly experienced and confident in his assertions is quite frequent, but the cult-like following and status of personalities like Carmack or Blow can seriously alter their own self perception and importance.
Of course it can. It’s hard not to assume you are right if everyone keeps telling you so. I can’t really fault them for that. The challenge is to continue being right more often than not.
Meta's AR/VR division has burned a huge amount of money and years of time, with relatively little to show for it. Now it seems to be on the verge of being cancelled or slashed back, and in response people are saying that this proves VR, something Carmack champions, is commercially untenable or even that Carmack himself is partly responsible for the failed initiative. I don't even entirely agree with him on the question of whether anyone should try developing a new OS, but he's been proven absolutely right that there was no room for him to be that complacent about the use of Meta's resources.
There were almost no kids on the XROS team. The bulk of the team were E6s with graying hairs, multiple kids, and very impressive history of work on other well-known operating systems -- and most of them wrote a lot of code. This was the senior-est team I ever was a member of. Also, the most enjoyable interview process I've ever been through, no bullshit whatsoever and a rare case that I actually had to implement the exact thing that I was asked about during the interview (took me 3 weeks compared to 20 minutes during the loop, go figure).
XROS was an org that hired for specific specialist positions (as opposed to the usual "get hired into FB, go through the bootcamp, and find your place within the company"). At one point we got two separate requests from the recruiting execs:
- Your tech screen pass rate is way too low compared to other teams at FB. Please consider making your tech screen easier to expand the pool of candidates.
- Your interview-to-offer rate is way too low compared to other teams at FB. Please consider making your tech screen more difficult to reduce time that engineers spend on interviewing and writing feedback.
Anyway, IMO it was a very strong team in a very wrong environment. Most of the folks on the team hated the Facebook culture, despised the PSC process (despite having no problems with delivering impact in a greenfield project), had very little respect for non-technical managers coming from FB proper (the XROS team saw themselves as part of Oculus), and the majority I believe fled to other companies as soon as the project was scrapped. The pay was good however, and the work was very interesting. My overall impression was that most people on the team saw XROS as a journey, not a destination, and it was one of the reasons why it was destined to never ship.
That’s what I’m saying. It sounds like a dream job. Like you said it’s a journey not a destination, but it’s also a journey on one of the wealthiest companies in the worlds dime, so it’s kinda lame when someone calls it out for being suboptimal. That’s why I said who cares. It’s not going to hurt meta in the long run.
I believe the suboptimality concern was more about time to market and innovation velocity, and less about money. At the time FB felt a real sense of urgency given the anticipated AR/VR explosion (in a good sense) and presence of competitors in the space, both real and imaginary.
>Is he now more concerned with protecting shareholder value? Who cares.
It doesn't sound like he's concerned with waste. It sounds like it's a typical Carmack argument - distilled and hyper logical, and his conclusion is more to do with the pointlessness of it. He actually concedes the point that the project may have been highly efficient (which it may or may not have been, he was steelmanning).
His main points seemed to be:
If every cycle matters and efficiency is paramount, just make the project monolithic C++ code. If every cycle matters, that is somewhat incompatible with general purpose OSs, and if it doesn't, the existing landscape is more than good enough. Presumably, he's calling out the absurdity of counter-arguments which are being unrealistic about the objectives of creating a new general purpose OS, while also focusing on extreme efficiency. He states that the requirements to fully achieve these objectives would require a "monastic coding enclave" like Plan 9 OS, and it wasn't realistic even with the high talent in Meta.
And that plays into the second point, which seems to essentially be "new OSs aren't a draw for developers, they are a burden". This is painfully obvious when looking at the history of OSs and software, and it's the obvious reason why "let the kids build their operating system ffs" should result in a reflexive "noooo..." from the greybeards. The deeper point though is that if A. is achieved, the B. Burden on devs will be even more onerous. Therefore unless the entire project is committed to truly moving crowds to new paradigms (good luck, literally billions have been lost here), just use the proven, faily high performance options that have widespread support.
The conclusion is "on balance, it's a bad idea." He's arguing it sharply (although I understand a Carmack steelman is intimidating to attack), but in essence it's a fairly banal and conservative conclusion, backed with strong precedent.
meta was a weird place for a while. because of psc (the performance rating stuff) being so important… a public post could totally demoralize a team because if a legend like carmack thinks that your project is a waste of resources, how is that going to look on your performance review?
impact is facebook for “how useful is this to the company” and its an explicit axis of judgement.
How large is their headcount these days? And how many actually useful products have they launched in the last decade? You could probably go full Twitter and fire 90% of the people, and it would make no difference from a user perspective.
But... That's not an HR violation. If something a team is working on is a waste of resources, it's a waste. You can either realize that and pivot to something more useful (like an effort to take the improvements of the current OS project and apply them to existing OSes), or stubbornly insist on your value.
Why is complaining to HR even an option on the table?
One could argue that if it’s not in your swim lane, you just let it fail. And if you aren’t that person’s manager, you tell them the code or design that you are reviewing and thus the gatekeeper is not adequate. Politely. You said your part and no need to get yourself in trouble. Document and move on. If the company won’t listen then you move on. No need to turn it into a HR issue.
No matter how big or small one's "swim lane" is, an argument on technical merits without getting personal or discriminatory (assuming this was the case with J.C.) is never an HR issue. The whole "Weaponizing HR" thing is a nightmare and should not be acceptable.
I have mixed feelings about this. In one part, JC is someone I look up to, at least from the perspective of engineering. On the other hand, putting myself in the shoes in someone who got the once in life chance to build a new OS with corp support for a new shiny device…I for hell would want to do this.
Look at the outcome of Meta's performance in AR/VR over the past few years: a fortune has been spent; relatively little has been achieved; the whole thing is likely about to be slashed back; VR, something Carmack believes in, remains a bit commercially marginal and easily dismissed; and Carmack's own reputation has taken a hit from association with it all. You can understand perfectly well why he doesn't feel that it would have been harmless to just let other people have whatever fun they wanted with the AR/VR Zuckbucks.
(Mind you, Carmack himself was responsible for Oculus' Scheme-based VRScript exploratory-programming environment, another Meta-funded passion project that didn't end up going far. It surely didn't cost remotely as much as XROS though.)
Exactly! It seems very narc-y. Just let me build my cool waste of company resources, it's not like Zucky is going to notice, he's too busy building his 11 homes.
Imagine being able to build an operating system, basically the end-game of being a programmer, and get PAID for it. Then some nerd tells on you.
I'm not sure if you are trying to be /s, but yeah that's basically what I'm trying to say. Definitely better than working on those recommendation systems.
Damn, I'd pay to work in some serious OS/Compiler teams, but hey why should they hire me? Oh well...Yeah I'm doing a bit of projects on my side but man I'm so burnt out by my 9am-5pm $$ job + 5pm-10pm kid job that I barely have any large chunk of time to work on those.
Not sarcastic at all. I'm in the same boat. I've been trying to get into contributing to Redox, but at the end of the work day when the kid is finally asleep it's hard to motivate.
I get it man. It’s really tough. How old is your kid? Mine is a 5 years old boy and he doesn’t seem to need a lot of sleep but a huge amount of companionship which really bugs me out.
In theory, it’s better to sleep early, get up around 5 and get 2 hours of quality time, but man he sometimes gets up around 6:15 and earlier, and I found it difficult to get good sleep anyway, so I tried to switch to 2 hours of night time, but he wants to sleep with me for 30 mins around 9pm before going to my wife’s bed, and I usually fell sleep sooner than he did…
Girls are much easier to raise. They sleep earlier and don’t fight too much, as far as I heard from friends.
> If the platform really needs to watch every cycle that tightly, you aren't going to be a general purpose platform, and you might as well just make a monolithic C++ embedded application, rather than a whole new platform that is very likely to have a low shelf life as the hardware platform evolves.
Which I think is agreeable, up to a certain point, because I think it's potentially naive. That monolithic C++ embedded application is going to be fundamentally built out of a scheduler, IO and driver interfaces, and a shell. That's the only sane way to do something like this. And that's an operating system.
I got the chance to do this at Microsoft, it is indeed awesome! Thankfully the (multiple!) legendary programmers on the team were all behind the effort.
Anyway, if anyone reading this gets a chance to build a custom OS for bespoke HW, and get paid FAANG salary to do so, go for it! :-D
I saw the same thing at Google. A distinguished engineer tried gently at first to get a Jr engineer to stop trying to do something that was a bad idea. They persisted so he told them very bluntly to stop. HR got involved.
I even found myself letting really bad things go by because it was just going to take way to much of my time to spoon feed people and get them to stop.
If you're in high leadership, even just being pessimistic can be a massive morale killer. It doesn't mean that going to HR is the right call but I could see how someone would vent that way.
Hehehe. I have talked to John Carmack a few times. He's super harsh and has zero filter or social niceties (Azperger's level, not that he is, but just sayin'). If you are not used to it or understand where it's coming from, it can be quite a shock. Or at least he was, many years ago. Maybe he's changed.
I can see that. Sadly, there are a lot of people in the world who simply don't know how to deal with people who can be direct, if not somewhat abrasive, in their communication style. Their intent can be noble, well-intentioned, and not meant to offend. They simply don't beat around the bush or worry about whether your fragile ego will be bruised when they make an observation.
I've had to coach people and help them understand the entitlement involved in demanding that everyone adjust and adhere to their personal preferences and communication style. In my experience, it's about seeking to understand the person and adapt accordingly. Not everyone is willing to do that.
> Their intent can be noble, well-intentioned, and not meant to offend. They simply don't beat around the bush or worry about whether your fragile ego will be bruised when they make an observation.
I mean maybe, but maybe Carmac is just an ass hole... He can be a "legend" in the software development world and also just not be a super great person socially. The two things aren't mutually exclusive.
I don't disagree with you entirely, but being "direct" isn't a get-out-of-jail-free card for poor interpersonal skills. It's not always about "fragile egos" or "entitlement", it's about basic professionalism and communication.
Expecting that you can change people, in my experience, is quixotic. What you can change is yourself and how you interpret and respond to events around you.
If I understand the hypothetical you've proposed, my advice would be for you to adapt and learn to be less sensitive rather than have you believe that you can manipulate the environment, or worse, directly manipulate people.
It's possible that you could be a positive influence for change, so I don't want to completed discount any effort there, but I also think it's worth being realistic about what you can actually affect.
But my point is that this is a highly competitive environment, the whole point in a highly competitive environment is to manipulate the environment and people. That’s what people get promoted for. So why not play the game? Again in the hypothetical if you are “less sensitive rather than believe you can manipulate the environment.” You lose.
But like it works for Musk and Trump, and probably hundreds of other leaders today, why not take their example? (Assuming again, your highly ambitious and competitive, I’m more pro social, so I’d take your route)
This is legitimately something I’ve been asking myself lately, we talk about a world that values one thing ( rationality, respect, pro social behavior) but reward another (pettiness, vindictive, selfishness). Why do we pretend?
Also, and maybe the most important point, John Carmack is 100% trying to manipulate his environment and people, that’s why he’s so successful!
The world is literally run by people who are good at manipulating people and their environment. That’s what an entrepreneur is, that’s what a politician is, that’s what an artist is. Your argument seems to mostly be people shouldn’t try to manipulate the world in a way that I don’t like.
Masters of Doom does seems to want to, however accurately or not, set Carmack up as the antagonist of its story against Romero as the hero sometimes. I think that readers just largely didn't notice that since Carmack's heroic image was already so firmly established. In fact some of the early-ID stuff really does seem to raise some questions. (Was Tim Willits mostly Carmack's protégé, for instance?)
I’ve been on both the same side and the opposing side of debates with him, both in person and over internal discussion threads. His public persona and private behavior match. I viewed it positively, though per the topic of the thread, not everyone did.
That’s very different than simply observing someone in public. Which is what my post was referring too to and so it remains accurate.
FWIW I like carmack from what I have seen publically (and Romero, who I have interacted with) but I wouldn’t pretend to know who either of them really are from my observation of them.
> They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad
This is one of the reasons I’m sick of working pretty much anywhere anymore: I can’t be myself.
Appreciating people for their differences when they are humble and gifted is easy. I side with liberals, but I have a mix of liberal, moderate, and conservative friends.
But there are only so many years of pretending to appreciate all of the self-focused people that could be so much better at contributing to the world if they could quietly and selflessly work hard and respect people with different beliefs and backgrounds.
I’m happy for the opportunity I have to work, and I understand how millennials think and work. But working with boomers and/or gen X-ers would be so much less stressful. I could actually have real conversations with people.
I don’t think the problem is really with HR. I think the problem is a generation that was overly pandered to just doesn’t mix with the other generations, and maybe they shouldn’t.
Sorry but if you know his story, seen candid videos of him, or talked to the people around him, he's a Linus-level "I'll say what I want" type.
There weird hagiographies need to go. Carmack is absolutely not known to be kind. I have no idea what happened here but the idea that's he's this kindly old grandpa who could never, ever be rude or unprofessional is really out there.
And stupid. Like it or hate it, a non-nonsense, direct speaking, but fair and objective boss is the one you want. No one is served by failure; not the people at the top, nor the people at the bottom.
There is a difference between “this project is not going to work” vs “these people are incompetent and the project should be cancelled as a result”. The former needs to be said, the latter is a HR violation.
Carmack absolutely 100% percent did not say "these people are incompetent". What he said boils down to "these people are world's best experts on writing operating systems and they'd love to write a new one from the scratch but I strongly believe that writing a new operating system is not the best path forward."
> Carmack absolutely 100% percent did not say "these people are incompetent".
Sorry if I wasn’t clear; this was the point I was actually trying to make. Direct and to the point should not a HR incident make. I was trying to contrast with something that would.
I was at Google when the Flutter team started building Fuchsia.
They had amazing talent. Seriously, some of the most brilliant engineers I've worked with.
They had a huge team. Hundreds of people.
It was so ambitious.
But it seemed like such a terrible idea from the start. Nobody was ever able to articulate who would ever use it.
Technically, it was brilliant. But there was no business plan.
If they wanted to build a new kernel that could replace Linux on Android and/or Chrome OS, that would have been worth exploring - it would have had at least a chance at success.
But no, they wanted to build a new OS from scratch, including not just the kernel but the UI libraries and window manager too, all from scratch.
That's why the only platform they were able to target was Google's Home Hub - one of the few Google products that had a UI but wasn't a complete platform (no third-party apps, for example). And even there, I don't think they had a compelling story for why their OS was worth the added complexity.
It boggles my mind that Fuchsia is still going on. They should have killed it years ago. It's so depressing that they did across-the-board layoffs, including taking away resources from critically underfunded teams, while leaving projects like Fuchsia around wasting time and effort on a worthless endeavor. Instead they just kept reducing Fuchsia while still keeping it going. For what?
Wasn’t Fuchsia supposed to be a platform where different OS could run in a virtual environment and software packages would be complete containers? Was not this a new way of tackling the ancient OS problem?
These were my imaginations. I thought maybe an OS that could run on the web. Or an OS that could be virtualized to run on several machines. Or an OS that could be run along several other instances on the same machine each catering to a different user.
I guess it's just a political shit show at this point. Ideas go hard if the people behind them aren't playing the game well enough, no matter their value.
Not only did they target Home Hub, they basically forced a rewrite on it (us, I worked on the team). After we already launched. And made our existing workable software stack into legacy. And then they were late. Then late again. And late again. With no consequences.
100% agree with your points. To me watching I was like -- yeah, hell, yeah, working on an OS from scratch sounds awesome, those guys have an awesome job. Too bad they're making everyone else's job suck.
And the crazy thing is there is arguably a lot more of a reason for Meta / Oculus to have had its own operating system because it is meant for a specific configuration of hardware and to utilize those hardware resources to a quite different goal than most other OSes out there. Even in that environment it was still a waste
I've written a lot of low level software, BSPs, and most of an OS, and the main reason to not write your own OS these days is silicon vendors. Back in the day, they would provide you a spec detailed enough that you could feasibly write your own drivers.
These days, you get a medium-level description and a Linux driver of questionable quality. Part of this is just laziness, but mostly this is a function of complexity. Modern hardware is just so complicated it would take a long time to completely document, and even longer to write a driver for.
Intel still does it. As far as I can see they're the only player in town that provide open, detailed documentation for their high-speed NICs [0]. You can actually write a driver for their 100Gb cards from scratch using their datasheet. Most other vendors would either (1) ignore you, (2) make you sign an NDA or (3) refer you to their poorly documented Linux/BSD driver.
Not sure what the situation is for other hardware like NVMe SSDs.
The NVMe spec is freely downloadable and sufficient to write a driver with, if your OS already has PCIe support (which doesn't have open specifications). You don't need any vendor-specific features for ordinary everyday use, so it's a bit of a different situation from NICs. (Also, NVMe was in very large part an Intel creation, though it's maintained by an industry consortium.)
Yeah this. I tried to modify a hobby OS recently so it would process the "soft reboot" button (to speed up being rebooted in GCP) and it was so unbelievably hard to figure out how to support it. I tried following the instructions on the OS Dev Wiki and straight up reading what both Linux and FreeBSD do and still couldn't make progress. Yes. The thing that happens when you tell Windows or Linux to "restart". Gave up on this after spending days on it.
The people who develop OSes are cut from a different cloth and are not under the usual economic pressures.
I also think that they have access to more helpful resources than people outside the field do, e.g. being able to contact people working on the lower layers to get the missing info. These channels exist in the professional world, but they are hard to access.
> You are kinda wasting your time, the only way to make it work is to make the hobby OS Linux.
Not the parent, but of course they're wasting their time... That's the point of a hobby OS.
I'm working on a hobby OS, and I have no illusions that it's most likely fewer than 10 people will ever run it, and less than 100 will hear about it, but it lets me explore some interesting (to me) ideas, and forces me to learn a little more about random pieces of computing. If I ran on GCP, I'd want the reboot button to work. That sounds useful.
On the topic, I don't see why anyone would want to build a general purpose OS. There's enough already and even with the shrinking of hardware variety, there's a lot of stuff to support to make a general purpose OS work on enough hardware for people to consider using it. You can take Linux or a BSD and hack it up pretty good to explore a lot of OS ideas. Chances are you're going to borrow some of their drivers anyway, and then you'll end up with at least some similarity... may as well start there and save a lot of time. (My hobby OS has a custom kernel and custom drivers, but I only support a bare minimum of devices... (pc) console i/o, one real NIC, and virtio-net... that's all I need; I might add support for more NICs and more consoles later)
They said they wanted the soft reboot button to work. I assumed they meant catching the button press, which having seen some of this stuff is probably very tricky.
I don't see why a kexec alike wouldn't work about the same on GCP vs qemu vs bare metal... Or what that has to do with a GCP soft reboot button (which again, I think is referring to the reboot button in the GCP console)
Either way, the whole thing is a waste of time, yes? Why not waste time on the part that's engaging?
> The biggest scam in the OS world is drivers, we should demand more out of our hardware. Drivers shouldn't be necessary.
I can't even fathom what you mean here? You've got to have some interface to communicate with hardware. That's a driver. Some hardware only needs a very small driver... Tell the hardware where to send input, how to notify when input is ready and when its ready for output, and tell the hardware where data to output is. Maybe some setup stuff for modes and whatever if the needs aren't obvious and universal. I don't see how you could possibly avoid that.
It would certainly be possible for more devices to use common interfaces so a single driver could operate many different devices. Maybe that's what you mean? There's some movement towards that... SATA controllers generally speak AHCI, human interface devices generally appear as USB HID devices, etc. NICs tend to have a wide variety of setup sequences, but data queues usually fit into one of a limited number of patterns.
I agree you need a driver but for most hardware, that should be pretty simple, and easily documented by the hardware vendor, shouldn't it? A button has to be about the simplest possible I/O device imaginable.
Modern hardware is just so complicated it would take a long time to completely document, and even longer to write a driver for.
That's what's claimed. That's what people say, yet it's just an excuse. I've heard the same sort of excuse people have, after they write a massive codebase, then say "Oops, sorry, didn't get around to documenting it".
And no, hardware is not more difficult than software to document.
If the system is complex, there's more need to document, just as with a huge codebase. On their end, they have new employees to train up, and they have to manage testing. So any excuse that silicon vendors have to deal with such immense complexity? My violin plays for them.
> If the system is complex, there's more need to document
It’s not first party documentation that’s the problem. The problem is that they don’t share that documentation, so in order to get documentation for an “unsupported” OS a 3rd party needs to reverse engineer it.
> "Oops, sorry, didn't get around to documenting it".
That's obviously the wrong message. They should say "Go ask the engineering VP to get us off any other projects for another cycle while we're writing 'satisfying' documentation".
Extensive documentation comes at a price few companies are willing to pay (and that's not just a matter of resources. Look at Apple's documentation)
I write documentation as I'm writing the code. In my opinion, the code is only as good as its documentation -- they're two parts of the same thing. It's mostly comments at the top of files, and sometimes a markdown file in the same directory.
This way, good documentation is priced into my estimate for the project. I don't have a work item "spend a few days documenting." Nope, if I'm doing a foo then that includes documenting a foo at the same time.
In my experience, coding is much faster when doing it this way.
Yes, you can produce a small amount of code faster if you don’t “waste” your time on documentation, but that becomes counterproductive as soon as you can no longer keep the entire codebase in your head.
At least for certain types of OSes, it should be relatively easy to get most of Linux's hardware support by porting LKL (https://github.com/lkl/linux) and adding appropriate hooks to access hardware.
Of course, your custom kernel will still have to have some of its own code to support core platform/chipset devices, but LKL should pretty much cover just about all I/O devices (and you also get stuff like disk filesystems and a network stack along with the device drivers).
Also, it probably wouldn't work so well for typical monolithic kernels, but it should work decently on something that has user-mode driver support.
>but LKL should pretty much cover just about all I/O devices (and you also get stuff like disk filesystems and a network stack along with the device drivers).
thus calling into question why you ever bothered writing a new kernel in the first place if you were just going to piggyback Linux's device drivers onto some userspace wrapper thingy.
Im not necessarily indoctrinated to the point where I can't conceive of Linux being suboptimal in a way which is so fundamental that it requires no less than a completely new OS from scratch but you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.
> you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.
thats in interesting point, and makes me wonder if some kind of open interface for drivers to write to (and os's could implement) wouldn't be worthwhile?
probably it would have to be very general in design, but something along the lines of driverkit or iokit might work?
My hunch is that for nearly anyone who is serious about it these days, the way forward is either to have unusually tight control over the underlying platform, or to include a servant Linux installation with your OS. If Windows is a buggy set of device drivers, then Linux is a free set of buggy device drivers. If you're happy with your OS running as a client of a Linux hypervisor indefinitely then you could go for that; otherwise you'd have to try to gradually move bits of the hardware support into your OS over time—ideally faster than new Linux dependencies arise...
Writing drivers is easy, getting vendors to write *correct* drivers is difficult. At work right now we are working with a Chinese OEM with a custom Wifi board with a chipset with firmware and drivers supplied by the vendor. It's actually not a new wifi chipset, they've used it in other products for years without issues. In conditions that are difficult to reproduce sometimes the chipset gets "stuck" and basically stops responding or doing any wifi things. This appears to be a firmware problem because unloading and reloading the kernel module doesn't fix the issue. We've supplied loads of pcap dumps to the vendor, but they're kind of useless to the vendor because (a) pcap can only capture what the kernel sees, not what the wifi chipset sees, (b) it's infeasible for the wifi chipset to log all its internal state and whatnot, and (c) even if this was all possible trying to debug the driver just from looking at gigabytes of low level protocol dumps would be impossible.
Realistically for the OEM to debug the issue they're going to need a way to reliably repro which we don't have for them, so we're kind of stuck.
This type of problem generalizes to the development of drivers and firmware for many complex pieces of modern hardware.
I don't like Meta, but there used to be a time where big corp used to spend 30% of its budget on R&D. It's how we got all the toys we have now, R&D labs of big Bell and others.
So please don't mock the spend. Big spends fail sometimes, and at least people were paid to do the work.
The difference is that organisations like Bell Labs and Xerox PARC were primarily tech-first: innovations were the result of very clever and creative people doing blue skies research. The most groundbreaking stuff shocked the world while it was still a hacked-together demo, and similarly the cost of failure was quite low.
On the other hand, Meta's experiment is primarily CEO-driven. The outcome is predetermined, changing direction is not possible. Sure, clever engineers get to draw the rest of the owl, but that's not very useful when it turns out that everyone needs a horse instead.
They are spending a fortune, but rather than getting 900 crappy ideas to throw away and 100 great ones to pick from for continued development, they are developing 1 technological marvel nobody is interested in.
It was also pretty obvious how the VR glasses would support Meta's existing goals. It would give Meta total power over what you see and who you can speak with through their system. It's a natural extension of their total control over how people interact with on the Internet. And I think the only reason it failed is because it was expensive and dumb-looking.
Arguably the distinction you're pointing at is macroinvention (the transistor) vs microinvention (a better VR headset): one is a refinement of something that exists, another is transformative opening up whole new worlds of possibility. https://www.antonhowes.com/blog/macroinvention-vs-microinven...
It's just that it was so obviously going to fail, because there's no mass market for a product that you have to strap onto your face. You didn't need to spend billions to learn that.
If they'd spent the money researching nuclear fusion or space flight or a new way to develop microprocessors, I would be cheering their efforts even if they had failed in the end.
XROS had a completely new and rapidly evolving system call surface. No vendor would've been able to even start working on a driver for their device, let alone hand off a stable, complete result. It wasn't a case of "just rename a few symbols in a FreeBSD implementation and run a bunch of tests".
I don't know how you get paid for it, but if you want to write your own kernel, I'd start with an osdev tutorial. started with this one [1], but this one [2] has a promissing name... and I haven't really looked around.
It helps to have a concept to guide you too, but you can certainly make some progress on the basics before you figure out what you really want to do.
Thanks. I got all the resources covered. But I don’t have the energy to work on them as a side project any more. Alas! I wasted my younger days and hope you fare better!
Was at Oculus post acquisition and can say that the whole XROS was an annoyance and distraction the core technology teams didn’t need. There were so many issues with multiple tech stacks that needed fixing first.
Mind you, this XROS idea came after Oculus reorged into FB proper. It felt to me like there were FB teams (or individuals) that wanted get on the ARVR train. Carmack was absolutely right, and after the reorg his influence slowly waned for the worse.
Just a small bunch of XROS people came from FB proper (mostly managers) because an average FB SWE has no required skills. Most folks were hired from the industry at E5/E6 and I think we had ever took one or two bootcampers that ultimately were not successful and quickly moved elsewhere in FB.
Mechanisms for getting the linux kernel out of the way is pretty decent these days, and CPUs with a lot of cores are common. That means you can isolate a bunch of cores and pin threads the way you want, and then use some kernel-bypass to access hardware directly. Communicate between cores using ring buffers.
This gives you best of both worlds - carefully designed system for the hardware with near optimal performance, and still with the ability to take advantage of the full linux kernel for management, monitoring, debugging, etc.
It matters who you communicate concerns to. Something as fundamental as "I think that your team shouldn't even exist" should go to the team leads and their managers exclusively at first. Writing that to the entire affected team is counterproductive in any organization because it unnecessarily raises anxiety and reduces team productivity and focus. Comments like this from influential people can have big mental and physical health impacts on people.
This entire situation looks very suspicious. Was Carmack even responsible for triaging research projects and allocating resources for them? If yes, then he should have fought that battle earlier. If no, then the best he could do is to refuse to use that OS in projects he controls.
But the name Carmack carries some clout and people listen to him (too) closely because of his reputation alone. This is soft power that automatically comes with responsibility.
Maybe on a mediocre team. But that was the parent comment's point.
On well-functioning teams, product feedback shouldn't have to be filtered through layers of management. In fact, it would be dishonest to discuss something like this with managers while hiding it from the rest of the team.
This assumes that you would be sincerely listened to, which you wouldn't in a case like this. Higher ups in large organizations don't have the bandwidth to listen to everybody.
Your sub's officers also need to constantly be aware of what to communicate to whom and in which language. Your superiors certainly kept you in the dark about a ton of concerns that were on their plate because simply mentioning them to subordinates would have been too distracting.
You say your piece and if not heard, do an internal transfer. This whole don't tell people the truth about technical matters to not hurt their feelings or disrupt some people's paychecks is not serious business.
> Comments like this from influential people can have big mental and physical health impacts on people.
So what are we supposed to do? Just let waste continue? The entire point of engineering is to understand the tradeoffs of each decision and to be able to communicate them to others...
I'm sure that kind of crap helped nudge JC out of there. He mentions (accurate and relevant) reasons why something is probably a bad idea, and the person in charge of doing it complains that JC brought up the critiques, rather than addressing the critiques themselves. What a pathetic, whiny thing to do.
You've got to remember that context is critical with stuff like this.
There's nothing wrong with well-founded and thoughtful criticism. On the other hand, it is very easy for this to turn into personal attacks or bullying - even if it wasn't intended to be.
If you're not careful you'll end up with juniors copying the style and phrasing of less-carefully-worded messages of their tech demigod, and you end up with a huge hostile workplace behaviour cesspit.
It's the same reason why Linus Torvalds took a break to reflect on his communication style: no matter how strongly you feel about a topic, you can't let your emotions end up harming the community.
So yes, I can totally see poorly-worded critiques leading to HR complaints. Having to think twice about the impact of the things you write is an essential part of being at a high level in a company, you simply can't afford to be careless anymore.
It's of course impossible to conclude that this is what happened in this specific case without further details, but it definitely wouldn't be the first time something like this happened with a tech legend.
Back in mmm like 2002 or 2003 or 2004 while at Microsoft I read an internal paper from a few OS guys who hackathoned something for Bill Gates's Think Week (which is when he used to go to some island in San Juans or somewhere similar and just read curated papers and think, it was a huge prestige to get such a paper to him) and that something was an OS written from scratch with GC and memory management on top of something very .NET framework'y (which was released a couple of years ago. They had it booting on all kinds of hardware and doing various neato things. One of explicitly called design principles was 0 compatibility with anything Windows before. Which is why it didn't go anywhere I assume. I remember it was just a handful of engineers (presumably OS folks) hacking for like a month. . It was awesome to read about.
Singularity was cool. I'm sad that it was abandoned. The concept of using software isolation instead of hardware memory protection was really interesting.
I am very certain in my recollection that this was started much earlier than this as hackathon skunkworks before something like this happened at MSR. It didn't do anything beyond kernel and command line, there was no browser. I don't know if those two shared roots either. Anyhow, but yeah, still both were intellectual feats!
What would be the real advantage of a custom OS over a Linux distribution?
The OS does process scheduling, program management, etc. Ok, you don’t want a VR headset to run certain things slowly or crash. But some Linux distributions are battle-tested and stable, and fast, so can’t you write ordinary programs that are fast and reliable (e.g. the camera movement and passthrough use RTLinux and have a failsafe that has been formally verified or extensively tested) and that’s enough?
I think the proper comparison point here is probably what game consoles have done since the Xbox 360, which is basically run a hypervisor on the metal with the app/game and management planes in separate VMs. That gives the game a bare metal-ish experience and doesn't throw away resources on true multitasking where it isn't really needed. At the same time it still lets the console run a dashboard plus background tasks like downloading and so on.
Maybe not applicable for the XR platform here, but you could add introspection capabilities not present in Linux, a la Genera letting the developer hotpatch driver-level code, or get all processes running on a shared address space which lets processes pass pointers around instead of the Unix model of serializing/deserializing data for communication (http://metamodular.com/Common-Lisp/lispos.html)
Based on the latter tweet in the chain, I'm wondering if Carmack is hinting that Foveated Rendering (more processing power is diverted towards the specific part of the screen you're looking at) was one advantage envisioned for it. But perhaps he's saying that he's not so sure if the performance gains from it actually justify building a custom OS instead of just overclocking the GPU along with an existing OS?
Wouldn't that be an application (or at most system library) concern though? The OS is just there to sling pixels, it wouldn't have any idea whether those pixels are blurry… well for VR it would all be OpenGL or equivalent so the OS just did hardware access permissions.
I think the context is that foveated rendering ties sensor input (measuring gaze direction) to the rendering pipeline in a way that requires very low latency. Past a certain point reducing latency requires optimizations that break normal abstractions made by user land, so you end up with something more custom. I'm not sure why that would require a whole new OS, the obvious path would be to put the latency-sensitive code onto dedicated hardware and leave the rest managed by Linux. If a bunch of smart people thought XROS was a good idea there's probably something there though, even if it didn't pan out.
Everyone wants to make an OS because that's super cool and technical and hard. I mean, that's just resume gold.
Using Linux is boring and easy. Yawwwwn. But nobody makes an OS from scratch, only crazy greybeard developers do that!
The problem is, you're not crazy greybeard developers working out of your basement for the advancement of humanity. No. Youre paid employees of a mega corporation. You have no principles, no vision. You're not Linus Trovalds.
My understanding is that this is a key tenant of visionOS’s design, where apps don’t get access to gaze data (I think unless they’re taking over the full screen?)
I was there when they wanted to do the custom XROS. I remember asking them in a Q&A session exactly why they would build this and I recall the reasoning behind it totally fell flat. Fundamentally it became clear these guys just wanted to write a new OS because they thought it would be cool or fun.
Much of the scenarios they tried to address could have been done with Mach or some realtime kernel or with fuchsia. I recall later on they did consider using fuchsia as the user space for the os for some time.
On another note, there was similarly an equally “just for fun” language effort in the org as well (eg “FNL”). That was also conceived by a bunch of randos who weren’t a bunch of compiler guys that had no product vision and just did it for fun.
Well when the era of efficiency arrived all of this stuff ended.
I worked on a completely different hardware project within meta and while they didn't want a custom OS, they used an off the shelf rtos with the intention of modifying it and it was a shit show. They had a million justifications for why they needed it, but they had no performance tests or metrics to verify to actually justify it. They incurred a huge development overhead for no verifiable performance improvements.
None of the code they wrote couldn't have just been written as a kernel module in Linux. It would've also been so much easier due to all the documentation and general knowledge people have about Linux both within the company and outside the company.
I think people have forgotten about Google Fuchsia which I guess is a good sign for a new OS. They’ve done quite well in deploying it seamlessly to their consumer devices.
"Quite well" by what metric? It shipped on one device. That's pretty much the lowest bar you can imagine! Did it provide any tangible benefit to anyone? Let alone a benefit commensurate with the enormous cost of developing it and continuing to maintain it?
I think it was insane to start a new OS effort written in C/C++. We have plenty of OSes written in C/C++! We know how that story ends. If you're going to spend the effort, at least try a new language that could enable a better security model.
Fun rumor: Google shut down the AR effort and transferred the team to project Fuchsia as a way to retain highly skilled employees. So essentially they didn’t have any real technical needs for a new OS.
Isn't that somewhat debatable? Originally they were aiming at much more (chromebook OS for example) but seems like they settled for Google Home only as their scope.
Still a very interesting project, but that feels like a similar story, for limited use cases (a smart thermostat/speaker with specific hardware) it works, but for wider use cases with heterogeneus hardware and complex interfaces (actual screen, peripherals) it didn't work.
The only reason Chinese companies can even get away with these big projects is because of state backing and state objectives. By itself, the market doesn't support a new general-purpose OS at this point.
Technically you are correct but the commenter you’re responding to means that with the amount of Western Governments spend on MS products and services, the are a d facto (if not de jure) state backed enterprise.
US government spending is (for now) easy to track, and you can get totals for spending by corporate entity.
In total across the entire US federal government, $518.8 million was paid to Microsoft for products and services in 2024. That is approximately 0.21% of their total annual revenue.
I assert that the threshold for "state sponsored" is well in excess of 0.21% of annual revenue.
MS has deep ties into the state department and intelligence apparatus that few other companies do. Just as deep as the defense contractors who have a near monopoly-monopsony relationship with the federal government. You can argue about how exclusive they are in particular qualities but the scale and depth they operate at makes their relationship approximate the relationship Huawei does with the Chinese government. They're just what state-backed enterprises look like under liberal-ish capitalism.
You could write a book on why it's practically impossible to create a new OS these days. Love Carmack for stating it so clearly. I also love that called out TempleOS, I also have a weird respect for it. Plan 9 is the probably the best example of a totally new OS and I hope someday it becomes viable because it's really a joy to use.
But ultimately it just makes sense to adapt existing kernels / OS (say, arch) and adapt it to your needs. It can be hair wrenchingly frustrating, and requires the company to be willing to upstream changes and it still takes years, but the alternative is decades, because what sounds good and well designed on paper just melts when it hits the real world, and linux has already gone through those decades of pain.
Android built a new, giant moat for Linux (or "Linux" depending on your opinions about Android) in the embedded application processor space - now the "standard" board support package target for new embedded AP hardware is almost always some random point-in-time snapshot of Android. Running "mainline" Linux is hard (because the GPU and media peripheral drivers are usually half-userspace Android flavored stuff and rely on ION and other Androidisms) and bare-metal is even worse (where previously, you'd get register-level documentation, now, you get some Android libXYZ.so library).
Much of that 5M are hardware register definitions expanded into C headers. I am not sure how you'd consolidate that but it's not like that's all bespoke C code.
I sympathize, but the reality is that except for very specialized cases, (hyper) optimizing for CPU performance is unnecessary, even in the embedded space. A Cortex-M0 has roughly the same performance as a 486, and is cheap and power efficient enough to be bundled in disposable test kits, vapes, etc.
Is the difficulty in theoretical complexity of operating systems, or in project scoping/scope creep?
It's probably not that hard to write bare metal code for a modern CPU that runs and crashes. It's obviously insurmountably hard to compete with Android in features with scratch built bare metal code. An "OS" can be anything between the two. But it's very easy to imagine an "XR OS" project snowballing quickly into the latter, and Carmack's concerns would be spot on(as always is, and as proven). Is it then an inherent difficulty in "designing a new operating system", or is it technically something else?
> To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers.
I mean, I'd give a fair shake to an OS from the SQLite team [1].
This is completely right from a product point of view, which is Carmack's argument.
But I have wondered why one of these companies with billions of dollars to burn hasn't tried to create something new as a strategic initiative. Yes, there wouldn't be any ROI for years, and yes, the first several products on the platform would probably be better off on something more traditional.
But the long term value could potentially be astronomical.
Just another case of quarterly-report-driven decision making, I suppose. Sigh.
Historically? The internet, the concept of a graphical user interface, the mouse, the smartphone, the LCD display, the laser printer...
It's about clever people trying weird stuff, and occasionally ending up with a world-changing idea. Asking for examples of to-be-discovered innovations is, by definition, an impossibility.
This is what Google has been trying to do with Fuchsia and the fact is that you can't escape the product point of view because the products exist, already have an OS stack, and get pretty defensive when another team tells them they're going to replace their OS, or their core if the product team is Android or Chrome OS.
If you're competing against nothing, then I see it: it opens up a wide variety of product possibilities. But linux exists. Why not spend 1/1000th the time to adapt linux?
That's not even counting the rather substantial risk that your new OS will never approach the capabilities of linux, and may very well never become generally usable at all.
Option A: spend years and millions on a project that may never be as good as existing solutions, diverting attention and resources from actual products, or...
Option B: work on products now, using an existing, high-quality, extensible, adaptable OS, with very friendly licensing terms, for which numerous experts exist, with a proven track record of maintenance, a working plan for sustainability, a large & healthy developer community exists, etc.
It's hard to imagine how it wouldn't be a complete waste of time.
I wonder how well that would play out. Surely you'd only do if you got a perpetual license or bought it outright. I can imagine RIM would fuck you over eventually with with any other arrangement.
Apple bought one of those in the 90s, and they are still reaping the benefits of that strategic initiative. But the thing is, NeXt allowed Apple to think up new, differentiated products. If you come at the problem of the OS from a purely technical perspective, you'll waste time for no gain.
While I appreciate Carmack and all, I'd love to hear from someone like Dave Cutler who's been over that bridge at least a couple of times successfully about if and what he'd do if he had resources to create whatever the hell he wants.
But I have wondered why one of these companies with billions of dollars to burn hasn't tried to create something new as a strategic initiative.
They have; Taligent comes to mind. You may not have heard of that -- or more likely, you have but just forgot about it -- but it's a good object lesson (no pun intended) in why a successful new OS is hard to just conjure into existence. There has to be a crying, desperate need for it, not just a vague sense that This Time We'll Get It Right.
You could probably cite OS/2 Warp as a less-obscure example of the same phenomenon.
In that case, why wouldn't they "just" fork Linux? Or 10-years-ago-Linux?
The technical justification for Meta writing their own OS is that they'd get to make design decisions that suited them at a very deep level, not that they could do the work equivalent of contributing a few drivers to an existing choice.
I'd love a truly new OS, but I just don’t know what it would look like at this point? "New OS" ideas tend to converge on the same trunk.
Building a hobby OS taught me how little is just "software". The CPU sets the rules. Page tables exist because the MMU says so. Syscalls are privilege flips. Task switches are register loads and TLB churn. Drivers are interrupt choreography. The OS to me is just policy wrapped around fixed machinery.
> I wish I could drop (so many of) my old internal posts publicly, since I don’t really have the incentive to relitigate the arguments today – they were carefully considered and prescient. They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad, but I expect many of them would acknowledge in hindsight that the Meta products would not be in a better place today if the new OS effort had been rammed into them.
So someone at Meta was so sensitive that being told their behemoth of a project was ill advised ended up getting reported to HR?
> To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers. Which was sort of Plan 9…
Jonathan Blow is the world’s most successful hobbyist programmer. His whole thing is doing projects from scratch. Every game he made could be done in Unity with far less effort.
Most opinions of this man exists in a vacuum space isolated from the real world software industry. Building an OS from scratch is one of those examples.
It’s never seems like there’s a significant reason behind them other than………”I made dat :P”
I don't think unity was as polished when braid came out in ~2008 that can also easily rewind time on low end Xbox hardware. The witness maybe in unreal? But there are some wild things there I've never seen an unreal game do that the witness does do
As an outsider...his games just look and feel different. They feel like bones-deep art, in a way that even the best of the best games (say, Hades) don't. Since Blow's games are puzzle games they're not even my favorite games! But the effort spent on making them exactly the way he wants them pays off.
I love this part: "To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers"
I’ve seen this firsthand. These giant tech companies try to just jump into a massive new project thinking that because they have built such an impressive website and have so much experience at scale they should just be able to handle building a new OS.
In my case it wasn’t even a new OS it was just building around an existing platform and even that was massively problematic.
The companies that build them from scratch have had it as one of their core competencies pretty much from the start.
They have contributors to the linux kernel.
Pretty sure all the big tech companies have the right people to create a new OS that is better than Linux, the hard part is getting that new OS to be adopted.
And yet, Sony did it, Nintendo did it, both have been pretty succeesful.
We also need to be clear what an OS is. Is it "darwin" or "macOS" - they have different scopes.
Things I'd want from an OS for an XR device.
1. Fast boot. I don't want to have to wait 2-3-4-5 minutes to reboot for those times I need to reboot.
I feel like Nintendo figured this out? It updates the OS in the background somehow and reboot is nearly instant.
2. Zero jank. I'm on XR, if the OS janks in any way people will get sick AND perceive the product as sucking. At least I do. iOS is smooth, Androind is jank AF.
Do any of the existing OSes provide this? Sure, maybe take an existing OS an modify it, assuming you can.
Nintendo is an interesting example though. According to Wikipedia they actually use a proprietary microkernel, which, if I'm reading this right, I think they developed themselves. Looks like the only open source components they have is some networking code which is published under the BSD license.
Sony and Nintendo both forked off of either NetBSD or FreeBSD. Sony's cameras at least up until the A7S2 run Linux (there's jailbreaks for these), although I never found any kernel / bootloader dump like it would be required.
Android suffers from being Java at the core, with all the baggage that brings with it.
Sony cameras all run Linux. Models with "PlayMemories Camera Apps" feature like A7M2 support runs Android userland on top. It's probably easier to count the cameras that don't(like old Olympus).
The problem with this guy is that it’s hard to criticize him, whether at work or in this forum. For example, I am going to be downvoted for mocking the fact that this guy thinks it’s some genius move to say “No” to making an operating system, whatever making an operating system means.
I like that the top reply to Carmack's wall of text is a screenshot of TempleOS with a doodle of an elephant lmao. And ironically, that meme reply is on topic and it says a thousand words with just one photo.
Another point I would add in support of that meme comment, is Google's recent rug-pull of Android not allowing sideloading apps from unsigned developers anymore starting this autumn, after over a decade of conquering the market with their "go with us, we're the open alternative to iOS" marketing.
The conclusion is to just never EVER trust big-tech/VC/PE companies, even when they do nice things, since they're 100% just playing the long game, getting buddy-buddy with you waiting till they smothered the competition with their warchest, and then the inevitable rug-pull comes once you're tied to their ecosystem and you have nowhere else to go.
Avoid these scumbags, go FOSS form the start, go TempleOS. /s but not really
In other words, unless God has specifically called upon you to build an OS, and maybe provided divine inspiration and assistance, you should avoid doing that. Seems to support Carmack's point!
Unless its for love or devotion, there's no compelling reason to create a new OS in 2025. Certainly that could change in the future but I think his observation (if I understand him correctly) is correct.
I'm not sure Carmack's point disagrees with you. Meta is still big tech, and if your goal is to monetize at scale, rolling out your own isn't the most efficient way to do it. I don't think he'd discourage you rolling out your own OS if it's your hobby FOSS project.
He's acting like their VR UX is top notch when it's as bad as it gets. Just yesterday I dusted off my Meta Quest 2 to play a bit, and spent around an hour trying to pair up my left controller to the helmet after replacing the battery.
You can't do it without going through their fucking app, that asks for every permissions under the sun, including GPS positioning for some reason. After finally getting this app working and pairing it with my headset, I could finally realize the controller was just dead and their was nothing to do.
You can pair the controllers in the settings you don't need an app. Their VR UX does suck that is true, and horizon worlds is such a collosal failure that I'm surprised they haven't cancelled that entirely yet. But carmack also stated the technical issues numerous times.
> You can't do it without going through their fucking app, that asks for every permissions under the sun, including GPS positioning for some reason.
If it uses bluetooth, which it might for the controller?, the permission for bluetooth on Android is fine location --- the same permission as for using GPS. That might be the same permission you need for wifi stuff, too? Because products and services exist to turn bluetooth and wifi mac addresses seen into a fine location.
But who knows what they do with the GPS signal after they ask for it?
Didn't even realize this was a thread and not a single tweet until you posted this link. Guess that's the downsides of not having a Twitter acct anymore.
If a professional can't give critical feedback in a professional setting without being rude or belittling others, then they need to improve their communication skills.
This is not that though. This is just developers being unable to handle constructive criticism, and when they can't win the argument on merits, went for the HR option. It happens.
I've had it happen to me too, but my response was to resign on the spot (I was already not satisfied with the company).
The "toxic behaviour" I had done? I reverted a commit on the master branch that didn't compile, and sent a slack to the Dev who had committed it saying "hi! There appears to have been a mistake in your latest commit, could you please check it out and fix it? I've reverted it in the meantime since I need to deploy this other feature"
The dev responded by force pushing the code that did not compile to master and contacted HR.
I decided there was greener grass on other pastures. I was right.
When I started breaking the build would end up with the person who did it having to wear the dunce hat for the day. This was before git so there are now fewer excuses for breaking the build today.
From what you know of Carmack, does "can't give critical feedback in a professional setting without being rude or belittling others" sound like him to you? It does not to me, though granted maybe he's different in his non public persona than what you can see in presentations and talks.
Being "reported to HR" doesn't mean "almost got fired". It likely meant a meeting where someone explained "hey, the way you communicated that caused some upset, let's discuss better ways to handle that situation next time." Very often in larger companies, complaints about things like "this bigwig from this other group jumped all over us" are automatically sent through HR because HR has staff whose job just is resolving conflicts between people and keeping things peaceful.
FWIW, it can also mean a meeting where HR says to the complainer: "Are you good, dude? Why are you complaining about this? This is perfectly normal workplace behavior. Stop wasting our time."
Having worked in the valley, I've seen what critical feedback meant in many companies there, and it removes all usefulness of the info because there is a ceiling of what is socially acceptable to say; therefore, you can't know how bad or urgent things are.
Everything is ASAP. They are super excited about everything. And nothing you do is wrong, it just could be improved or they like it but don't love it.
You don't know if something is important, basically.
Just like Louis CK said, "if you used 'amazing' on chicken nuggets, what are you going to say when your first child is born?". But in reverse.
Personally, I'd rather work with someone who would tell me my work is terrible if it is.
In Germany, you can't even legally say somebody did a bad job at your company in a recommendation letter. Companies created a whole subtext to workaround that, it's crazy.
Some things are just bad. You should be able to say it is. Not by saying it could be better. Not by using euphemism. It's just something that needs to go to the trash.
In fact, I don't trust people who can't receive this information, even if not packaged with tact (which you should attempt to, but life happens). If you can't handle people not being perfectly polite every time, I can't help but feel I won't be able to count on you when things get hard.
> In Germany, you can't even legally say somebody did a bad job at your company in a recommendation letter. Companies created a whole subtext to workaround that, it's crazy.
I don't think it's just about legality. Whether the recommendation letter is included in the application is at the distinction of the applicant. When you want it to reach the next company, you must write is so, that the former employee considers it to be a good recommendation.
I am with you on this. Worked in our Dublin office was so refreshing to have straight up communication vs whatever the song and dance we do here in US.
You've concluded this from a single, brief, throwaway line? Any madness you perceive about this situation has been fabricated by you, based on the details we have.
People have been getting mad at being made to feel bad at work for much longer than “safe space culture” has existed. If someone or some team had more power than you at an organization you for sure will get reprimanded for making them feel bad.
Something tells me that if we heard the other side of the story it might hit different. There's a lot of wiggle room in what "making his team members feel bad" could mean, and I would be surprised if constructively voiced criticism would have gotten someone written up.
With my experience of being written up for constructive criticism the reasoning was that I didn’t give constructive criticism to others and they felt singled out. I only give such criticism in private so of course they were not there to see the others. Apparently that wasn’t a sufficient explanation.
it is madness, you would be surprised how many ppl take things too serious. been there, had talk with HR cause i've said that the solution is mediocre and we have to do something better than that.
Reading between the lines, it sounds like he got reported for giving a lot of what might kindly be described as unsolicited advice. The guy left Meta ages ago, but he apparently still can't let this one go.
If you're in the middle of trying to write a new operating system, then it's probably not helpful to have John Carmack standing over you repeatedly telling you that you shouldn't be doing it. In this case Carmack gets the last laugh. Then again, it is easy to get the last laugh by predicting that a project will fail, given that most projects do.
> If you're in the middle of trying to write a new operating system, then it's probably not helpful to have John Carmack standing over you repeatedly telling you that you shouldn't be doing it. In this case Carmack gets the last laugh. Then again, it is easy to get the last laugh by predicting that a project will fail, given that most projects do.
I mean, if you're working on a project that is likely to fail, wouldn't it be nice if someone gave you cover to stop working on it, and then you could figure out something else to do that might not fail? Can't get any impact if your OS will never ship.
tbh linux has quite a bit of cruft in it these days at the syscall and interface layer.
if youre apple, it does make sense to do stuff from scratch. i think in a way, software guys wind up building their own prisons. an api is created to solve problem X given world Y, but world Y+1 has a different set of problems - problems that may no longer be adequately addressed given the api invented for X.
people talk about "rewrite everything in rust" - I say, why stop there? lets go down to the metal. make every byte, every instruction, every syscall a commodity. imagine if we could go all the way back to bare metal programming, simply by virtue of the LLM auto-coding the bootloader, scheduler, process manager, all in-situ.
the software world is full of circularities like that. we went from Mainframe -> local -> mainframe, why not baremetal -> hosted -> baremetal?
Apple doesn't do a lot of baremetal development from scratch that I'm aware of. The Lightning to HDMI dongle bootstraps an XNU kernel with an AirPlay decoder into 256MB RAM, for instance.
You can still do “unsafe” stuff in rust, and people do. It’s perfectly possible to write safe C and C++ these days. And you don’t have to deal with a borrow checker, and a very small pool of developers available to hire.
oh, i didnt mean to invoke rust in any technical sense - i brought up rust to introduce an example of the attitude that rust people are known for, namely "why not rewrite everything?", which a lot of people have a kneejerk rejection of.
> They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad
I've only seen John Carmack's public interactions, but they've all been professional and kind.
It's depressing to imagine HR getting involved because someone's feelings had been hurt by an objective discussion from a person like John Carmack.
I'm having flashbacks to the times in my career when coworkers tried to weaponize HR to push their agenda. Every effort was eventually dismissed by HR, but there is a chilling effect on everyone when you realize that someone at the company is trying to put your job at stake because they didn't like something you said. The next time around, the people targeted are much more hesitant to speak up.
I followed his posts internally before he left. He was strict about resource waste. Hand tracking would break constantly and he brought metrics to his posts. His whole point was that Apple has hardware nailed down and it’ll be efficient software that will be the differentiator. The bloat at Meta was the result of empire building.
I remember watching Carmack at a convention 15 years ago. He took a short sabbatical and came back with ID Tech 3 on an iPhone, and it still looks amazing well over a decade later.
https://www.youtube.com/watch?v=52hMWMWKAMk&t=1s
This is a guy who figures that what he wants to do most with his 3 free weekends is to port his latest, greatest engine to a Cortex-A8. Leading corporate strategy? Maybe not. But Carmack on efficiency? Just do it.
Impressive. JC is always one of the engineers I look up to and read up to when depressed.
John Carmack, David Cutler, Tom West, Cameron Zwarich, etc. There are about maybe 50 of them.
Carmack and Jim Keller for me. Hardware engineering for the latter!
HW is kinda too magical for me at this age :)
Tim Sweeney of Epic is up there for me too.
I followed his posts internally too. It's amazing how many people were arguing against fucking John Carmack. What a waste of talent.
> were arguing against fucking John Carmack
I am sure Carmack himself encourages debates and discussions. Lionizing one person can't be expected of every employee (unless that person is also the founder or the company is tiny).
Can’t really imagine a better person to argue against?
This is what got Lucovsky pushed out. He wanted to build OS from scratch and couldn't see past the technical argument and acknowledge the Product's team urgency to actually land something in the hands of customers. Meanwhile, he left a trail of toxicity that he doesn't even realize was there[0].
Interestingly, he was pulling the same bs at Google until reason prevailed and he got pushed out (but allowed to save face and claim he resigned willingly[1]).
[0] https://x.com/yewnyx/status/1793684535307284948 [1] https://x.com/marklucovsky/status/1678465552988381185
John can be quite blunt and harsh in person, from everyone I know who’s interacted with him.
If he doesn’t believe in something, he can sometimes be over critical and it’s hard to push back in that kind of power imbalance.
Carmack is a legend and I admire his work, but he seems to believe his own legend these days (like a few others big-ego gamedevs) and that can lead to arbitrary preferences being sold as gospel.
I'm sure that's true but I've worked with a lot of engineers that are of this caliber and as long as you can form a coherent logical explanation they will bend they're way more open than you expect. But you got to put in the work to make that argument. They won't take it on faith
It is an entirely different thing, gifted, highly experienced and confident in his assertions is quite frequent, but the cult-like following and status of personalities like Carmack or Blow can seriously alter their own self perception and importance.
Of course it can. It’s hard not to assume you are right if everyone keeps telling you so. I can’t really fault them for that. The challenge is to continue being right more often than not.
fucking christ there is truly no end to this contrived "power imbalance" narrative is there
Are you arguing that everyone's power is equal? What an asinine position.
It must either be a belief that everyone is equal, and that they would be comfortable telling Carmack that he’s wrong.
Or it’s an appeal to authority in that if a person of authority says it then they must be right so there’s no reason to push back in the first place.
Which makes sense when you are one of 3 developers at ID software. There's absolutely no room for waste.
This is Meta. Let the kids build their operating system ffs. Is he now more concerned with protecting shareholder value? Who cares.
Meta's AR/VR division has burned a huge amount of money and years of time, with relatively little to show for it. Now it seems to be on the verge of being cancelled or slashed back, and in response people are saying that this proves VR, something Carmack champions, is commercially untenable or even that Carmack himself is partly responsible for the failed initiative. I don't even entirely agree with him on the question of whether anyone should try developing a new OS, but he's been proven absolutely right that there was no room for him to be that complacent about the use of Meta's resources.
No, no. If you want your VR apps running in two years on something that looks like an OS, just build an app that runs on an existing one.
If you want to be the dominant player in the market in 10-15 years, build the OS and keep funding it.
There were almost no kids on the XROS team. The bulk of the team were E6s with graying hairs, multiple kids, and very impressive history of work on other well-known operating systems -- and most of them wrote a lot of code. This was the senior-est team I ever was a member of. Also, the most enjoyable interview process I've ever been through, no bullshit whatsoever and a rare case that I actually had to implement the exact thing that I was asked about during the interview (took me 3 weeks compared to 20 minutes during the loop, go figure).
XROS was an org that hired for specific specialist positions (as opposed to the usual "get hired into FB, go through the bootcamp, and find your place within the company"). At one point we got two separate requests from the recruiting execs: - Your tech screen pass rate is way too low compared to other teams at FB. Please consider making your tech screen easier to expand the pool of candidates. - Your interview-to-offer rate is way too low compared to other teams at FB. Please consider making your tech screen more difficult to reduce time that engineers spend on interviewing and writing feedback.
Anyway, IMO it was a very strong team in a very wrong environment. Most of the folks on the team hated the Facebook culture, despised the PSC process (despite having no problems with delivering impact in a greenfield project), had very little respect for non-technical managers coming from FB proper (the XROS team saw themselves as part of Oculus), and the majority I believe fled to other companies as soon as the project was scrapped. The pay was good however, and the work was very interesting. My overall impression was that most people on the team saw XROS as a journey, not a destination, and it was one of the reasons why it was destined to never ship.
That’s what I’m saying. It sounds like a dream job. Like you said it’s a journey not a destination, but it’s also a journey on one of the wealthiest companies in the worlds dime, so it’s kinda lame when someone calls it out for being suboptimal. That’s why I said who cares. It’s not going to hurt meta in the long run.
I believe the suboptimality concern was more about time to market and innovation velocity, and less about money. At the time FB felt a real sense of urgency given the anticipated AR/VR explosion (in a good sense) and presence of competitors in the space, both real and imaginary.
>Is he now more concerned with protecting shareholder value? Who cares.
It doesn't sound like he's concerned with waste. It sounds like it's a typical Carmack argument - distilled and hyper logical, and his conclusion is more to do with the pointlessness of it. He actually concedes the point that the project may have been highly efficient (which it may or may not have been, he was steelmanning).
His main points seemed to be:
If every cycle matters and efficiency is paramount, just make the project monolithic C++ code. If every cycle matters, that is somewhat incompatible with general purpose OSs, and if it doesn't, the existing landscape is more than good enough. Presumably, he's calling out the absurdity of counter-arguments which are being unrealistic about the objectives of creating a new general purpose OS, while also focusing on extreme efficiency. He states that the requirements to fully achieve these objectives would require a "monastic coding enclave" like Plan 9 OS, and it wasn't realistic even with the high talent in Meta.
And that plays into the second point, which seems to essentially be "new OSs aren't a draw for developers, they are a burden". This is painfully obvious when looking at the history of OSs and software, and it's the obvious reason why "let the kids build their operating system ffs" should result in a reflexive "noooo..." from the greybeards. The deeper point though is that if A. is achieved, the B. Burden on devs will be even more onerous. Therefore unless the entire project is committed to truly moving crowds to new paradigms (good luck, literally billions have been lost here), just use the proven, faily high performance options that have widespread support.
The conclusion is "on balance, it's a bad idea." He's arguing it sharply (although I understand a Carmack steelman is intimidating to attack), but in essence it's a fairly banal and conservative conclusion, backed with strong precedent.
meta was a weird place for a while. because of psc (the performance rating stuff) being so important… a public post could totally demoralize a team because if a legend like carmack thinks that your project is a waste of resources, how is that going to look on your performance review?
impact is facebook for “how useful is this to the company” and its an explicit axis of judgement.
How large is their headcount these days? And how many actually useful products have they launched in the last decade? You could probably go full Twitter and fire 90% of the people, and it would make no difference from a user perspective.
But... That's not an HR violation. If something a team is working on is a waste of resources, it's a waste. You can either realize that and pivot to something more useful (like an effort to take the improvements of the current OS project and apply them to existing OSes), or stubbornly insist on your value.
Why is complaining to HR even an option on the table?
One could argue that if it’s not in your swim lane, you just let it fail. And if you aren’t that person’s manager, you tell them the code or design that you are reviewing and thus the gatekeeper is not adequate. Politely. You said your part and no need to get yourself in trouble. Document and move on. If the company won’t listen then you move on. No need to turn it into a HR issue.
Carmack's swim lane was exceptionally wide. My understanding was that this sort of criticism was actually his main job duty.
No matter how big or small one's "swim lane" is, an argument on technical merits without getting personal or discriminatory (assuming this was the case with J.C.) is never an HR issue. The whole "Weaponizing HR" thing is a nightmare and should not be acceptable.
Imagine being a meta engineer and not taking Carmack's advice seriously.
Why the fuck is he even hired there if you are not going to listen to him.
Dude has forgotten more things about game development than you will ever know...
There were quite a few of high-caliber individuals with equally impressive resumes in the organization to match Carmack's wisdom and ego.
Fully agree with this point we all know as engineers this shit is nails on the chalkboard.
Complaining is always an option. The problem is that HR actually takes the complaint seriously.
I have mixed feelings about this. In one part, JC is someone I look up to, at least from the perspective of engineering. On the other hand, putting myself in the shoes in someone who got the once in life chance to build a new OS with corp support for a new shiny device…I for hell would want to do this.
Look at the outcome of Meta's performance in AR/VR over the past few years: a fortune has been spent; relatively little has been achieved; the whole thing is likely about to be slashed back; VR, something Carmack believes in, remains a bit commercially marginal and easily dismissed; and Carmack's own reputation has taken a hit from association with it all. You can understand perfectly well why he doesn't feel that it would have been harmless to just let other people have whatever fun they wanted with the AR/VR Zuckbucks.
(Mind you, Carmack himself was responsible for Oculus' Scheme-based VRScript exploratory-programming environment, another Meta-funded passion project that didn't end up going far. It surely didn't cost remotely as much as XROS though.)
Exactly! It seems very narc-y. Just let me build my cool waste of company resources, it's not like Zucky is going to notice, he's too busy building his 11 homes.
Imagine being able to build an operating system, basically the end-game of being a programmer, and get PAID for it. Then some nerd tells on you.
I'm not sure if you are trying to be /s, but yeah that's basically what I'm trying to say. Definitely better than working on those recommendation systems.
Damn, I'd pay to work in some serious OS/Compiler teams, but hey why should they hire me? Oh well...Yeah I'm doing a bit of projects on my side but man I'm so burnt out by my 9am-5pm $$ job + 5pm-10pm kid job that I barely have any large chunk of time to work on those.
Not sarcastic at all. I'm in the same boat. I've been trying to get into contributing to Redox, but at the end of the work day when the kid is finally asleep it's hard to motivate.
I get it man. It’s really tough. How old is your kid? Mine is a 5 years old boy and he doesn’t seem to need a lot of sleep but a huge amount of companionship which really bugs me out.
In theory, it’s better to sleep early, get up around 5 and get 2 hours of quality time, but man he sometimes gets up around 6:15 and earlier, and I found it difficult to get good sleep anyway, so I tried to switch to 2 hours of night time, but he wants to sleep with me for 30 mins around 9pm before going to my wife’s bed, and I usually fell sleep sooner than he did…
Girls are much easier to raise. They sleep earlier and don’t fight too much, as far as I heard from friends.
I actually have some ideas about this kind of situation, drop me a line, email in my profile!
Reading on from that he says:
> If the platform really needs to watch every cycle that tightly, you aren't going to be a general purpose platform, and you might as well just make a monolithic C++ embedded application, rather than a whole new platform that is very likely to have a low shelf life as the hardware platform evolves.
Which I think is agreeable, up to a certain point, because I think it's potentially naive. That monolithic C++ embedded application is going to be fundamentally built out of a scheduler, IO and driver interfaces, and a shell. That's the only sane way to do something like this. And that's an operating system.
I got the chance to do this at Microsoft, it is indeed awesome! Thankfully the (multiple!) legendary programmers on the team were all behind the effort.
Anyway, if anyone reading this gets a chance to build a custom OS for bespoke HW, and get paid FAANG salary to do so, go for it! :-D
Yup. This is how bloat is created.
I saw the same thing at Google. A distinguished engineer tried gently at first to get a Jr engineer to stop trying to do something that was a bad idea. They persisted so he told them very bluntly to stop. HR got involved.
I even found myself letting really bad things go by because it was just going to take way to much of my time to spoon feed people and get them to stop.
If you're in high leadership, even just being pessimistic can be a massive morale killer. It doesn't mean that going to HR is the right call but I could see how someone would vent that way.
Hehehe. I have talked to John Carmack a few times. He's super harsh and has zero filter or social niceties (Azperger's level, not that he is, but just sayin'). If you are not used to it or understand where it's coming from, it can be quite a shock. Or at least he was, many years ago. Maybe he's changed.
I can see that. Sadly, there are a lot of people in the world who simply don't know how to deal with people who can be direct, if not somewhat abrasive, in their communication style. Their intent can be noble, well-intentioned, and not meant to offend. They simply don't beat around the bush or worry about whether your fragile ego will be bruised when they make an observation.
I've had to coach people and help them understand the entitlement involved in demanding that everyone adjust and adhere to their personal preferences and communication style. In my experience, it's about seeking to understand the person and adapt accordingly. Not everyone is willing to do that.
> Their intent can be noble, well-intentioned, and not meant to offend. They simply don't beat around the bush or worry about whether your fragile ego will be bruised when they make an observation.
I mean maybe, but maybe Carmac is just an ass hole... He can be a "legend" in the software development world and also just not be a super great person socially. The two things aren't mutually exclusive.
I don't disagree with you entirely, but being "direct" isn't a get-out-of-jail-free card for poor interpersonal skills. It's not always about "fragile egos" or "entitlement", it's about basic professionalism and communication.
For another take - what’s the game theory here?
If I’m kinda sensitive but also hyper-ambitious, I acknowledge that Facebook has
1. Some of the highest pay in the industry. 2. Ultra-competitive environment. 3. Low moral principles.
Seems like the strategy would be to use every lever at your disposal to manipulate your environment, rather than leave and risk getting paid less.
Expecting that you can change people, in my experience, is quixotic. What you can change is yourself and how you interpret and respond to events around you.
If I understand the hypothetical you've proposed, my advice would be for you to adapt and learn to be less sensitive rather than have you believe that you can manipulate the environment, or worse, directly manipulate people.
It's possible that you could be a positive influence for change, so I don't want to completed discount any effort there, but I also think it's worth being realistic about what you can actually affect.
But my point is that this is a highly competitive environment, the whole point in a highly competitive environment is to manipulate the environment and people. That’s what people get promoted for. So why not play the game? Again in the hypothetical if you are “less sensitive rather than believe you can manipulate the environment.” You lose.
But like it works for Musk and Trump, and probably hundreds of other leaders today, why not take their example? (Assuming again, your highly ambitious and competitive, I’m more pro social, so I’d take your route)
This is legitimately something I’ve been asking myself lately, we talk about a world that values one thing ( rationality, respect, pro social behavior) but reward another (pettiness, vindictive, selfishness). Why do we pretend?
Also, and maybe the most important point, John Carmack is 100% trying to manipulate his environment and people, that’s why he’s so successful!
The world is literally run by people who are good at manipulating people and their environment. That’s what an entrepreneur is, that’s what a politician is, that’s what an artist is. Your argument seems to mostly be people shouldn’t try to manipulate the world in a way that I don’t like.
Masters of doom portrays carmack as a total dictator of a boss. Doom Guy by John Romero seems to back this up
Masters of Doom does seems to want to, however accurately or not, set Carmack up as the antagonist of its story against Romero as the hero sometimes. I think that readers just largely didn't notice that since Carmack's heroic image was already so firmly established. In fact some of the early-ID stuff really does seem to raise some questions. (Was Tim Willits mostly Carmack's protégé, for instance?)
> I've only seen John Carmack's public interactions, but they've all been professional and kind.
You don't know someone or how they really behave because they are a public figure.
I’ve been on both the same side and the opposing side of debates with him, both in person and over internal discussion threads. His public persona and private behavior match. I viewed it positively, though per the topic of the thread, not everyone did.
That’s very different than simply observing someone in public. Which is what my post was referring too to and so it remains accurate.
FWIW I like carmack from what I have seen publically (and Romero, who I have interacted with) but I wouldn’t pretend to know who either of them really are from my observation of them.
> They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad
This is one of the reasons I’m sick of working pretty much anywhere anymore: I can’t be myself.
Appreciating people for their differences when they are humble and gifted is easy. I side with liberals, but I have a mix of liberal, moderate, and conservative friends.
But there are only so many years of pretending to appreciate all of the self-focused people that could be so much better at contributing to the world if they could quietly and selflessly work hard and respect people with different beliefs and backgrounds.
I’m happy for the opportunity I have to work, and I understand how millennials think and work. But working with boomers and/or gen X-ers would be so much less stressful. I could actually have real conversations with people.
I don’t think the problem is really with HR. I think the problem is a generation that was overly pandered to just doesn’t mix with the other generations, and maybe they shouldn’t.
It is very much not an objective discussion if you are discussing whether it makes sense to develop a new operating system.
Sorry but if you know his story, seen candid videos of him, or talked to the people around him, he's a Linus-level "I'll say what I want" type.
There weird hagiographies need to go. Carmack is absolutely not known to be kind. I have no idea what happened here but the idea that's he's this kindly old grandpa who could never, ever be rude or unprofessional is really out there.
And stupid. Like it or hate it, a non-nonsense, direct speaking, but fair and objective boss is the one you want. No one is served by failure; not the people at the top, nor the people at the bottom.
There is a difference between “this project is not going to work” vs “these people are incompetent and the project should be cancelled as a result”. The former needs to be said, the latter is a HR violation.
Carmack absolutely 100% percent did not say "these people are incompetent". What he said boils down to "these people are world's best experts on writing operating systems and they'd love to write a new one from the scratch but I strongly believe that writing a new operating system is not the best path forward."
> Carmack absolutely 100% percent did not say "these people are incompetent".
Sorry if I wasn’t clear; this was the point I was actually trying to make. Direct and to the point should not a HR incident make. I was trying to contrast with something that would.
I was at Google when the Flutter team started building Fuchsia.
They had amazing talent. Seriously, some of the most brilliant engineers I've worked with.
They had a huge team. Hundreds of people.
It was so ambitious.
But it seemed like such a terrible idea from the start. Nobody was ever able to articulate who would ever use it.
Technically, it was brilliant. But there was no business plan.
If they wanted to build a new kernel that could replace Linux on Android and/or Chrome OS, that would have been worth exploring - it would have had at least a chance at success.
But no, they wanted to build a new OS from scratch, including not just the kernel but the UI libraries and window manager too, all from scratch.
That's why the only platform they were able to target was Google's Home Hub - one of the few Google products that had a UI but wasn't a complete platform (no third-party apps, for example). And even there, I don't think they had a compelling story for why their OS was worth the added complexity.
It boggles my mind that Fuchsia is still going on. They should have killed it years ago. It's so depressing that they did across-the-board layoffs, including taking away resources from critically underfunded teams, while leaving projects like Fuchsia around wasting time and effort on a worthless endeavor. Instead they just kept reducing Fuchsia while still keeping it going. For what?
Wasn’t Fuchsia supposed to be a platform where different OS could run in a virtual environment and software packages would be complete containers? Was not this a new way of tackling the ancient OS problem?
These were my imaginations. I thought maybe an OS that could run on the web. Or an OS that could be virtualized to run on several machines. Or an OS that could be run along several other instances on the same machine each catering to a different user.
Reinventing QNX will be cutting edge for decades to come.
I guess it's just a political shit show at this point. Ideas go hard if the people behind them aren't playing the game well enough, no matter their value.
There's few things worse for the long-term health of a software project than people who have hammers and are hunting for nails for them.
Not only did they target Home Hub, they basically forced a rewrite on it (us, I worked on the team). After we already launched. And made our existing workable software stack into legacy. And then they were late. Then late again. And late again. With no consequences.
100% agree with your points. To me watching I was like -- yeah, hell, yeah, working on an OS from scratch sounds awesome, those guys have an awesome job. Too bad they're making everyone else's job suck.
And the crazy thing is there is arguably a lot more of a reason for Meta / Oculus to have had its own operating system because it is meant for a specific configuration of hardware and to utilize those hardware resources to a quite different goal than most other OSes out there. Even in that environment it was still a waste
I've written a lot of low level software, BSPs, and most of an OS, and the main reason to not write your own OS these days is silicon vendors. Back in the day, they would provide you a spec detailed enough that you could feasibly write your own drivers.
These days, you get a medium-level description and a Linux driver of questionable quality. Part of this is just laziness, but mostly this is a function of complexity. Modern hardware is just so complicated it would take a long time to completely document, and even longer to write a driver for.
Intel still does it. As far as I can see they're the only player in town that provide open, detailed documentation for their high-speed NICs [0]. You can actually write a driver for their 100Gb cards from scratch using their datasheet. Most other vendors would either (1) ignore you, (2) make you sign an NDA or (3) refer you to their poorly documented Linux/BSD driver.
Not sure what the situation is for other hardware like NVMe SSDs.
[0] 2750 page datasheet for the e810 Ethernet controller https://www.intel.com/content/www/us/en/content-details/6138...
The NVMe spec is freely downloadable and sufficient to write a driver with, if your OS already has PCIe support (which doesn't have open specifications). You don't need any vendor-specific features for ordinary everyday use, so it's a bit of a different situation from NICs. (Also, NVMe was in very large part an Intel creation, though it's maintained by an industry consortium.)
On the other hand, see the complete mess that are the IPU6/7 camera chipsets and their Linux support.
Yeah this. I tried to modify a hobby OS recently so it would process the "soft reboot" button (to speed up being rebooted in GCP) and it was so unbelievably hard to figure out how to support it. I tried following the instructions on the OS Dev Wiki and straight up reading what both Linux and FreeBSD do and still couldn't make progress. Yes. The thing that happens when you tell Windows or Linux to "restart". Gave up on this after spending days on it.
The people who develop OSes are cut from a different cloth and are not under the usual economic pressures.
I also think that they have access to more helpful resources than people outside the field do, e.g. being able to contact people working on the lower layers to get the missing info. These channels exist in the professional world, but they are hard to access.
The VMM on GCP has only really been tested with Linux. You are kinda wasting your time, the only way to make it work is to make the hobby OS Linux.
> You are kinda wasting your time, the only way to make it work is to make the hobby OS Linux.
Not the parent, but of course they're wasting their time... That's the point of a hobby OS.
I'm working on a hobby OS, and I have no illusions that it's most likely fewer than 10 people will ever run it, and less than 100 will hear about it, but it lets me explore some interesting (to me) ideas, and forces me to learn a little more about random pieces of computing. If I ran on GCP, I'd want the reboot button to work. That sounds useful.
On the topic, I don't see why anyone would want to build a general purpose OS. There's enough already and even with the shrinking of hardware variety, there's a lot of stuff to support to make a general purpose OS work on enough hardware for people to consider using it. You can take Linux or a BSD and hack it up pretty good to explore a lot of OS ideas. Chances are you're going to borrow some of their drivers anyway, and then you'll end up with at least some similarity... may as well start there and save a lot of time. (My hobby OS has a custom kernel and custom drivers, but I only support a bare minimum of devices... (pc) console i/o, one real NIC, and virtio-net... that's all I need; I might add support for more NICs and more consoles later)
I didn't say they were wasting their time on their hobby OS, they are wasting their time trying to get it to do very esoteric thing on GCP.
They aren't trying to get reboot to work, they are trying to get their version of kexec to work so their hobby os reboots faster.
https://wiki.archlinux.org/title/Kexec
The biggest scam in the OS world is drivers, we should demand more out of our hardware. Drivers shouldn't be necessary.
They said they wanted the soft reboot button to work. I assumed they meant catching the button press, which having seen some of this stuff is probably very tricky.
I don't see why a kexec alike wouldn't work about the same on GCP vs qemu vs bare metal... Or what that has to do with a GCP soft reboot button (which again, I think is referring to the reboot button in the GCP console)
Either way, the whole thing is a waste of time, yes? Why not waste time on the part that's engaging?
> The biggest scam in the OS world is drivers, we should demand more out of our hardware. Drivers shouldn't be necessary.
I can't even fathom what you mean here? You've got to have some interface to communicate with hardware. That's a driver. Some hardware only needs a very small driver... Tell the hardware where to send input, how to notify when input is ready and when its ready for output, and tell the hardware where data to output is. Maybe some setup stuff for modes and whatever if the needs aren't obvious and universal. I don't see how you could possibly avoid that.
It would certainly be possible for more devices to use common interfaces so a single driver could operate many different devices. Maybe that's what you mean? There's some movement towards that... SATA controllers generally speak AHCI, human interface devices generally appear as USB HID devices, etc. NICs tend to have a wide variety of setup sequences, but data queues usually fit into one of a limited number of patterns.
> Tell the hardware where to send input, ...
I agree you need a driver but for most hardware, that should be pretty simple, and easily documented by the hardware vendor, shouldn't it? A button has to be about the simplest possible I/O device imaginable.
Yeah, problem is it's likely an acpi button, which ties you into all the fun of that.
Of course, ACPI is supposed to make interfacing with lots of similar things easier, kind of, so there you go.
To clarify, are you having trouble getting the signal to reboot from the gcp console into your OS? Or are you having trouble rebooting on gcp?
Modern hardware is just so complicated it would take a long time to completely document, and even longer to write a driver for.
That's what's claimed. That's what people say, yet it's just an excuse. I've heard the same sort of excuse people have, after they write a massive codebase, then say "Oops, sorry, didn't get around to documenting it".
And no, hardware is not more difficult than software to document.
If the system is complex, there's more need to document, just as with a huge codebase. On their end, they have new employees to train up, and they have to manage testing. So any excuse that silicon vendors have to deal with such immense complexity? My violin plays for them.
> If the system is complex, there's more need to document
It’s not first party documentation that’s the problem. The problem is that they don’t share that documentation, so in order to get documentation for an “unsupported” OS a 3rd party needs to reverse engineer it.
> "Oops, sorry, didn't get around to documenting it".
That's obviously the wrong message. They should say "Go ask the engineering VP to get us off any other projects for another cycle while we're writing 'satisfying' documentation".
Extensive documentation comes at a price few companies are willing to pay (and that's not just a matter of resources. Look at Apple's documentation)
I write documentation as I'm writing the code. In my opinion, the code is only as good as its documentation -- they're two parts of the same thing. It's mostly comments at the top of files, and sometimes a markdown file in the same directory.
This way, good documentation is priced into my estimate for the project. I don't have a work item "spend a few days documenting." Nope, if I'm doing a foo then that includes documenting a foo at the same time.
In my experience, coding is much faster when doing it this way.
Yes, you can produce a small amount of code faster if you don’t “waste” your time on documentation, but that becomes counterproductive as soon as you can no longer keep the entire codebase in your head.
At least for certain types of OSes, it should be relatively easy to get most of Linux's hardware support by porting LKL (https://github.com/lkl/linux) and adding appropriate hooks to access hardware.
Of course, your custom kernel will still have to have some of its own code to support core platform/chipset devices, but LKL should pretty much cover just about all I/O devices (and you also get stuff like disk filesystems and a network stack along with the device drivers).
Also, it probably wouldn't work so well for typical monolithic kernels, but it should work decently on something that has user-mode driver support.
>but LKL should pretty much cover just about all I/O devices (and you also get stuff like disk filesystems and a network stack along with the device drivers).
thus calling into question why you ever bothered writing a new kernel in the first place if you were just going to piggyback Linux's device drivers onto some userspace wrapper thingy.
Im not necessarily indoctrinated to the point where I can't conceive of Linux being suboptimal in a way which is so fundamental that it requires no less than a completely new OS from scratch but you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.
probably it would have to be very general in design, but something along the lines of driverkit or iokit might work?
My hunch is that for nearly anyone who is serious about it these days, the way forward is either to have unusually tight control over the underlying platform, or to include a servant Linux installation with your OS. If Windows is a buggy set of device drivers, then Linux is a free set of buggy device drivers. If you're happy with your OS running as a client of a Linux hypervisor indefinitely then you could go for that; otherwise you'd have to try to gradually move bits of the hardware support into your OS over time—ideally faster than new Linux dependencies arise...
Presumably if you’re meta you could pay the vendors enough to write drivers for any arbitrary OS
Writing drivers is easy, getting vendors to write *correct* drivers is difficult. At work right now we are working with a Chinese OEM with a custom Wifi board with a chipset with firmware and drivers supplied by the vendor. It's actually not a new wifi chipset, they've used it in other products for years without issues. In conditions that are difficult to reproduce sometimes the chipset gets "stuck" and basically stops responding or doing any wifi things. This appears to be a firmware problem because unloading and reloading the kernel module doesn't fix the issue. We've supplied loads of pcap dumps to the vendor, but they're kind of useless to the vendor because (a) pcap can only capture what the kernel sees, not what the wifi chipset sees, (b) it's infeasible for the wifi chipset to log all its internal state and whatnot, and (c) even if this was all possible trying to debug the driver just from looking at gigabytes of low level protocol dumps would be impossible.
Realistically for the OEM to debug the issue they're going to need a way to reliably repro which we don't have for them, so we're kind of stuck.
This type of problem generalizes to the development of drivers and firmware for many complex pieces of modern hardware.
But is that a good use of Meta's money? Compared to making a few patches to Linux to fix any performance problems they find.
(And I feel bad saying this since Meta obviously did waste eleventy billion on their ridiculous Second Life recreation project ...)
I don't like Meta, but there used to be a time where big corp used to spend 30% of its budget on R&D. It's how we got all the toys we have now, R&D labs of big Bell and others.
So please don't mock the spend. Big spends fail sometimes, and at least people were paid to do the work.
The difference is that organisations like Bell Labs and Xerox PARC were primarily tech-first: innovations were the result of very clever and creative people doing blue skies research. The most groundbreaking stuff shocked the world while it was still a hacked-together demo, and similarly the cost of failure was quite low.
On the other hand, Meta's experiment is primarily CEO-driven. The outcome is predetermined, changing direction is not possible. Sure, clever engineers get to draw the rest of the owl, but that's not very useful when it turns out that everyone needs a horse instead.
They are spending a fortune, but rather than getting 900 crappy ideas to throw away and 100 great ones to pick from for continued development, they are developing 1 technological marvel nobody is interested in.
It was also pretty obvious how the VR glasses would support Meta's existing goals. It would give Meta total power over what you see and who you can speak with through their system. It's a natural extension of their total control over how people interact with on the Internet. And I think the only reason it failed is because it was expensive and dumb-looking.
Arguably the distinction you're pointing at is macroinvention (the transistor) vs microinvention (a better VR headset): one is a refinement of something that exists, another is transformative opening up whole new worlds of possibility. https://www.antonhowes.com/blog/macroinvention-vs-microinven...
It's just that it was so obviously going to fail, because there's no mass market for a product that you have to strap onto your face. You didn't need to spend billions to learn that.
If they'd spent the money researching nuclear fusion or space flight or a new way to develop microprocessors, I would be cheering their efforts even if they had failed in the end.
XROS had a completely new and rapidly evolving system call surface. No vendor would've been able to even start working on a driver for their device, let alone hand off a stable, complete result. It wasn't a case of "just rename a few symbols in a FreeBSD implementation and run a bunch of tests".
Vendors might say that they don't have the resources (man hours) and don't want to hand over documentation to external developers.
Yeah reverse engineering all the drivers is going to be a huge headache.
Sounds like super fun if I could be paid a bit for it.
What is an easy gate task to get into “reverse engineering some drivers for some OS”?
Second thought: I don’t even know how to write a driver or a kernel, so I better start from there.
I don't know how you get paid for it, but if you want to write your own kernel, I'd start with an osdev tutorial. started with this one [1], but this one [2] has a promissing name... and I haven't really looked around.
It helps to have a concept to guide you too, but you can certainly make some progress on the basics before you figure out what you really want to do.
[1] https://wiki.osdev.org/User:Zesterer/Bare_Bones
[2] https://osdev.wiki/wiki/Multiboot1_Bare_Bones
Thanks. I got all the resources covered. But I don’t have the energy to work on them as a side project any more. Alas! I wasted my younger days and hope you fare better!
Asahi Linux.
Was at Oculus post acquisition and can say that the whole XROS was an annoyance and distraction the core technology teams didn’t need. There were so many issues with multiple tech stacks that needed fixing first.
Mind you, this XROS idea came after Oculus reorged into FB proper. It felt to me like there were FB teams (or individuals) that wanted get on the ARVR train. Carmack was absolutely right, and after the reorg his influence slowly waned for the worse.
Just a small bunch of XROS people came from FB proper (mostly managers) because an average FB SWE has no required skills. Most folks were hired from the industry at E5/E6 and I think we had ever took one or two bootcampers that ultimately were not successful and quickly moved elsewhere in FB.
Mechanisms for getting the linux kernel out of the way is pretty decent these days, and CPUs with a lot of cores are common. That means you can isolate a bunch of cores and pin threads the way you want, and then use some kernel-bypass to access hardware directly. Communicate between cores using ring buffers.
This gives you best of both worlds - carefully designed system for the hardware with near optimal performance, and still with the ability to take advantage of the full linux kernel for management, monitoring, debugging, etc.
> use some kernel-bypass to access hardware directly
You can always mmap /dev/mem to get at physical memory.
No, that's not really what kernel bypass means.
> my old internal posts... got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad
That jives with my sense that META is a mediocre company
It matters who you communicate concerns to. Something as fundamental as "I think that your team shouldn't even exist" should go to the team leads and their managers exclusively at first. Writing that to the entire affected team is counterproductive in any organization because it unnecessarily raises anxiety and reduces team productivity and focus. Comments like this from influential people can have big mental and physical health impacts on people.
This entire situation looks very suspicious. Was Carmack even responsible for triaging research projects and allocating resources for them? If yes, then he should have fought that battle earlier. If no, then the best he could do is to refuse to use that OS in projects he controls.
Carmack had no direct say over research AFAIK.
It should be fine to give your opinion on efforts.
Yeah it sounds to me here like the urge to reach for HR had less to do with Carmack and more to do with the overall culture at Meta.
Not when this is his personal opinion he thought nothing should follow from.
"I think that your team shouldn't even exist" doesn't mean "I want your team to no longer exist.".
But the name Carmack carries some clout and people listen to him (too) closely because of his reputation alone. This is soft power that automatically comes with responsibility.
Maybe on a mediocre team. But that was the parent comment's point.
On well-functioning teams, product feedback shouldn't have to be filtered through layers of management. In fact, it would be dishonest to discuss something like this with managers while hiding it from the rest of the team.
If I was on that team I'd welcome the opportunity to tell John Carmac why he was wrong or if I agreed start looking for another project to work on.
When I was on nuclear submarines we'd call what you are advocating "keep us in the dark and feed us bullshit."
This assumes that you would be sincerely listened to, which you wouldn't in a case like this. Higher ups in large organizations don't have the bandwidth to listen to everybody.
Your sub's officers also need to constantly be aware of what to communicate to whom and in which language. Your superiors certainly kept you in the dark about a ton of concerns that were on their plate because simply mentioning them to subordinates would have been too distracting.
You say your piece and if not heard, do an internal transfer. This whole don't tell people the truth about technical matters to not hurt their feelings or disrupt some people's paychecks is not serious business.
> Comments like this from influential people can have big mental and physical health impacts on people.
So what are we supposed to do? Just let waste continue? The entire point of engineering is to understand the tradeoffs of each decision and to be able to communicate them to others...
I'm sure that kind of crap helped nudge JC out of there. He mentions (accurate and relevant) reasons why something is probably a bad idea, and the person in charge of doing it complains that JC brought up the critiques, rather than addressing the critiques themselves. What a pathetic, whiny thing to do.
You've got to remember that context is critical with stuff like this.
There's nothing wrong with well-founded and thoughtful criticism. On the other hand, it is very easy for this to turn into personal attacks or bullying - even if it wasn't intended to be.
If you're not careful you'll end up with juniors copying the style and phrasing of less-carefully-worded messages of their tech demigod, and you end up with a huge hostile workplace behaviour cesspit.
It's the same reason why Linus Torvalds took a break to reflect on his communication style: no matter how strongly you feel about a topic, you can't let your emotions end up harming the community.
So yes, I can totally see poorly-worded critiques leading to HR complaints. Having to think twice about the impact of the things you write is an essential part of being at a high level in a company, you simply can't afford to be careless anymore.
It's of course impossible to conclude that this is what happened in this specific case without further details, but it definitely wouldn't be the first time something like this happened with a tech legend.
Ugly people like to blame the mirrors.
Back in mmm like 2002 or 2003 or 2004 while at Microsoft I read an internal paper from a few OS guys who hackathoned something for Bill Gates's Think Week (which is when he used to go to some island in San Juans or somewhere similar and just read curated papers and think, it was a huge prestige to get such a paper to him) and that something was an OS written from scratch with GC and memory management on top of something very .NET framework'y (which was released a couple of years ago. They had it booting on all kinds of hardware and doing various neato things. One of explicitly called design principles was 0 compatibility with anything Windows before. Which is why it didn't go anywhere I assume. I remember it was just a handful of engineers (presumably OS folks) hacking for like a month. . It was awesome to read about.
Was it Singularity?
https://en.wikipedia.org/wiki/Singularity_(operating_system)
https://www.microsoft.com/en-us/research/project/singularity...
Singularity was cool. I'm sad that it was abandoned. The concept of using software isolation instead of hardware memory protection was really interesting.
It was a multi-year project at Microsoft Research with a team of >100 developers.
https://www.zdnet.com/article/whatever-happened-to-microsoft...
I am very certain in my recollection that this was started much earlier than this as hackathon skunkworks before something like this happened at MSR. It didn't do anything beyond kernel and command line, there was no browser. I don't know if those two shared roots either. Anyhow, but yeah, still both were intellectual feats!
What would be the real advantage of a custom OS over a Linux distribution?
The OS does process scheduling, program management, etc. Ok, you don’t want a VR headset to run certain things slowly or crash. But some Linux distributions are battle-tested and stable, and fast, so can’t you write ordinary programs that are fast and reliable (e.g. the camera movement and passthrough use RTLinux and have a failsafe that has been formally verified or extensively tested) and that’s enough?
I think the proper comparison point here is probably what game consoles have done since the Xbox 360, which is basically run a hypervisor on the metal with the app/game and management planes in separate VMs. That gives the game a bare metal-ish experience and doesn't throw away resources on true multitasking where it isn't really needed. At the same time it still lets the console run a dashboard plus background tasks like downloading and so on.
Maybe not applicable for the XR platform here, but you could add introspection capabilities not present in Linux, a la Genera letting the developer hotpatch driver-level code, or get all processes running on a shared address space which lets processes pass pointers around instead of the Unix model of serializing/deserializing data for communication (http://metamodular.com/Common-Lisp/lispos.html)
Based on the latter tweet in the chain, I'm wondering if Carmack is hinting that Foveated Rendering (more processing power is diverted towards the specific part of the screen you're looking at) was one advantage envisioned for it. But perhaps he's saying that he's not so sure if the performance gains from it actually justify building a custom OS instead of just overclocking the GPU along with an existing OS?
Wouldn't that be an application (or at most system library) concern though? The OS is just there to sling pixels, it wouldn't have any idea whether those pixels are blurry… well for VR it would all be OpenGL or equivalent so the OS just did hardware access permissions.
I think the context is that foveated rendering ties sensor input (measuring gaze direction) to the rendering pipeline in a way that requires very low latency. Past a certain point reducing latency requires optimizations that break normal abstractions made by user land, so you end up with something more custom. I'm not sure why that would require a whole new OS, the obvious path would be to put the latency-sensitive code onto dedicated hardware and leave the rest managed by Linux. If a bunch of smart people thought XROS was a good idea there's probably something there though, even if it didn't pan out.
Think you answered your own question. No real differences except more articles, $, and hype
And, let's be real here: engineering prestige.
Everyone wants to make an OS because that's super cool and technical and hard. I mean, that's just resume gold.
Using Linux is boring and easy. Yawwwwn. But nobody makes an OS from scratch, only crazy greybeard developers do that!
The problem is, you're not crazy greybeard developers working out of your basement for the advancement of humanity. No. Youre paid employees of a mega corporation. You have no principles, no vision. You're not Linus Trovalds.
My objection is that there is no universe in which Meta can be trusted with direct access to your raw gaze tracking data.
The only thing I can imagine that would be more invasive would require a brain implant.
My understanding is that this is a key tenant of visionOS’s design, where apps don’t get access to gaze data (I think unless they’re taking over the full screen?)
sadly they are working on it
I was there when they wanted to do the custom XROS. I remember asking them in a Q&A session exactly why they would build this and I recall the reasoning behind it totally fell flat. Fundamentally it became clear these guys just wanted to write a new OS because they thought it would be cool or fun.
Much of the scenarios they tried to address could have been done with Mach or some realtime kernel or with fuchsia. I recall later on they did consider using fuchsia as the user space for the os for some time.
On another note, there was similarly an equally “just for fun” language effort in the org as well (eg “FNL”). That was also conceived by a bunch of randos who weren’t a bunch of compiler guys that had no product vision and just did it for fun.
Well when the era of efficiency arrived all of this stuff ended.
I worked on a completely different hardware project within meta and while they didn't want a custom OS, they used an off the shelf rtos with the intention of modifying it and it was a shit show. They had a million justifications for why they needed it, but they had no performance tests or metrics to verify to actually justify it. They incurred a huge development overhead for no verifiable performance improvements.
None of the code they wrote couldn't have just been written as a kernel module in Linux. It would've also been so much easier due to all the documentation and general knowledge people have about Linux both within the company and outside the company.
I think people have forgotten about Google Fuchsia which I guess is a good sign for a new OS. They’ve done quite well in deploying it seamlessly to their consumer devices.
"Quite well" by what metric? It shipped on one device. That's pretty much the lowest bar you can imagine! Did it provide any tangible benefit to anyone? Let alone a benefit commensurate with the enormous cost of developing it and continuing to maintain it?
I think it was insane to start a new OS effort written in C/C++. We have plenty of OSes written in C/C++! We know how that story ends. If you're going to spend the effort, at least try a new language that could enable a better security model.
It was so good Google cancelled plans to use it in meaningful products and instead delegated it to the bottom shelf products.
Fun rumor: Google shut down the AR effort and transferred the team to project Fuchsia as a way to retain highly skilled employees. So essentially they didn’t have any real technical needs for a new OS.
Fuchsia is pretty much a dead product at this point. For things like Phones and Laptops google is using only Android going forward.
I thought they rolled back all of those efforts. What devices shipping today come with Fuchsia installed?
According to Wikipedia, looks like only the Nest Nub.
I think some of the Nest audio devices now run it as well.
Google Nest Hub
Isn't that somewhat debatable? Originally they were aiming at much more (chromebook OS for example) but seems like they settled for Google Home only as their scope.
Still a very interesting project, but that feels like a similar story, for limited use cases (a smart thermostat/speaker with specific hardware) it works, but for wider use cases with heterogeneus hardware and complex interfaces (actual screen, peripherals) it didn't work.
XROS was a Fuschia fork, actually.
The reviews I've seen of its stability and usefulness have not been good.
That was also frustrating in that sel4 was right there. Why not invest efforts in the existing thing?
The microkernel core isn't the buggy/hard part; Fuscia's got a similar microkernel that also works great. It's everything else that's hard.
Yup it’s because they had to rewrite the drivers I presume. Always going to be the biggest issue with any new OS.
Huawei seem pretty committed to building their own OS and uncoupling from the Western technology stack in total
https://en.wikipedia.org/wiki/HarmonyOS_NEXT https://www.usenix.org/conference/osdi24/presentation/chen-h...
The only reason Chinese companies can even get away with these big projects is because of state backing and state objectives. By itself, the market doesn't support a new general-purpose OS at this point.
> because of state backing and state objectives
MS is a state backed company. Very natural that China went the same path.
No it isn't
Technically you are correct but the commenter you’re responding to means that with the amount of Western Governments spend on MS products and services, the are a d facto (if not de jure) state backed enterprise.
US government spending is (for now) easy to track, and you can get totals for spending by corporate entity.
In total across the entire US federal government, $518.8 million was paid to Microsoft for products and services in 2024. That is approximately 0.21% of their total annual revenue.
I assert that the threshold for "state sponsored" is well in excess of 0.21% of annual revenue.
Federal Spending: https://www.usaspending.gov/recipient/dd77b7c3-663e-cb91-229...
Microsoft Annual Revenue: https://www.microsoft.com/investor/reports/ar24/index.html
Yes, but the power play seams to be more like MS backes several states.
MS has deep ties into the state department and intelligence apparatus that few other companies do. Just as deep as the defense contractors who have a near monopoly-monopsony relationship with the federal government. You can argue about how exclusive they are in particular qualities but the scale and depth they operate at makes their relationship approximate the relationship Huawei does with the Chinese government. They're just what state-backed enterprises look like under liberal-ish capitalism.
You're downvoted but you're 100% correct.
It makes absolutely zero financial sense to create a new general purpose operating system.
That's billions of lines of code. With a B. And that's just the code - getting it to work with hardware?
Do YOU want to talk to 10,000 hardware vendors and get them on board? No! Nobody does! That's just money burning!
But, there are valid political reasons for creating a new general purpose OS.
They actually reuse Linux driver stack for hardware compatibility
You could write a book on why it's practically impossible to create a new OS these days. Love Carmack for stating it so clearly. I also love that called out TempleOS, I also have a weird respect for it. Plan 9 is the probably the best example of a totally new OS and I hope someday it becomes viable because it's really a joy to use.
But ultimately it just makes sense to adapt existing kernels / OS (say, arch) and adapt it to your needs. It can be hair wrenchingly frustrating, and requires the company to be willing to upstream changes and it still takes years, but the alternative is decades, because what sounds good and well designed on paper just melts when it hits the real world, and linux has already gone through those decades of pain.
I would read that book.
OS isnt the hard part.
The driver ecosystem is the moat. Linux finally overcame it decades later
Android built a new, giant moat for Linux (or "Linux" depending on your opinions about Android) in the embedded application processor space - now the "standard" board support package target for new embedded AP hardware is almost always some random point-in-time snapshot of Android. Running "mainline" Linux is hard (because the GPU and media peripheral drivers are usually half-userspace Android flavored stuff and rely on ION and other Androidisms) and bare-metal is even worse (where previously, you'd get register-level documentation, now, you get some Android libXYZ.so library).
Yeah, the Linux kernel has ~12m lines of code. <1m are the core, the rest are drivers.
29 Millions according to https://www.phoronix.com/news/Linux-6.16-AMDGPU-Driver-Size
5 Millions alone for the AMD graphic driver.
Much of that 5M are hardware register definitions expanded into C headers. I am not sure how you'd consolidate that but it's not like that's all bespoke C code.
What an inefficient disaster to carry around all those drivers to run hardware that isn't on the device you're running on.
* The vast majority aren't compiled into any given kernel
* Those that are are only loaded when needed
It's not that bad
I sympathize, but the reality is that except for very specialized cases, (hyper) optimizing for CPU performance is unnecessary, even in the embedded space. A Cortex-M0 has roughly the same performance as a 486, and is cheap and power efficient enough to be bundled in disposable test kits, vapes, etc.
Most of it is in modules that don't have to be compiled at all (and usually aren't)
It's modular so you don't have to compile drivers you don't need.
Why bother making a new OS when you can make a new user interface for an existing OS?
The drivers are the hard part. It takes a lot of inter-industry collaboration to get driver compatibility
I wonder whether an unarticulated desire of Meta's was to avoid any license issues that could incur by using Linux or any other existing OS.
When you're at a certain scale it makes sense.
That scale is when creating an OS gives you a clear advantage over licensing or working with an open source OS.
Every other scale below that it's for knowledge, growth, research, or fun.
Is the difficulty in theoretical complexity of operating systems, or in project scoping/scope creep?
It's probably not that hard to write bare metal code for a modern CPU that runs and crashes. It's obviously insurmountably hard to compete with Android in features with scratch built bare metal code. An "OS" can be anything between the two. But it's very easy to imagine an "XR OS" project snowballing quickly into the latter, and Carmack's concerns would be spot on(as always is, and as proven). Is it then an inherent difficulty in "designing a new operating system", or is it technically something else?
XROS definitely suffered from scope creep and an outrageous set of goals/priorities.
> To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers.
I mean, I'd give a fair shake to an OS from the SQLite team [1].
1. https://sqlite.org/codeofethics.html
Where do I apply to join this monastic order of OS programmers?
Just walk up to the gate of your nearest Concent next Apert and they will take you right in!
Actually, I don't know how you join the Ita now that you mention it.
This is completely right from a product point of view, which is Carmack's argument.
But I have wondered why one of these companies with billions of dollars to burn hasn't tried to create something new as a strategic initiative. Yes, there wouldn't be any ROI for years, and yes, the first several products on the platform would probably be better off on something more traditional.
But the long term value could potentially be astronomical.
Just another case of quarterly-report-driven decision making, I suppose. Sigh.
> But I have wondered why one of these companies with billions of dollars to burn hasn't tried to create something new as a strategic initiative.
See Google's Fuschia: https://en.wikipedia.org/wiki/Fuchsia_(operating_system)
> But the long term value could potentially be astronomical.
Such as what?
> Such as what?
Historically? The internet, the concept of a graphical user interface, the mouse, the smartphone, the LCD display, the laser printer...
It's about clever people trying weird stuff, and occasionally ending up with a world-changing idea. Asking for examples of to-be-discovered innovations is, by definition, an impossibility.
Companies are risk averse, ask DARPA to do it.
This is what Google has been trying to do with Fuchsia and the fact is that you can't escape the product point of view because the products exist, already have an OS stack, and get pretty defensive when another team tells them they're going to replace their OS, or their core if the product team is Android or Chrome OS.
How does it pay off in the long run?
If you're competing against nothing, then I see it: it opens up a wide variety of product possibilities. But linux exists. Why not spend 1/1000th the time to adapt linux?
That's not even counting the rather substantial risk that your new OS will never approach the capabilities of linux, and may very well never become generally usable at all.
Option A: spend years and millions on a project that may never be as good as existing solutions, diverting attention and resources from actual products, or...
Option B: work on products now, using an existing, high-quality, extensible, adaptable OS, with very friendly licensing terms, for which numerous experts exist, with a proven track record of maintenance, a working plan for sustainability, a large & healthy developer community exists, etc.
It's hard to imagine how it wouldn't be a complete waste of time.
How would that be better than just grabbing a bsd and starting with that, PlayStation and Apple did it and actually ended up with functional products
Hell, if you're Meta you could just by QNX from RIM.
I wonder how well that would play out. Surely you'd only do if you got a perpetual license or bought it outright. I can imagine RIM would fuck you over eventually with with any other arrangement.
(Actually I just realised you meant "buy")
Apple bought one of those in the 90s, and they are still reaping the benefits of that strategic initiative. But the thing is, NeXt allowed Apple to think up new, differentiated products. If you come at the problem of the OS from a purely technical perspective, you'll waste time for no gain.
While I appreciate Carmack and all, I'd love to hear from someone like Dave Cutler who's been over that bridge at least a couple of times successfully about if and what he'd do if he had resources to create whatever the hell he wants.
Microsoft had Singularity - canceled after 12 years in development
Google has Fuchsia - is about 10 years in development. Recently was a target for layoffs
Western companies haven't thought about long term value in decades
But I have wondered why one of these companies with billions of dollars to burn hasn't tried to create something new as a strategic initiative.
They have; Taligent comes to mind. You may not have heard of that -- or more likely, you have but just forgot about it -- but it's a good object lesson (no pun intended) in why a successful new OS is hard to just conjure into existence. There has to be a crying, desperate need for it, not just a vague sense that This Time We'll Get It Right.
You could probably cite OS/2 Warp as a less-obscure example of the same phenomenon.
What if instead of writing the entire OS, a company were to pick up an existing “hobby” OS and refine it?
For example any of the systems listed in Carmack’s post. Or perhaps Serenity OS, RedoxOS, etc.
In that case, why wouldn't they "just" fork Linux? Or 10-years-ago-Linux?
The technical justification for Meta writing their own OS is that they'd get to make design decisions that suited them at a very deep level, not that they could do the work equivalent of contributing a few drivers to an existing choice.
I'd love a truly new OS, but I just don’t know what it would look like at this point? "New OS" ideas tend to converge on the same trunk.
Building a hobby OS taught me how little is just "software". The CPU sets the rules. Page tables exist because the MMU says so. Syscalls are privilege flips. Task switches are register loads and TLB churn. Drivers are interrupt choreography. The OS to me is just policy wrapped around fixed machinery.
try to quickly spawn a lot of processes on windows
> I wish I could drop (so many of) my old internal posts publicly, since I don’t really have the incentive to relitigate the arguments today – they were carefully considered and prescient. They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad, but I expect many of them would acknowledge in hindsight that the Meta products would not be in a better place today if the new OS effort had been rammed into them.
So someone at Meta was so sensitive that being told their behemoth of a project was ill advised ended up getting reported to HR?
Yep, I can 100% believe it having worked in such environments
> To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers. Which was sort of Plan 9…
Roll call!
Jonathan Blow is the world’s most successful hobbyist programmer. His whole thing is doing projects from scratch. Every game he made could be done in Unity with far less effort.
Most opinions of this man exists in a vacuum space isolated from the real world software industry. Building an OS from scratch is one of those examples.
It’s never seems like there’s a significant reason behind them other than………”I made dat :P”
I don't think unity was as polished when braid came out in ~2008 that can also easily rewind time on low end Xbox hardware. The witness maybe in unreal? But there are some wild things there I've never seen an unreal game do that the witness does do
As an outsider...his games just look and feel different. They feel like bones-deep art, in a way that even the best of the best games (say, Hades) don't. Since Blow's games are puzzle games they're not even my favorite games! But the effort spent on making them exactly the way he wants them pays off.
I love this part: "To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers"
Did they have people who have built an OS before?
I’ve seen this firsthand. These giant tech companies try to just jump into a massive new project thinking that because they have built such an impressive website and have so much experience at scale they should just be able to handle building a new OS.
In my case it wasn’t even a new OS it was just building around an existing platform and even that was massively problematic.
The companies that build them from scratch have had it as one of their core competencies pretty much from the start.
I’m unsurprised meta had issues like this.
> Did they have people who have built an OS before?
Yes.
They have contributors to the linux kernel. Pretty sure all the big tech companies have the right people to create a new OS that is better than Linux, the hard part is getting that new OS to be adopted.
How about ReactOS?
And yet, Sony did it, Nintendo did it, both have been pretty succeesful.
We also need to be clear what an OS is. Is it "darwin" or "macOS" - they have different scopes.
Things I'd want from an OS for an XR device.
1. Fast boot. I don't want to have to wait 2-3-4-5 minutes to reboot for those times I need to reboot.
I feel like Nintendo figured this out? It updates the OS in the background somehow and reboot is nearly instant.
2. Zero jank. I'm on XR, if the OS janks in any way people will get sick AND perceive the product as sucking. At least I do. iOS is smooth, Androind is jank AF.
Do any of the existing OSes provide this? Sure, maybe take an existing OS an modify it, assuming you can.
You mean for the PlayStation? That is a FreeBSD fork, probably chosen over Linux because of the license.
Nintendo is an interesting example though. According to Wikipedia they actually use a proprietary microkernel, which, if I'm reading this right, I think they developed themselves. Looks like the only open source components they have is some networking code which is published under the BSD license.
https://en.m.wikipedia.org/wiki/Nintendo_Switch_system_softw...
PS4/5 was BSD. Only Nintendo and Microsoft runs own corporate proprietary OS.
Sony and Nintendo both forked off of either NetBSD or FreeBSD. Sony's cameras at least up until the A7S2 run Linux (there's jailbreaks for these), although I never found any kernel / bootloader dump like it would be required.
Android suffers from being Java at the core, with all the baggage that brings with it.
Sony forked FreeBSD, but Nintendo didn't. They have BSD license headers because of some BSD socket code they include.
Sony cameras all run Linux. Models with "PlayMemories Camera Apps" feature like A7M2 support runs Android userland on top. It's probably easier to count the cameras that don't(like old Olympus).
Lost me and TempleOS
The problem with this guy is that it’s hard to criticize him, whether at work or in this forum. For example, I am going to be downvoted for mocking the fact that this guy thinks it’s some genius move to say “No” to making an operating system, whatever making an operating system means.
You can prove him wrong by doing thing what he claims should not be done.
See? This is why the Doom man got paid the big bucks. The Doom man has a literacy distortion field.
I like that the top reply to Carmack's wall of text is a screenshot of TempleOS with a doodle of an elephant lmao. And ironically, that meme reply is on topic and it says a thousand words with just one photo.
Another point I would add in support of that meme comment, is Google's recent rug-pull of Android not allowing sideloading apps from unsigned developers anymore starting this autumn, after over a decade of conquering the market with their "go with us, we're the open alternative to iOS" marketing.
The conclusion is to just never EVER trust big-tech/VC/PE companies, even when they do nice things, since they're 100% just playing the long game, getting buddy-buddy with you waiting till they smothered the competition with their warchest, and then the inevitable rug-pull comes once you're tied to their ecosystem and you have nowhere else to go.
Avoid these scumbags, go FOSS form the start, go TempleOS. /s but not really
In other words, unless God has specifically called upon you to build an OS, and maybe provided divine inspiration and assistance, you should avoid doing that. Seems to support Carmack's point!
Unless its for love or devotion, there's no compelling reason to create a new OS in 2025. Certainly that could change in the future but I think his observation (if I understand him correctly) is correct.
I'm not sure Carmack's point disagrees with you. Meta is still big tech, and if your goal is to monetize at scale, rolling out your own isn't the most efficient way to do it. I don't think he'd discourage you rolling out your own OS if it's your hobby FOSS project.
He's acting like their VR UX is top notch when it's as bad as it gets. Just yesterday I dusted off my Meta Quest 2 to play a bit, and spent around an hour trying to pair up my left controller to the helmet after replacing the battery.
You can't do it without going through their fucking app, that asks for every permissions under the sun, including GPS positioning for some reason. After finally getting this app working and pairing it with my headset, I could finally realize the controller was just dead and their was nothing to do.
You can pair the controllers in the settings you don't need an app. Their VR UX does suck that is true, and horizon worlds is such a collosal failure that I'm surprised they haven't cancelled that entirely yet. But carmack also stated the technical issues numerous times.
> You can't do it without going through their fucking app, that asks for every permissions under the sun, including GPS positioning for some reason.
If it uses bluetooth, which it might for the controller?, the permission for bluetooth on Android is fine location --- the same permission as for using GPS. That might be the same permission you need for wifi stuff, too? Because products and services exist to turn bluetooth and wifi mac addresses seen into a fine location.
But who knows what they do with the GPS signal after they ask for it?
https://xcancel.com/ID_AA_Carmack/status/1961172409920491849
Didn't even realize this was a thread and not a single tweet until you posted this link. Guess that's the downsides of not having a Twitter acct anymore.
We'll put that link in the top text too. Thanks!
Neat website, thanks for posting. Basically necessary to avoid the twitter “paywall”
> they also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad
This is madness. The safe space culture has really gone too far.
I'll offer a different interpretation:
If a professional can't give critical feedback in a professional setting without being rude or belittling others, then they need to improve their communication skills.
This is not that though. This is just developers being unable to handle constructive criticism, and when they can't win the argument on merits, went for the HR option. It happens.
I've had it happen to me too, but my response was to resign on the spot (I was already not satisfied with the company).
The "toxic behaviour" I had done? I reverted a commit on the master branch that didn't compile, and sent a slack to the Dev who had committed it saying "hi! There appears to have been a mistake in your latest commit, could you please check it out and fix it? I've reverted it in the meantime since I need to deploy this other feature"
The dev responded by force pushing the code that did not compile to master and contacted HR.
I decided there was greener grass on other pastures. I was right.
When I started breaking the build would end up with the person who did it having to wear the dunce hat for the day. This was before git so there are now fewer excuses for breaking the build today.
Carmack says he can't post the actual discussion, but clearly you have access to it - can you post it?
Even if that didn't violate NDAs & etc, that would be super rude. You realize that, right?
From what you know of Carmack, does "can't give critical feedback in a professional setting without being rude or belittling others" sound like him to you? It does not to me, though granted maybe he's different in his non public persona than what you can see in presentations and talks.
This.
Being "reported to HR" doesn't mean "almost got fired". It likely meant a meeting where someone explained "hey, the way you communicated that caused some upset, let's discuss better ways to handle that situation next time." Very often in larger companies, complaints about things like "this bigwig from this other group jumped all over us" are automatically sent through HR because HR has staff whose job just is resolving conflicts between people and keeping things peaceful.
FWIW, it can also mean a meeting where HR says to the complainer: "Are you good, dude? Why are you complaining about this? This is perfectly normal workplace behavior. Stop wasting our time."
Having worked in the valley, I've seen what critical feedback meant in many companies there, and it removes all usefulness of the info because there is a ceiling of what is socially acceptable to say; therefore, you can't know how bad or urgent things are.
Everything is ASAP. They are super excited about everything. And nothing you do is wrong, it just could be improved or they like it but don't love it.
You don't know if something is important, basically.
Just like Louis CK said, "if you used 'amazing' on chicken nuggets, what are you going to say when your first child is born?". But in reverse.
Personally, I'd rather work with someone who would tell me my work is terrible if it is.
In Germany, you can't even legally say somebody did a bad job at your company in a recommendation letter. Companies created a whole subtext to workaround that, it's crazy.
Some things are just bad. You should be able to say it is. Not by saying it could be better. Not by using euphemism. It's just something that needs to go to the trash.
In fact, I don't trust people who can't receive this information, even if not packaged with tact (which you should attempt to, but life happens). If you can't handle people not being perfectly polite every time, I can't help but feel I won't be able to count on you when things get hard.
That must be the French in me talking.
> In Germany, you can't even legally say somebody did a bad job at your company in a recommendation letter. Companies created a whole subtext to workaround that, it's crazy.
I don't think it's just about legality. Whether the recommendation letter is included in the application is at the distinction of the applicant. When you want it to reach the next company, you must write is so, that the former employee considers it to be a good recommendation.
I'd just quote Mourinho / Carlsen. "If I say anything I'm in big trouble".
I am with you on this. Worked in our Dublin office was so refreshing to have straight up communication vs whatever the song and dance we do here in US.
You've concluded this from a single, brief, throwaway line? Any madness you perceive about this situation has been fabricated by you, based on the details we have.
People have been getting mad at being made to feel bad at work for much longer than “safe space culture” has existed. If someone or some team had more power than you at an organization you for sure will get reprimanded for making them feel bad.
A meeting with HR is not madness. No one got maimed or died, or even lost work, seemingly. Some people exchanged words.
Cool off.
Something tells me that if we heard the other side of the story it might hit different. There's a lot of wiggle room in what "making his team members feel bad" could mean, and I would be surprised if constructively voiced criticism would have gotten someone written up.
With my experience of being written up for constructive criticism the reasoning was that I didn’t give constructive criticism to others and they felt singled out. I only give such criticism in private so of course they were not there to see the others. Apparently that wasn’t a sufficient explanation.
it is madness, you would be surprised how many ppl take things too serious. been there, had talk with HR cause i've said that the solution is mediocre and we have to do something better than that.
Reading between the lines, it sounds like he got reported for giving a lot of what might kindly be described as unsolicited advice. The guy left Meta ages ago, but he apparently still can't let this one go.
If you're in the middle of trying to write a new operating system, then it's probably not helpful to have John Carmack standing over you repeatedly telling you that you shouldn't be doing it. In this case Carmack gets the last laugh. Then again, it is easy to get the last laugh by predicting that a project will fail, given that most projects do.
> unsolicited advice
He was the CTO of Oculus. Surely it is appropriate for the CTO to give advice on any big technical decisions, if not outright have veto power.
Clearly he didn't have veto power on the project he's talking about.
> If you're in the middle of trying to write a new operating system, then it's probably not helpful to have John Carmack standing over you repeatedly telling you that you shouldn't be doing it. In this case Carmack gets the last laugh. Then again, it is easy to get the last laugh by predicting that a project will fail, given that most projects do.
I mean, if you're working on a project that is likely to fail, wouldn't it be nice if someone gave you cover to stop working on it, and then you could figure out something else to do that might not fail? Can't get any impact if your OS will never ship.
tbh linux has quite a bit of cruft in it these days at the syscall and interface layer.
if youre apple, it does make sense to do stuff from scratch. i think in a way, software guys wind up building their own prisons. an api is created to solve problem X given world Y, but world Y+1 has a different set of problems - problems that may no longer be adequately addressed given the api invented for X.
people talk about "rewrite everything in rust" - I say, why stop there? lets go down to the metal. make every byte, every instruction, every syscall a commodity. imagine if we could go all the way back to bare metal programming, simply by virtue of the LLM auto-coding the bootloader, scheduler, process manager, all in-situ.
the software world is full of circularities like that. we went from Mainframe -> local -> mainframe, why not baremetal -> hosted -> baremetal?
Apple doesn't do a lot of baremetal development from scratch that I'm aware of. The Lightning to HDMI dongle bootstraps an XNU kernel with an AirPlay decoder into 256MB RAM, for instance.
Doesn't Apple pretty much own their whole stack? You mentioned XNU, which they made...
Their CPU, their boards, their firmware (presumably), their OS[1], much of the perhipherals are theirs, too.
Lots of companies try to emulate Apple, but it's very hard to pull off.
[1] Yes, they use some parts from Mach and FreeBSD in their OS, but the amalgamation is theirs, and they support and change the whole thing as needed.
You can still do “unsafe” stuff in rust, and people do. It’s perfectly possible to write safe C and C++ these days. And you don’t have to deal with a borrow checker, and a very small pool of developers available to hire.
> It’s perfectly possible to write safe C and C++ these days.
It's also very hard to do so.
The chrome team doesn't manage that, but perhaps they are just bad at C++ /s
oh, i didnt mean to invoke rust in any technical sense - i brought up rust to introduce an example of the attitude that rust people are known for, namely "why not rewrite everything?", which a lot of people have a kneejerk rejection of.