This is a much better experience than the previous Qualcomm debug experience, which was a hand-rolled set of read/write/execute primitives exposed over USB. It was hilariously undersecured, allowing a few of us to continually get root on various Qualcomm models.
In seriousness, these debug ports are seriously lacking in most mobile chipsets. MediaTek still has the old-style approach in many of their devices, requiring some incantations which expose serial over USB, but not in the way you think -- it's serial over USB pins!
I've done tonnes of work with mobile chipsets and security and this seems like they've finally started down the road to making this functionality accessible. Don't be surprised if you don't see this supported out of the box in most places, though. Most OEMs will certainly disable this once they've adapted their bootloaders to it. The big G doesn't like debuggability in end user devices.
It's just a UART; you can use the UART to debug the device in various ways.
On Pixel devices, the UART is not configured or brought up by default in locked production mode (as things should be), but by unlocking the device and then using `fastboot oem uart enable` you can flip the bits to turn it on. On early Pixel devices it was on the headphone jack and on newer ones it's on the SBU pins.
By default I think it's still configured as the kernel console in the kernel command line, so once it's enabled it will show the kernel debug output and present a TTY. But of course you can subsequently configure it to do whatever you'd want a UART for: kgdb for kernel-debugging, earlier stuff in the bootloader, and so on.
So, the implications are just: there's a convenient debugging interface available to you that turns on if you unlock the device and ask for it.
On Chromebook devices there's a more complicated and fancy debugging system where the SBU pins can be muxed to the security processor's USB host interface by presenting a debug cable called a SuzyQ, which presents a whole suite of debugging facilities. This used to be used quite frequently for unbricking purposes.
On the newer pixel phones (starting with the ones containing the titan chip) you can also mux the SBU pins to the security chip USB interface with "fastboot oem citadel suzyq".
And BTW, the SuzyQ cable is nothing more then two pull up resistors and a USB hub connected to the normal usb D+/D- pins on one port and the SBU pins on the 2nd port. Nothing fancy about it, people have even made their own (minus the hub) by soldering some wires and resistors to a usb-c breakout board. Google has also published the schematics for it:
Been wanting to play with the citadel interface. Last time I looked at it there was 0 public information, but that was years ago. And if there still isn't, then it'll be a good excuse to RE
Also, the titan chip has been around since the Pixel 3 I believe, so "newer" might be a strong word :)
Most of those boards have a separate physical JTAG connector (at least in development kits, this article indicates JTAG over USB is disabled in production systems anyways so no difference there) which is what they are expecting you to use for low-level debugging. It only costs like 1,000 $ for a JTAG probe which is like 1 fully-burdened engineer-day of cost. Even fully featured probes enabling hardware trace and time-travel debugging only cost like 1 engineer-week.
> Most of those boards have a separate physical JTAG connector (at least in development kits, this article indicates JTAG over USB is disabled in production systems anyways so no difference there
There's generally an entire phase of prototyping where engineers will be using production boards but still need JTAG, which is why it's fused and why these kinds of features exist. It's a lot easier to have your lower-level software team (drivers/BSP, perf, etc.) sitting with production-ready units provisioned with engineering keys and debug enabled than to have them having to use some kind of case-off JTAG header setup, cost aside.
The probes cost enough to exceed individual purchasing limits at hardware companies, which means you need to go through the requisition process. That takes long enough that you have to plan ahead and you don't order more as your needs increase. Then everyone's fighting for the limited probes right before a ship date and they get jealously guarded like priceless jewels.
JTAG also isn't usually exposed through enclosures, so using the probe on a field unit might require destructive entry depending on the application.
Well the problem there is companies who are too stupid to invest in cheap tooling with massive ROI for their developers. A pretty constant problem in software development.
And I am not knocking JTAG over USB. It is certainly convenient and beneficial since you can enable it in production or deployed units. I was commenting on how the GP (and even article) was making it out to be missing capability. They just do not have the cheap tools that are the intended way to access that capability.
edit: The article even mentions how the "Qualcomm Landing Team at Linaro", which seems to be the team that works with pre-production hardware to get them working on launch day, has a development process where "debuggers have never been a staple of our work for all the typical reasons you'd expect (cost and complexity being the main ones)". That is literally the team that should have pre-production units in the lab which will have debug connectors and where JTAG probes should be par for the course, yet they are apparently hardly using them partly because of "cost".
The landing team's job is to "land" patches into upstream. They take Qualcomm code and spin straw into gold until it's eventually good enough to contribute to projects like the Linux kernel.
Having read a lot of Qualcomm code myself, I don't envy their job.
There's something wrong if you're using Jtag on a field unit. In my experience, once a chip is up and running we implement higher level communication and debug methods.
To give an automotive example, a field unit may fail and be RMA'd to the tier 1 to improve their production line. They don't have access/documentation for the production software, and it wasn't developed to interact with their diagnostic tooling. They'll need to reliably reflash their own test software onto a board with potential hardware failures. JTAG is sometimes what they request to do that.
Sadly these two things can be true at the same time. As a hacker I love devices which (maybe accidentally) expose a way for me to run arbitrary software on them. At the same time as a person carrying a computer in my pocket containing my entire life I'm not a fan if that allows anyone grabbing said computer to run arbitrary software on it.
The ideal here is the standard approach where a user has to explicitly take some sort of action which can't be found accidentally and done without understanding the implications to activate debug mode. On Android that's tapping a specific item in the about menu way more times than you might think would be necessary. On iOS it's hooking up to a computer, then authorising the connection on the phone. Either way you get both the ability to run arbitrary(ish) software without someone picking up your device when left on a train being able to do so.
You bring me back to the olden days of QXDM in my adolescence, when I used it to save myself from bricking my phone, and again when I had my first internship working on an MDM9x50 device
MediaTek still has the old-style approach in many of their devices, requiring some incantations which expose serial over USB, but not in the way you think -- it's serial over USB pins!
Wasn't that only in their old feature-phone (MT62xx) SoCs? All the smartphone ones AFAIK (at least since the MT657x days) use USB CDC in their BootROM and preloader.
It will be really interesting to see what production devices this is enabled on - It mentions the OnePlus 6 at least which has it fused out but is still accessible.
Edit: How are they reading the eFuses on a production OnePlus 6? Do they have a Qualcomm-signed EL3 EDL loader?
It seems to exist as qcom,msm-eud in the device tree of a (unfortunately production) SM4350 device I have along with an eud_enable_reg. Time to recompile the kernel with `/dev/mem`.
I'm not at all sure on the interpretation of this, but the reading at the efuse_addr (so I guess certain ones can be read from EL0?) is 0x0e000000 which has bits 25-27 set and QFPROM fuses seem to have a blown value of 1 according to Qualcomm docs, so it might be fused out?
heya, author of the blog post here. the op6 happens to work but it's limited to debugging in el1, if you do manage to trap in el2 all the registers read 0
the reason this works at all on this production device is because oneplus screwed up and shipped a "debug profile" in production. this is a vendor signed elf (flashed to the dpdb partition or something like that) containing configuration to enable certain debugging features.
the one we already know about is crashdump mode, the op6 will crashdump when you trigger an XPU violation or some kinds of bus abort, then you can dump the entire RAM and you dont even need to authenticate (no firehose just run bkerlers edl.py)
this has come in handy for mainline development a few times heh
but it seems that this profile also enables EUD which is super nice, that said i didnt manage to get breakpoints working at all yet....
Oh hey, really cool article. Do you know if I'm correct in my attempt at enabling EUD? Also I was unsure how you determined that it was disabled in the OnePlus 6? I thought only EL3 can read qfuses in general?
And you mean the apdp partition, right? That's a weird ELF file, contains almost nothing obvious (test key sig? DEBUG mention) and doesn't seem to be any executable code which I guess makes sense, but I wonder why they made it an ELF. Is there any info on interpreting these profiles?
So that profile gives you the unauthenticated ramdump as well? Seems to be a common theme with OnePlus, messing up security features.
yeah EDL loaders for a bunch of production devices exist here [0] also more on various XDA Forum posts for stuff like unbricking guides. It is worth noting for people who don't
But reading QFUSES specifically requires an EL3 loader "edl qfp qfp.bin -> To dump qfprom fuses (only on EL3 loaders)" and I don't believe most devices programmers (especially as relatively new as the OnePlus 6) run under that privilege level.
So just to get this straight, Qualcomm has a piece of custom silicon, as a peripheral controlled by registers, that when enabled reroutes the ARMs USB pins through it (adding a USB hub in the middle), and on that hub it adds a SWD programmer and a serial port that connect back to the ARM core's IOs? Amazing!
This USB-based debug approach is becoming standard practice across SoC vendors (MediaTek's preloader, Apple's special cables) as it eliminates dedicated debug headers while maintaining controlled access through eFuse-gated authentication.
On some SoCs we could get to the ETM port and it was even parallel (what a nice waste of pins ;)), little down-clocking and you can get a full program trace out, i.e. with a Lauterbach. Can debug application together with the OS and all drivers down to single assembler instruction level.
Different topic, but I was crazy impressed to see Qualcomm's dedication on getting USB audio offload going, having the audio device forward data to the USB host controller, for it to send it to the USB audio device.
> This feature offers major power savings on embedded devices where a USB audio stream can continue to flow while the rest of the system is sleeping, something that devices running on battery power really care about.
How many shipping devices use USB for audio? I was under the impression that most phones/tablets/laptops support usb audio, but it is a niche/power user feature, and not the way the main device speakers are connected.
Do these same systems have bluetooth audio offload so the system can sleep for multiple minutes while a song is playing via bluetooth? (which seems to be a far more common usecase)
I admit to mystification myself at the work out in here for what feels like a pretty rare use case!
With 3.5mm stereo jacks gone, I do think there are some folks who use USB audio a lot now! But it also feels like it's gotta be a pretty small crowd! This sure feels like a long long way to go to give these select few a battery life boost! It'd be interesting to know how big the impact is, to at least weigh this long effort.
Like you, would love to know if there's anything cute folks have done for offloading Bluetooth! my gut says there are vastly vastly vastly more Bluetooth users.
USB-C headphones for phones have their own audio interfaces, yep. Headsets and webcams have usb audio interfaces too.
Also, I guess there are a good bunch of devices that have audio interfaces that, internally, might be connected to the USB bus.
But I'm also thinking about sound "processing" oriented hardware that could run Linux. Sound recorders, mixers, synths, smart speakers... that could be based on an SBC or an ARM or similar processor.
This is a much better experience than the previous Qualcomm debug experience, which was a hand-rolled set of read/write/execute primitives exposed over USB. It was hilariously undersecured, allowing a few of us to continually get root on various Qualcomm models.
In seriousness, these debug ports are seriously lacking in most mobile chipsets. MediaTek still has the old-style approach in many of their devices, requiring some incantations which expose serial over USB, but not in the way you think -- it's serial over USB pins!
I've done tonnes of work with mobile chipsets and security and this seems like they've finally started down the road to making this functionality accessible. Don't be surprised if you don't see this supported out of the box in most places, though. Most OEMs will certainly disable this once they've adapted their bootloaders to it. The big G doesn't like debuggability in end user devices.
Google exposes serial Serial over the SBU pins on all the Pixel devices
What are the effective implications of this?
It's just a UART; you can use the UART to debug the device in various ways.
On Pixel devices, the UART is not configured or brought up by default in locked production mode (as things should be), but by unlocking the device and then using `fastboot oem uart enable` you can flip the bits to turn it on. On early Pixel devices it was on the headphone jack and on newer ones it's on the SBU pins.
By default I think it's still configured as the kernel console in the kernel command line, so once it's enabled it will show the kernel debug output and present a TTY. But of course you can subsequently configure it to do whatever you'd want a UART for: kgdb for kernel-debugging, earlier stuff in the bootloader, and so on.
So, the implications are just: there's a convenient debugging interface available to you that turns on if you unlock the device and ask for it.
On Chromebook devices there's a more complicated and fancy debugging system where the SBU pins can be muxed to the security processor's USB host interface by presenting a debug cable called a SuzyQ, which presents a whole suite of debugging facilities. This used to be used quite frequently for unbricking purposes.
On the newer pixel phones (starting with the ones containing the titan chip) you can also mux the SBU pins to the security chip USB interface with "fastboot oem citadel suzyq".
And BTW, the SuzyQ cable is nothing more then two pull up resistors and a USB hub connected to the normal usb D+/D- pins on one port and the SBU pins on the 2nd port. Nothing fancy about it, people have even made their own (minus the hub) by soldering some wires and resistors to a usb-c breakout board. Google has also published the schematics for it:
https://www.chromium.org/chromium-os/ccd/951-00273-01_201806...
Been wanting to play with the citadel interface. Last time I looked at it there was 0 public information, but that was years ago. And if there still isn't, then it'll be a good excuse to RE
Also, the titan chip has been around since the Pixel 3 I believe, so "newer" might be a strong word :)
Most of those boards have a separate physical JTAG connector (at least in development kits, this article indicates JTAG over USB is disabled in production systems anyways so no difference there) which is what they are expecting you to use for low-level debugging. It only costs like 1,000 $ for a JTAG probe which is like 1 fully-burdened engineer-day of cost. Even fully featured probes enabling hardware trace and time-travel debugging only cost like 1 engineer-week.
> Most of those boards have a separate physical JTAG connector (at least in development kits, this article indicates JTAG over USB is disabled in production systems anyways so no difference there
There's generally an entire phase of prototyping where engineers will be using production boards but still need JTAG, which is why it's fused and why these kinds of features exist. It's a lot easier to have your lower-level software team (drivers/BSP, perf, etc.) sitting with production-ready units provisioned with engineering keys and debug enabled than to have them having to use some kind of case-off JTAG header setup, cost aside.
The probes cost enough to exceed individual purchasing limits at hardware companies, which means you need to go through the requisition process. That takes long enough that you have to plan ahead and you don't order more as your needs increase. Then everyone's fighting for the limited probes right before a ship date and they get jealously guarded like priceless jewels.
JTAG also isn't usually exposed through enclosures, so using the probe on a field unit might require destructive entry depending on the application.
Well the problem there is companies who are too stupid to invest in cheap tooling with massive ROI for their developers. A pretty constant problem in software development.
And I am not knocking JTAG over USB. It is certainly convenient and beneficial since you can enable it in production or deployed units. I was commenting on how the GP (and even article) was making it out to be missing capability. They just do not have the cheap tools that are the intended way to access that capability.
edit: The article even mentions how the "Qualcomm Landing Team at Linaro", which seems to be the team that works with pre-production hardware to get them working on launch day, has a development process where "debuggers have never been a staple of our work for all the typical reasons you'd expect (cost and complexity being the main ones)". That is literally the team that should have pre-production units in the lab which will have debug connectors and where JTAG probes should be par for the course, yet they are apparently hardly using them partly because of "cost".
The landing team's job is to "land" patches into upstream. They take Qualcomm code and spin straw into gold until it's eventually good enough to contribute to projects like the Linux kernel.
Having read a lot of Qualcomm code myself, I don't envy their job.
[dead]
There's something wrong if you're using Jtag on a field unit. In my experience, once a chip is up and running we implement higher level communication and debug methods.
To give an automotive example, a field unit may fail and be RMA'd to the tier 1 to improve their production line. They don't have access/documentation for the production software, and it wasn't developed to interact with their diagnostic tooling. They'll need to reliably reflash their own test software onto a board with potential hardware failures. JTAG is sometimes what they request to do that.
> this article indicates JTAG over USB is disabled in production systems anyways
Well, should be. I bet there've been screwups now and then...
[dead]
Nope, Many phases you have to debug on real product (no access or hard to access to jtag/swd header).
You say "undersecured", I say "allows people to own the things they bought".
Sadly these two things can be true at the same time. As a hacker I love devices which (maybe accidentally) expose a way for me to run arbitrary software on them. At the same time as a person carrying a computer in my pocket containing my entire life I'm not a fan if that allows anyone grabbing said computer to run arbitrary software on it.
The ideal here is the standard approach where a user has to explicitly take some sort of action which can't be found accidentally and done without understanding the implications to activate debug mode. On Android that's tapping a specific item in the about menu way more times than you might think would be necessary. On iOS it's hooking up to a computer, then authorising the connection on the phone. Either way you get both the ability to run arbitrary(ish) software without someone picking up your device when left on a train being able to do so.
Also there's "allow anything on the other side of this public 'charging port' own the things you bought"
You bring me back to the olden days of QXDM in my adolescence, when I used it to save myself from bricking my phone, and again when I had my first internship working on an MDM9x50 device
MediaTek still has the old-style approach in many of their devices, requiring some incantations which expose serial over USB, but not in the way you think -- it's serial over USB pins!
Wasn't that only in their old feature-phone (MT62xx) SoCs? All the smartphone ones AFAIK (at least since the MT657x days) use USB CDC in their BootROM and preloader.
Is there any information on those old debug protocols?
at work we adapted to "usb c debug accessory mode", which should be a standard everyone should follow.
It will be really interesting to see what production devices this is enabled on - It mentions the OnePlus 6 at least which has it fused out but is still accessible.
Edit: How are they reading the eFuses on a production OnePlus 6? Do they have a Qualcomm-signed EL3 EDL loader?
It seems to exist as qcom,msm-eud in the device tree of a (unfortunately production) SM4350 device I have along with an eud_enable_reg. Time to recompile the kernel with `/dev/mem`.
Well, no luck.
In the device tree I see (snippet):
but `devmem 0x162A000 4 0x1` causes the system to lock up and I see the following in ramoops: I'm not at all sure on the interpretation of this, but the reading at the efuse_addr (so I guess certain ones can be read from EL0?) is 0x0e000000 which has bits 25-27 set and QFPROM fuses seem to have a blown value of 1 according to Qualcomm docs, so it might be fused out?heya, author of the blog post here. the op6 happens to work but it's limited to debugging in el1, if you do manage to trap in el2 all the registers read 0
the reason this works at all on this production device is because oneplus screwed up and shipped a "debug profile" in production. this is a vendor signed elf (flashed to the dpdb partition or something like that) containing configuration to enable certain debugging features.
the one we already know about is crashdump mode, the op6 will crashdump when you trigger an XPU violation or some kinds of bus abort, then you can dump the entire RAM and you dont even need to authenticate (no firehose just run bkerlers edl.py)
this has come in handy for mainline development a few times heh
but it seems that this profile also enables EUD which is super nice, that said i didnt manage to get breakpoints working at all yet....
Oh hey, really cool article. Do you know if I'm correct in my attempt at enabling EUD? Also I was unsure how you determined that it was disabled in the OnePlus 6? I thought only EL3 can read qfuses in general?
And you mean the apdp partition, right? That's a weird ELF file, contains almost nothing obvious (test key sig? DEBUG mention) and doesn't seem to be any executable code which I guess makes sense, but I wonder why they made it an ELF. Is there any info on interpreting these profiles?
So that profile gives you the unauthenticated ramdump as well? Seems to be a common theme with OnePlus, messing up security features.
yeah EDL loaders for a bunch of production devices exist here [0] also more on various XDA Forum posts for stuff like unbricking guides. It is worth noting for people who don't
[0]: https://github.com/bkerler/Loaders
But reading QFUSES specifically requires an EL3 loader "edl qfp qfp.bin -> To dump qfprom fuses (only on EL3 loaders)" and I don't believe most devices programmers (especially as relatively new as the OnePlus 6) run under that privilege level.
A better resource is https://www.temblast.com/ref/loaders.htm
I know it works on the quectel rm520 module
So just to get this straight, Qualcomm has a piece of custom silicon, as a peripheral controlled by registers, that when enabled reroutes the ARMs USB pins through it (adding a USB hub in the middle), and on that hub it adds a SWD programmer and a serial port that connect back to the ARM core's IOs? Amazing!
This USB-based debug approach is becoming standard practice across SoC vendors (MediaTek's preloader, Apple's special cables) as it eliminates dedicated debug headers while maintaining controlled access through eFuse-gated authentication.
Just wait until you find out about Apple's magical USB shenanigans like the Chimp Cable https://www.theiphonewiki.com/wiki/Chimp_Cable
On some SoCs we could get to the ETM port and it was even parallel (what a nice waste of pins ;)), little down-clocking and you can get a full program trace out, i.e. with a Lauterbach. Can debug application together with the OS and all drivers down to single assembler instruction level.
Different topic, but I was crazy impressed to see Qualcomm's dedication on getting USB audio offload going, having the audio device forward data to the USB host controller, for it to send it to the USB audio device.
Feels like a weird thing to spend so much effort optimizing but neat as heck to see. https://www.phoronix.com/news/Linux-6.16-USB-Audio-Offload
> This feature offers major power savings on embedded devices where a USB audio stream can continue to flow while the rest of the system is sleeping, something that devices running on battery power really care about.
How many shipping devices use USB for audio? I was under the impression that most phones/tablets/laptops support usb audio, but it is a niche/power user feature, and not the way the main device speakers are connected.
Do these same systems have bluetooth audio offload so the system can sleep for multiple minutes while a song is playing via bluetooth? (which seems to be a far more common usecase)
I admit to mystification myself at the work out in here for what feels like a pretty rare use case!
With 3.5mm stereo jacks gone, I do think there are some folks who use USB audio a lot now! But it also feels like it's gotta be a pretty small crowd! This sure feels like a long long way to go to give these select few a battery life boost! It'd be interesting to know how big the impact is, to at least weigh this long effort.
Like you, would love to know if there's anything cute folks have done for offloading Bluetooth! my gut says there are vastly vastly vastly more Bluetooth users.
USB-C headphones for phones have their own audio interfaces, yep. Headsets and webcams have usb audio interfaces too.
Also, I guess there are a good bunch of devices that have audio interfaces that, internally, might be connected to the USB bus.
But I'm also thinking about sound "processing" oriented hardware that could run Linux. Sound recorders, mixers, synths, smart speakers... that could be based on an SBC or an ARM or similar processor.