DevOps for Embedded

Paul’s first post in a long while – I stumbled on this series of articles which are excellent if you happen to be developing and deploying embedded firmware.

Part 1:

Part 2:

Part 3:

Also if your job still has some hardware for which you’re somehow responsible (ie: you’re not 100% on someone else’s cloud service!) these are worth a read. While your employer might just be a consumer of the end product – shipped versions of firmware on hardware that you buy / already own, it is worth understanding what is involved in making that firmware and shipping updates.!

Securing Bare Metal Hardware at Scale: Matt King and Paul McMillan at BSides PDX 2018

I’ve been eagerly waiting for this video since I couldn’t make the talk in person. This is hands down one of the best talks I’ve seen in firmware security, including some great coverage of the issues with security, development and deployment that are applicable to all sorts of devices, not just servers.

The solution presented is (sadly) only workable for relatively large deployments of relatively homogeneous servers. But it IS a fairly complete solution, unlike the various partial firmware-blob-specific solutions like Secure Boot. Where’s the Secure Boot for my NVMe SSD firmware?

Well worth the time to watch, for anyone responsible for the security of any hardware, or software running on hardware. So really, everyone. I think it is helpful to understand the problem and this solution, even if you’re only responsible for say, your personal laptop and smartphone.

ATA/ATAPI Support in fwupd (LVFS)

A bit beyond my reading level for ATA/ATAPI and firmware updates on these devices, but from extensive conversations with our friends over at Progressive Technology (not an affiliate link!), the state of firmware security for storage devices is pretty bad. Following the historic firmware pattern – devices are often shipped with updataABLE firmware, meaning it can be supplanted by malware, but seldom/never receive firmware updates, nor does the manufacturer expect to supply firmware updates. Let alone via any sort of automated mechanism, like LVFS or Windows Update.

This sounds like progress, and progress is good.

Marvell Avastar WiFi Over The Air RCE

In addition to the description of the specific attack, this article outlines the entire process to evaluate the given WiFI SoC and go from “knowing nothing” to a working attack. It is one of the better written guides I’ve seen.

Originally released as a talk at ZeroNights 2018.


Firmware attacks uncommon?

Priceonomics has a great new summary of cyberattacks over time:

Why Security Breaches Just Keep Getting Bigger and More Expensive

I’d like to draw your attention to just one chart – which I’ll embed below (link may break):

One might look at this and say that firmware is virtually irrelevant. Certainly the baseline security advice is to start with the the things that are the most likely and impactful. So – if something is likely, but the impact is low, or if something is unlikely but the impact is high.. deprioritize it.

Note that the total here is far greater than 100%.. so I guess they’re saying that successful attacks often involve multiple of these categories. eg: Phishing that gets you to click on a bogus link is also a web-based attack? I guess?

But consider that:

  • For certain firmware is involved in EVERY attack in the “compromised / stolen devices” category, which represents 25%.
    • If a device is compromised in this way, even if you have full disk encryption, a firmware or side-channel attack may work.
    • Even if you get it back, the device cannot be trusted any longer
  • Malicious insider threats should also be considered to possibly involve firmware. What physical devices has that person touched? Ever?
  • Because the totals are > 100%, it is possible that firmware is involved in some of the other types of attacks as well – though I acknowledge that the percentage is probably low. Why attack firmware when you can execute a successful phishing attack?
  • General Malware – if the firmware can be updated while the OS is running (typically with admin/root privs), then malicious firmware can be installed by any regular malware that gets admin.

While the probability is STILL low, it is worth looking at numbers like this and reading between the lines a bit. Some types of protection, such as Secure Boot are worth enabling, or rather, not disabling!

Every cross fleet operation can balloon into a huge effort, such as deploying and ensuring anti-virus is up to date. So how about firmware updates? Some are now delivered with OS updates via Windows Update and or fwupd on Linux. If you can’t afford the massive amount of labor to consistently ensure that all your machines have up-to-date firmware, AT A MINIMUM, shoot an email to your purchasing department, and pressure your sales engineer that ALL future systems you purchase get firmware updates for ALL OS-updatable firmware – not just UEFI/BIOS, via the OS update mechanisms. Signed, forward-only, please.

AMI joins LVFS (fwupd)

Great! Welcome AMI!

AMI is the world’s largest BIOS firmware vendor, supplying firmware and tools to customers such as Asus, Clevo, Intel, AMD and many others. If you’ve heard of a vendor using Aptio for firmware updates, that means it’s from them. AMI has been testing the LVFS, UpdateCapsule and fwupd for a few months and is now fully compatible.

And a small teaser:

Also, expect another large vendor announcement soon. It’s the one quite a few people have been waiting for.

More router uPnP hacks, and

Another day, another uPnP mass hack:

Of course, I’ve disabled uPnP on my router, but why take the router’s word for it? Tenable released this handy Python utility a while back:

It lets you know what devices in multicast range have uPnP enabled, as well as enumerating the service XML description. Handy!

Surprisingly Turing Complete – How many computers are in your computer?

one might think that such universality as a system being smart enough to be able to run any program might be difficult or hard to achieve, but it turns out to be the opposite and it is difficult to write a useful system which does not immediately tip over into TC

And a nice reference to this old Tweet:

It’s amazing how many heterogeneous CPU cores were integrated in Intel Silvermont’s Moorefield SoC (ANN): x86, ARC, LMT, 8051, Audio DSP, each running own firmware and supporting JTAG interface

Self-encrypting deception: weaknesses in the encryption of solid state drives (SSDs)


Often, I get my news from Hacker News, but the discussion isn’t that great. In this case, I think the discussion is well worth reading!

Some choice quotes:

Here’s what I love the most about this: If you have a full-disk encrypted Windows laptop, which is fully powered down (or hybernated), and the laptop contains PHI, _and_ you lose the laptop, then you probably do _not_ have to report it as a data breach.(…)

But with this revelation, if you have an affected SSD, and you are running Windows, then losing such a laptop may now be a reportable event.



Litany of failures:* Firmware protection in drives is almost uniformly broken, so that they can get code execution (through JTAG or through hacked firmware images) routinely. This is bad, but shouldn’t be the end of the world, since in the drive encryption threat model you don’t want to have to depend on the firmware anyways. But:

* Two Crucial SSDs encrypt the drive with a key unrelated to the password; the password check is enforced only with an “if” statement in the firmware code, which can be overridden.

* Another Crucial SSD uses PBKDF2 to derive keys, but then has a master override key, which is blank by default. It also has a multi-volume encryption interface (Opal) with slots for volume keys, all of which are populated whether they’re in use or not, and if they’re not in use, they’re protected with an all-zeroes key that recovers the master key for the device.

* Two Samsung drives implement PBKDF2, but not in the default mode, which is “password is checked in an if statement, like the Crucial drive”. Also, the wear-leveling logic in one of the drives doesn’t zero out old copies of the master key, so that when you change your disk password (or set it for the first time), unprotected copies of the data encryption key are left in blocks on the device.

* The Samsung T3 portable drive uses the drive password in an “if” statement and is trivially unlocked through JTAG. Its successor, the T5, is no more cryptographically sound, but is simply harder to obtain code execution on.

People have strange ideas about what disk encryption is good for (in reality, full-disk encryption really only protects you from the situation where your powered-down, locked device is physically stolen from you and never recovered [if you get the drive back, you have to assume, at least from a cryptographic standpoint, that it’s now malicious.])

But the net result of this work is that Samsung and Crucial couldn’t even get that right. This paper is full of attacks where someone simply steals your drive and then unlocks it on their own. It’s bananas.

Side note: Look forward to a high-level overview of hard drive (spinning rust) firmware, along with some distinctions between manufacturers, model families and individual models as a joint effort with our local data recovery friends over at

BLEEDINGBIT: Bluetooth firmware vulnerabilities

Many WiFi access points have Bluetooth built into them now, and Bluetooth chips typically have firmware of their own.

In this case, a software stack called “BLE-STACK” that runs on a Cortex-M3 MCU.

So far, it seems to impact various Cisco, Meraki and Aruba access points.



Why are there Bluetooth chips in enterprise/commercial grade wifi APs? From Ars:

The BLE chips offer a variety of enhancements to traditional Wi-Fi APs. Retailers, for instance, can use them to monitor customer movements inside stores by monitoring the Bluetooth beacons sent by the customers’ phones. Hospitals can use BLE to keep track of Bluetooth-enabled medical equipment.


Remote management as a liability?

Another think-piece, follow up to my previous post on threat modeling firmware versus attacker time:

Many more, and smarter people than myself are talking about viewing data, particularly PII as a liability. and

I believe, on the corporate books, things can be both assets and liabilities, and the pollution metaphor is a great one in this regard – a factory is generally a (depreciating) physical asset. It would cost $millions to replace, and can produce $millions in capital gains over time. But, depending on what it is producing, it may also be a massive liability. eg: Union Carbide

While people may not generally be dying over data leaks, the scale of some of the leaks that have already happened is enormous, and the pending liability is much bigger.

In the world of firmware – hardware, really, we’ve got remote management systems. At this point it is very clear that everyone in the supply chain considers these to be major selling points – assets, if you will. For the most part, nobody seemed to consider that an end customer might want to disable Intel ME, Intel AMT, or AMD PSP. Some IPMI based systems actively dodge attempts to work around them – if you avoid plugging ethernet into the dedicated management NIC, the IPMI system will simply hop onto the next (onboard) NIC, and the only way to avoid it is to pay extra for an additional NIC! Because, obviously, they are SO USEFUL, why would you want to work around them?

But – as illustrated in this graphic, remote management systems, unlike most other types of firmware /hardware attack allow for continuous, cheap (low risk, low skill) attacks, and grant an extremely high level of power. I should perhaps have drawn the “power” column for Remote attacks as high as the “Hands on for days” attack implant requiring a lab. A lab implant is still less powerful than something baked in from the beginning – in the supply chain itself… as a feature!

Really the only technical difference between BMC and management features, is that (presumably) a supply chain implant is designed for some specific attacks (eg: data exfiltration) whereas remote management features were simply designed to… control the entire system remotely.


I assert that if a customer doesn’t want or need remote management features, they are solidly a liability, not an asset. As in the IPMI “jumping onto any active onboard NIC” example, active countermeasures need to be taken – money spent, by the end customer just to avoid the the risk involved. Such features should be disabled completely from the factory and only enabled when wanted or needed.

Even when they are needed by the end customer, they are both an asset and a liability, like that factory. Sure they help with managing systems – a great deal! But they also inherently add very desirable, and cheap/easy attack surface.

Automating UEFI Firmware Updates by GCHQ (UK version of NSA)

I think I need to start following the GCHQ blog.

We were surprised that many of the devices were running out-of-date firmware

Wow.. the GCHQ were surprised by this?!

Unfortunately, DellHP and Lenovo don’t currently update UEFI firmware through Windows Update. Instead, they all offer their own enterprise management tools for UEFI firmware. HP and Dell also publish catalogues of UEFI firmware updates for their platforms.

I think they’re referring to older hardware here. The current (6th) generation Lenovo X1 Carbon receives UEFI updates via Windows update. I was surprised to find it running what appears to be a DOS (CLI) utility to do it, but the update itself was delivered (and therefore cryptographically signed by Lenovo and Microsoft!) via Windows update. I believe this is also the case with the current generation Dell XPS.

So, as a result of this work, we are updating our Windows 10 EUD guidance to explain how you can automate your own UEFI firmware updates. Look out for the guidance later this month and let us know if you find our approach useful.

Use your purchasing power to help firmware security: LVFS (fwupd) and the UK GCHQ (UK version of NSA)

If you’re following and you have any interest in Linux, you should also follow the LVFS (fwupd) blog by Richard Hughes:

The National Cyber Security Centre (part of GCHQ, the UK version of the NSA) wrote a nice article on using the LVFS to influence procurement decisions. It’s probably also worth noting that the two biggest OEMs making consumer hardware also require all their ODMs to also support firmware updates on the LVFS. More and more mega-corporations also have “supports the LVFS” as a requirement for procurement.

The linked article:

We believe data such as this can help determine the firmware support lifetimes for a device or whether a device is still receiving regular firmware updates. It can also be used to aid in predicting typical support lifetimes for future devices. These may be important factors when considering whether your device should still be considered ‘in support’.

I assert that proactively purchasing devices that are well supported under LVFS (fwupd) today is a good move. It says a lot about your vendor as noted above. Even if you have no Linux at all in your infrastructure.. that could all change well within the lifespan of your hardware.

ASUS Z390 Motherboards Automatically Push Software into Windows

The ASUS UEFI firmware exposes an ACPI table to Windows 10, called “WPBT” or “Windows Platform Binary Table”. WPBT is used in the pre-built OEM industry, and is referred to as “the Vendor’s Rootkit.” Put simply, it is a script that makes Windows copy data from the BIOS to the System32 folder on the machine and execute it during Windows startup – every single time the system is booted. According to the Microsoft WPBT reference, which describes this feature as useful for “anti-theft software”, this binary is a “native, user-mode application that is executed by the Windows Session Manager during operating system initialization.”, which means “before all other programs, with administrative privileges”. This gives pretty much full control over everything, including protected folders and the registry.

Basic BMC and IPMI Management Security Practices

From Serve The Home

In light of multiple stories about BMC security breaches, we wanted to put a basic BMC and IPMI management security practices article together. This is likely a piece we will update over time. It is also one where there is an entire industry catering to management interface security, so this is only going to provide some bare minimum basics. If you are a new administrator, this should help avoid the top mistakes at a minimal incremental cost.

Editorial side-note – BMC, IPMI, ILo, Redfish, Intel AMT, Intel ME, AMD PSP – these are *computers* that control your computer. Sure, they run firmware, but in almost every case it is a full blown multi-tasking, typically multi-user networked computer. So.. their security, is networked computer security. It is really boring (credit to James Mickens). Encrypted network connections. Strong, non-default passwords.. for all users. 2FA if you can manage it!

Just because you think you might not have connected it to a network, or you think the “management network” to which you attached it is secure….

New Details on Google’s Titan M (2nd generation) Security Module

Much more detail than in the past, and a promise of open-source software release soon:

Titan M performs several security sensitive functions, including:

  • Storing and enforcing the locks and rollback counters used by Android Verified Boot.
  • Securely storing secrets and rate-limiting invalid attempts at retrieving them using the Weaver API.
  • Providing backing for the Android Strongbox Keymaster module, including Trusted User Presence and Protected Confirmation. Titan M has direct electrical connections to the Pixel’s side buttons, so a remote attacker can’t fake button presses. These features are available to third-party apps, such as FIDO U2F Authentication.
  • Enforcing factory-reset policies, so that lost or stolen phones can only be restored to operation by the authorized owner.
  • Ensuring that even Google can’t unlock a phone or install firmware updates without the owner’s cooperation with Insider Attack Resistance.

Reaper: Characterization and Fast Detection of Card Skimmers

From USENIX Security in August:


Payment card fraud results in billions of dollars in losses annually. Adversaries increasingly acquire card data using skimmers, which are attached to legitimate payment devices including point of sale terminals, gas pumps, and ATMs. Detecting such devices can be difficult, and while many experts offer advice in doing so, there exists no large-scale characterization of skimmer technology to support such defenses. In this paper, we perform the first such study based on skimmers recovered by the NYPD’s Financial Crimes Task Force over a 16 month period. After systematizing these devices, we develop the Skim Reaper, a detector which takes advantage of the physical properties and constraints necessary for many skimmers to steal card data. Our analysis shows the Skim Reaper effectively detects 100% of devices supplied by the NYPD. In so doing, we provide the first robust and portable mechanism for detecting card skimmers.

This is some excellent stuff! Some immediate questions come to mind:

  • When will the Payment Card Industry (PCI) require this sort of relatively cheap, fast audit of every card interface? ATMs at a minimum require someone to regularly physically attend to them to load/unload cash.
  • Why did it take so long, and so many attacks for someone to come up with this?
  • Who is working on similar technology for physical interfaces? (admittedly, much less common than payment card attacks!)
    • USB
    • PCIe/Thunderbolt/USB-C
    • Ethernet

Electric Scooter Hacking

Old news, summarized nicely:

Most of these hacks have nothing to do with firmware at all – they are basic physical access analog hacks… or in some variants, simple theft and vandalism.

The 3 primary components of these (and any e-bike, or electric car!) are:

  • Battery – with it’s own resale / reuse value
  • Motor – with it’s own resale / reuse value
  • Computer – with it’s own resale / reuse value, AND virtually impossible to prevent it from being completely hacked “in place.”

Most of the current activity on the streets of San Francisco seems to be either simply tearing apart the scooter to reuse/resell the current parts, or on basic hotwiring (routing entirely around the computer) to make the scooter function as – just an electric scooter.

Some response from the scooter providers has been firmware updates to disable some things, such as suspending billing by picking up a scooter and relocating it.

Much more interesting, and relevant to firmware security will be to watch the cat and mouse game play out with regards to the firmware on the computer.

Manufacturers of highly computerized, shared use vehicles beware – your threat model is that of a “Hands-on for days” (SUPER Evil Maid) attack, scaling up to unlimited physical access. Much closer to trying to protect an entire manufacturing supply chain than it is to making sure TSA isn’t getting too handsy with your laptop. Worse, if you consider that Attackers can:

  • Move your vehicle (even if you were to lock the axles!) to a full laboratory setting
  • Rewrite any storage, including pulling and reflashing chips
  • Create scalable, automated, fast, reusable attacks, developed in a laboratory

Hardware & Firmware Attacks Versus Time – Threat Modeling

During the Spring 2018 UEFI Forum Plugfest, Brent Holtsclaw and John Loucaides of Intel presented an “Introduction to Platform Security.“(slides pdf).  There is also a Youtube video:

I suggest this is required material for anyone interested in this topic!

Slide #8 “Classes of Attacker” with an illustration of attacker power relative to level of access: physical but limited, unlimited physical, and privileged versus unprivileged malware. It is elegant and simple.

Here’s my chart, inspired by the above, which is neither. Feedback welcome!


This illustrates graphically what many people have been saying in words. I’ll need to explain it in words anyway. Perfect for a talk! Or a long blog post.

  1. Scale is:
    • fairly precise for time
    • fairly precise for attacker power, not quantified in this illustration
    • only relative for attacker cost / effort / sophistication
  2. Attacker power and time are illustrated as bars, to show the discrete nature of each type of attack.
    • Time: An attacker who believes they are going to have hands on your hardware for seconds cannot safely assume that they might have minutes or hours.
    • Power: An attacker who has the ability to reboot the machine via BMC compromise does not necessarily have the ability to read/write data from the hard drive, as in SATA controller firmware compromise.
  3. OBSERVATION: Time, attacker power and cost are not always directly correlated. The most dramatic example is remote. By design, remote management systems give the fullest control of their host computer possible, and are full Internet connected computers running an operating system in their own right. A classic “weakest link” – if your BMC is not equally secure to the primary system, then it defines the security of the entire system. Attacking these remotely is just as cheap and efficient as attacking the host system remotely.
  4. Remote, unprivileged malware and privileged malware share similarity in that:
    • they should only be able to compromise unsecured or misconfigured firmware. Unfortunately that is most of the firmware, most of the time.
    • ideally they should be verifiable and repairable with software only.

Detail for each type of hardware/firmware attack, as a sort of taxonomy:


Attacks a management computer built into, or added on to the primary computer. Examples include: IPMI, Redfish, BMCs, Intel AMT and AMD PSP, HP iLo, Dell DRAC, etc. In every way possible, this is the same as simply attacking any network-attached computer. A baseline assumption by manufacturers is often that these specially privileged computers will be attacked to a specially secured network.

Unprivileged Malware

Any standard computer virus / malware can modify at least some of the system firmware, by executing as an unprivileged executable. Special skill and effort is typically required to do this over the creation of a normal virus intended to interact with eg: data files on the filesystem, owned by the unprivileged user. And it is significantly less portable as it often must target a specific SuperIO chip or particular make/model/revision of SATA disk.

Privileged malware

Escalating privilege is understood to be fairly easy on most systems.The primary gain with this attack is using firmware update mechanisms for firmware that can be updated through software (as opposed to requiring direct interaction with the chip in a programmer).  UEFI provides the framework for requiring:

  • Updates cryptographically signed by the manufacturer
  • Forward-only updates, to ensure an attacker cannot revert to an earlier, vulnerable firmware version

But this is not uniformly, or well implemented in the UEFI space, and is not present for most other types of platform firmware.

Hands-on For Seconds

Creates the opportunity for a known-fast side-channel attack, such as the CIA’s “Sonic Screwdriver” exploiting PCIe/Thunderbolt DMA. Or for a stealthy hardware implant, such as the NSA’s Firewalk ethernet jack shim, or an ATM skimmer. Example: Airport security (TSA in the USA) handles a laptop passing through security.

Hands-on For Minutes

Expands the opportunity to slower side-channel attacks, and much more stealthy implants. There may be sufficient time to open the case, and implant something internally, though definitely not for soldering. There is probably an (unillustrated) distinction between hands-on out of sight of the hardware owner, versus in-sight. Example: Airport security (TSA in USA) selects an individual for “enhanced” screening, and separates them from their laptop for some minutes.

Hands-on For Hours

The classic Evil Maid attack, for hardware left unattended in an assumed-safe location such as a hotel room, at home, or an office, or separated from the owner such as at border control for a “suspicious person.” Hours ensures that any case can be opened, and (de)soldering can happen, with the limitation of portable tools only.

Hands-on For Days

An interdiction attack, such as the NSA TAO implants on Cisco routers. More expense and secrecy risk is incurred due to the facilities and number of people involved, but much finer grained (de)soldering is possible.

Unlimited Time (Supply Chain)

Fully invisible implants sandwiched between layers of circuit board are now possible.