Tracing flash reads (and writes) during boot [on OpenPOWER systems]

On OpenPOWER POWER9 systems, we typically talk to the flash chips that hold firmware for the host (i.e. the POWER9) processor through a daemon running on the BMC (aka service processor) rather than directly. We have host firmware map “windows” on the LPC bus to parts of the flash chip. This flash chip can in fact be a virtual one, constructed dynamically from files on the BMC. Since we’re mapping windows into this flash address space, we have some knowledge as to what IO the host is doing to/from the pnor. We can use this to output data in the blktrace format and feed into existing tools used to analyze IO patterns. So, with a bit of learning of the data format and learning how to drive the various tools, I was ready to patch the BMC daemon (mboxbridge) to get some data out.[…]

OpenPOWER firmware development

Stewart Smith of IBM has a new blog post that gives an introduction to OpenPOWER firmware dev.

A (simplified) view of OpenPOWER Firmware Development
I’ve been working on trying to better document the whole flow of code that goes into a build of firmware for an OpenPOWER machine. This is partially to help those not familiar with it get a better grasp of the sheer scale of what goes into that 32/64MB of flash. I also wanted to convey the components that we heavily re-used from other Open Source projects, what parts are still “IBM internal” (as they relate to the open source workflow) and which bits are primarily contributed to by IBMers (at least at this point in time).[…]


OpenPOWER firmware updates using ZMODEM

Stewart Smith of IBM has a new blog post about adding ZMODEM support to OpenPOWER firmware.

From checkin: This enables the use of rz/sz to send/receive files using ZMODEM. This enables error detection and correction when using the console to transfer files to/from the host.

From blog:

ZMODEM saves the day! Or, why my firmware for a machine with a CPU from 2017 contains a serial file transfer protocol from the 1980s

Recently, I added the package lrzsz to op-build in this commit. This package provides the rz and sz commands – for receive zmodem and send zmodem respectively. For those who don’t know, op-build builds a firmware image for OpenPOWER machines, and adding this package adds the commands to the petitboot shell (the busybox environment you get when you “exit to shell” from the boot menu).[…]


What’s next, a UEFI runtime service for Kermit, using CKermit? UEFI NNTP Boot, using signed images on alt.binaries.firmware.*? 🙂

Porting UEFI to Apple PowerPC…

Porting UEFI to a new architecture:
So it turns out that blogging about something after the fact is pretty tough. I really wanted to blog about my PoC port of UEFI to the OpenPower ecosystem, but it’s incredibly difficult to go back and try to systematize something that’s been a few years back. So let’s try this again. This time, our victim will be a G4 12″ PowerBook6,8 with a 7447A. That’s a 32-bit PowerPC. Now, I’ll go in small steps and document everything. For added fun, we’ll begin porting on the target itself, at least until that gets too tedious. Also, I’ve a few OldWorld machines, a spare G4 12″ for parts and a G5, so hopefully this odyssey won’t be interrupted by old and failing hardware ;-). Keep in mind that each part is checked in along with the source code, so look at the entire commit. Each blog post will focus on the most important details.[…]



IBM OpenPower secure and trusted boot, Part 2

OpenPOWER secure and trusted boot, Part 2
Protecting system firmware with OpenPOWER secure boot
Making your system safe against boot code cyberattacks
Dave Heller and Nageswara Sastry
Published on June 05, 2017

This content is part 2 of 2 in the series: OpenPOWER secure and trusted boot. IBM® OpenPOWER servers offer two essential security features, trusted boot and secure boot, to help ensure the integrity of your server and safeguard against a boot code cyberattack. Trusted boot works by creating secure recordings, or measurements, of executable code as the system boots. Using a process known as remote attestation, you can retrieve these measurements securely and use them to verify the integrity of your firmware or target operating system (OS). Secure boot helps ensure the integrity of your OS and firmware as well. But rather than taking measurements for later examination, secure boot performs the validation in place, during boot, and will halt the boot process if the validation fails. These two features are complementary and work together to provide comprehensive protection of platform boot code. This article explores the secure boot method, with particular focus on protection of system firmware.[…]

Part 1 is from Feburary:


IBM Monacle and PowerVM firmware updates

[…]Under the Hood of Power Firmware Maintenance

The Service Processor of the server is running an embedded operating system with complex power firmware applications running on it; one of which is an application responsible for handling code updates. […]!/wiki/Power%20Systems/page/Monocle%20Patch%20Management

Talos FlexVer technology -vs- Evil Maids

Talos has a new post on their use of FPGAs on their OpenPower-based workstation.


OpenPOWER code added to FWTS

Deb McLemore of IBM has submitted multiple updates to FWTS, the FirmWare Test Suite, adding a lot more support for OpenPOWER OPAL firmware.

opal: pci_info: Add OPAL PCI Info validation
opal: mem_info: Add OPAL MEM Info validation
opal: cpu_info: Add OPAL CPU Info validation
devicetree: dt_sysinfo: Add OPAL firmware version checks
olog: olog.json: Update OPAL skiboot errors to check on olog scan

There is a lot of useful diagnostic information in this code, example:
“You are running in manufacturing mode. This mode should only be enabled in a factory during manufacturing.”

More information:

Stewart on compiling your IBM S822LC’s firmware

Stewart Smith of IBM has a new blog post on how to compile your own firmware for the OpenPOWER-based IBM S822LC:

[…] IBM (my employer) recently announced  the new S822LC for HPC POWER8+NVLINK NVIDIA P100 GPUs server. The “For HPC” suffix on the model number is significant, as the S822LC is a different machine. What makes the “for HPC” variant different is that the POWER8 CPU has (in addition to PCIe), logic for NVLink to connect the CPU to NVIDIA GPUs.[…]

Talos Secure Workstation: coreboot + POWER8

New potential product on CrowdSupply with a NICE set of features (…and I wonder how secure it will be):

* Blob-free operation
* Fully libre (open-source) IBM OPAL primary firmware w/ PetitBoot interface
* Fully libre (open-source) OpenBMC secondary (IPMI / OoBM) firmware
* NO signing keys preventing firmware modification

coreboot 4.4 released!

coreboot version 4.4 has been released, which includes 850 commits by 90 authors added 59000 lines to the codebase since their last quarterly release. This release includes OpenPOWER support and experimental U-Boot payload support. It also has an Intel ME tool which is new (at least to me). There are many other changes, excerpting their announcement:

Significant changes:
* Build system (30 commits) – Add postcar stage, ‘timeless’ builds, extend site-local, test toolchain by version string, update dependencies, catch ACPI errors, l add additional macros.
* Toolchain updates (40+ patches) – Update IASL to v20160318 , LLVM to v3.7.1, add GNU make, add nds32le GCC compiler
* Lint tools (30 patches) – Update existing lint utilities, add lint tests for executable bit, make sure site-local isn’t committed, add test to break all lint tests.
* Payloads (60 commits) – Fixes for libpayload, coreinfo and nvramcui, add new payloads, see below.
* Maintainers file – (8 patches) – continue adding maintainers for various areas.
* Documentation for adding Intel FSP-based platforms (20 commits)

New architectures:
* power8
* qemu-power8

New payloads:
* depthcharge: For ChromeOS verified boot
* iPXE: For network booting
* Memtest86+: Updated with fixes for correctly testing coreboot with payloads
* U-Boot (Experimental): Alternate payload for booting an OS

New/updated mainboards:
* asus/kcma-d8
* emulation/qemu-power8
* google/auron_paine
* google/gru
* intel/amenia
* intel/apollolake_rvp
* intel/camelbackmountain_fsp
* intel/galileo
* lenovo/t420
* asus/kgpe-d16
* google/oak
* google/chell
* intel/kunimitsu

New/updated socs:
* intel/apollolake
* intel/fsp_broadwell_de
* intel/quark
* marvell/armada38x
* rockchip/rk3399
* cpuamd/mct_ddr3
* drivers/intel/fsp2_0
* northbridge/intel/sandybridge/raminit
* soc/intel/apollolake
* soc/intel/fsp_baytrail
* soc/intel/skylake
* soc/mediatek/mt8173

New vendorcode directories:
* siemens

New/updated submodules:
* chromeec
* 3rdparty/arm-trusted-firmware (329 commits)
* 3rdparty/vboot (28 commits)
* util/nvidia/cbootimage (13 commits)

New/updated Utilities:
* archive – Concatenates files into a single blob with an indexed header
* chromeos – Download and extract blobs from a ChromeOS image
* futility – vboot Firmware utility
* intelmetool – Shows information about the Intel ME on a platform.
* marvell/doimage_mv – No usage notes
* post – Simple utility to test post cards

Full announcement:

OpenPOWER firmware update from Stewart

Stewart Smith has a new blog post about OpenPOWER, focusing on firmware development community changes, including comments on OpenBMC and other projects. As well, apparently now non-IBM developers can now contribute to OpenPOWER firmware, as someone from has recently done, which sounds like an improvement.

Interview with Andrei Warkentin, OpenPOWER UEFI porter

There are two ports of UEFI to the OpenPOWER architecture. Andrei E. Warkentin is a developer of one of them. I did an email interview with Andrei. I’ll hopefully have another post in the future with Benjamin, author of the second port. The answers below are from Andrei:

Q: Andrei, how goes your port?

A: So UEFI on OpenPOWER is my idea of a nice relaxing side project to hack on in the late hours with my family asleep, and I haven’t had much time in the last few months to work on it due to vacations, work, … 🙂 But the current state is being able to boot to a UEFI shell with just the OPAL console being supported.  A FAT filesystem image can be passed to qemu via the ‘initrd’ option to be accessible from UEFI. This means that my port is good enough to start writing UEFI applications, for example, or to start thinking about booting PPC64LE Linux, but there is no I/O support for PCI devices (no disk, no networking, no video console, etc).

Q: Is there any consideration of merging two projects into a single one?

A: Probably not. I’m not affiliated with Benjamin’s work, IBM or OzLabs and I am making a point of not looking over at Ben’s work as I address various parts of the UEFI port, because it’s fun to figure out things like this on your own and that’s why I’m working on OpenPOWER UEFI in the first place. I think it will be interesting to see the difference in approaches.

Q: Does OpenPOWER have any involvement in these ports, and official UEFI support, or are these strictly unofficial?

A: Actually one of the reasons I embarked on porting UEFI to PPC64LE is because I couldn’t find any evidence that a project like it was already being done by someone else :-). Power CPUs are not an officially supported platform for UEFI, but proof-of-concept work like this is the first step towards putting together an official specification, provided there is industry interest of course! Having two independent ports goes a long way towards identifying the real minimum requirements and any implementation assumptions to filter out. I think UEFI on OpenPower is interesting because it reduces the barrier to entry for hardware vendors to potentially start making more OpenPower gear. They will be able to use existing knowledge, skills, tooling, etc.

Q: Is there any hardware-level debugger, like an Intel ICE/ITP, for tracing at the instruction level, through OpenPOWER firmware? Perhaps a GUI dev toolchain like like ARM DS-5 or Intel ISS?

A: While I’m not aware of any hardware debuggers, I highly recommend Ben’s OpenPower QEMU ‘powernv’ target. This is what I have been using for my Power8 hacks and UEFI development, and while there’s always the risk of writing code that may not be 100% architecturally correct when using an emulator, it feels 1000x faster than the official IBM simulator, and you can use GDB for an ICE-like experience. For UEFI I have a Py script to facilitate source debugging (See

Q: How does your UEFI implementation interact — replace or chain after or otherwise integrate with — OpenPOWER’s firmware, OPAL (Open Power Abstraction Layer) firmware, hostboot, petitboot, and other related components I’m not aware of?

A: My UEFI implementation is just another ‘skiboot’ payload, to be loaded instead of ‘petitboot’, and fully meant to leverage OPAL. OPAL runtime services are used today for console access, and would be used for NVRAM, RTC, simple I/O, PCI, power state transitions, etc. The goal is a fully generic UEFI implementation that is not specific to any particular OpenPower platform. This is achieved by using OPAL and the skiboot-produced FDT to abstract platform binding and configuration. This way if a particular work flow requires UEFI, then it’s just another loader stage, similar to the OFW->ARC shim used for the old Windows NT PReP port. The goal is that OPAL services that don’t have corresponding UEFI RT matches would be provided as extra UEFI RT services, to avoid having the OS to deal with supporting two completely different runtime services environments.

Stepping back a bit, conceptually, I don’t think UEFI ought to replace the existing OpenPOWER booting strategy, but simply complement to give more power to the user, to the IHV, the OEM and the OS vendor. My 10 years of experience with both architecture- and platform ports of UEFI, both in writing and in dealing with buggy implementations and drivers has convinced me that it was a mistake to use this highly complex mini-OS, with a non-trivial driver and firmware<->OS interaction model as the first-stage system IPL to be flashed into ROM. This is not meant as a slight against UEFI or Tiano itself, as I think it would be equally a mistake to leverage any other OS or kernel here: it’s just too big. There is way too much code involved – too many moving parts. A lot of these moving parts are part of ‘generic’ code distributed as Tiano where no one looks until something catastrophic happens. The expectation that IHV and hardware engineers are going to write functionally correct drivers is flawed – not everyone is a kernel engineer! For example, UEFI implementers are expected to come up with a proper platform NVRAM driver and the ‘Bds’ driver, the later being a key part that basically unifies all of the end user’s interaction and boot selection activities. Writing NVRAM drivers and ‘Bds’ correctly is hard because introducing subtle bugs is easy and few seem to consider the ‘unexpected’ situations. We’ve already seen plenty industry bugs where setting or clearing the wrong NV variable from the OS turns computers into unusable bricks. I’ve seen a lot of bugs relating to folks not really understanding how runtime services really work, or what sort of things are allowed to be done during UEFI quiesce callbacks. But the conceptual problem here is an environment that pushes all the complexity of OS development onto firmware integrators.

Q: As I understand it, OpenPOWER has some form of Flattened Device Tree (FDT) support? Will that be exposed via your UEFI implementation?

A: The device tree is consumed by the UEFI implementation to figure out how the machine is configured, how to set itself and its own drivers and I/O support up. The device tree would be exposed to a booting Linux kernel, similar to how it is exposed on AArch32 and AArch64 UEFI implementations.

Q: FDT aside, what/how is your port dealing with ACPI, if at all?

A: One of the things I wanted to scope out specifically is presenting OpenPower as a ‘reduced-hardware’ ACPI environment, similar to how the AArch64 ecosystem is doing. ‘Reduced hardware’ means that the requirements around some legacy x86 fixed hardware are dropped, making it possible to support ACPI for platform discovery, configuration and power management across a wide range of very different environments, ranging from embedded SoCs to powerful servers. From a birds eye view, ACPI and device tree are quite similar – both describe hardware as a hierarchical tree of devices with certain identifiers and properties. ACPI differs from DT in that it removes the need to describe certain platform knowledge (and consume it in a driver) and hides it instead in device methods written in this interpreted byte code called AML. It is a heavier system, in many ways significantly less elegant than device tree, but it allows for potentially more homogenous-looking hardware, and it is a de facto standard across server and motherboard manufacturers, so I think having ACPI support for OpenPower would reduce the barrier to entry for hardware vendors to potentially start making more compatible OpenPower gear. I have not started thinking about ACPI support just yet, and this would of course involve changes to the ‘powernv’ support in the Linux kernel.

Q: Besides Github project, do either of you have any upcoming conference presentions or research papers on your projects that we should be looking for?

A: Not that I am aware of. Certainly, if there is interest, I would be happy to present my work at any applicable conferences.

Thanks for the answers, Andrei!

BTW, originally I thought that both ports were from IBM developers, but that’s not the case, Andrei is not from IBM. My previous OpenPOWER blog post is incorrect.

Stewart Smith on OpenPOWER firmware

Stewart Smith of IBM posted a new blog entry, announcing availability of the video of his recent OpenPOWER firmware talk at LinuxConf.AU:

In mid 2014, IBM released the first POWER8 based systems with the new Free and Open Source OPAL firmware. Since then, several members of the OpenPower foundation have produced (or are currently producing) machines based on the POWER8 processor with the OPAL firmware. This talk will cover the POWER8 chip with an open source firmware stack and how it all fits together. We will walk through all of the firmware components and what they do, including the boot sequence from power being applied up to booting an operating system. We’ll delve into:
– the time before you have RAM
– the time before you have thermal management
– the time before you have PCI
– runtime processor diagnostics and repair
– the bootloader (and extending it)
– building and flashing your own firmware
– using a simulator instead
– the firmware interface that Linux talks to
– device tree and OPAL calls
– fun in firmware QA and testing

Anandtech OpenPOWER update

From Vincent Zimmer’s twitter feed, Johan De Gelas has a story on Anandtech, discussing IBM’s OpenPOWER8.

So what caused us to investigate the IBM POWER8 as a viable alternative to the mass market Xeon E5s and not simply the high-end quad (and higher) socket Xeon E7 parts? A lot. IBM sold its x86 server division to Lenovo. So there is only one true server processor left at IBM: the POWER family. But more importantly, the OpenPOWER fondation has a lot of momentum since its birth in 2013. IBM and the OpenPOWER Foundation Partners like Google, NVIDIA, and Mellanox are all committed to innovating around the POWER processor-based systems from the chip level up through the whole platform. The foundation has delivered some tangible results:

 * Open Firmware which includes both the firmware to boot the hardware (similar to the BIOS) …
 * … as OPAL (OpenPOWER Abstraction Layer) to boot and launch a hypervisor kernel.
 * OpenBMC
 * Cheaper and available to third parties (!) POWER8 chips
 * CAPI over PCIe, to make it easier to link the POWER8 to GPUs (and other PCIe cards)
 * And much more third party hardware support (Mellanox IB etc.)
 * A much large software ecosystem (see further)

The impact of opening up firmware under the Apache v2 license and BMC (IBM calls it “field processor”) code should not be underestimated. The big hyperscale companies – Google, Amazon, Microsoft, Facebook, Rackspace – want as much control over their software stack as they can.

Full article:

OpenPOWER port comments, and EFI creation story

A few weeks ago, I noticed a new Github project where someone at IBM had ported UEFI to OpenPOWER. Later that day, someone *ELSE* at IBM had *ALSO* ported UEFI to OpenPOWER — two ports! And they found out about each other because of my blog, too funny! Anyway, they’re actively conversing on the Comments thread on the blog:

In addition to OpenPOWER issues, the Comments section has MULTIPLE people posting comments, which is a first for this blog! There’s some good background on history of EFI, including a pointer to 2005-era message that includes the ‘creation saga’ of EFI! In the past, Apple used OpenFirmware on PowerPC-based Macs, and PC vendors used BIOS on x86-based PCs. When Windows NT started support for non-x86 systems, ARC was used for the RISC vendors. Then Intel Itanium needed a solution. They nearly used ARC or OpenFirmware, before going down the route for EFI for Itanium, which later grew into UEFI and is now on Apple and PC systems as well. Here’s an excerpt with Andrew Fish talking about how EFI was created:

“At that time, two firmware solutions were put on the table as
replacements for BIOS architectures for the Itanium: Alpha Reference
Console (ARC) and Open Firmware. It turned out that nobody really
owned the inellectual property to ARC, and in any case, it did not
have enough extensible properties to make it practical for a
horizontal industry. At this point, Open Firmware became the
frontrunner as Apple and Sun both used it. However, Open Firmware was
not without its own technical challenges. The PC had started down a
path of using the Advanced Configuration and Power Interface (ACPI) as
its runtime namespace to describe the platform to the operating
system. As I liked to say at the time, the only thing worse than one
namespace is keeping two namespaces in sync. The other problem was the
lack of third party support for Open Firmware. We invited the
FirmWorks guys to come visit us at Dupont (WA), and we had a great
talk. Given we had just gone through an exercise of inventing a
firmware base from scratch, I think we were uniquely qualified to
appreciate what Open Firmware had been able to achieve. Unfortunately,
it became clear that the infrastructure to support a transition to
Open Firmware did not exist. Given the namespace issue with Open
Firmware and the lack of industry enabling infrastructure, we decided
to go on and make EFI a reality.”

Full post:

I thought ARC stood for “Advanced RISC Consortium”, but I can’t argue with Andrew, maybe it originally stood for “Alpha Reference Console”.  (During these years, I was doing “ring0′ work, not firmware, and my favorite box at the time was a DEC Alpha AXP with ARC-based firmware.) ARC is very similar to early EFI: a single firmware image with a FAT partition where additional firmware images are stored, using ‘variables’ to specify the boot loader to invoke. ARC died off, like all the RISC-based Windows NT platforms, a shame, IMO. The ARC spec is still archived by the NetBSD project, who still supports ARC:

I wonder how life would be today if Intel would have chosen ARC or OpenFirmware?

Second port of Tianocore to OpenPOWER

Check out the Comments section of the recent post on Tianocore being ported to OpenPOWER:

So there are TWO ports of Tianocore to OpenPOWER, and a few days ago I knew about neither of them.

I wonder if OpenPOWER will officially adopt UEFI as a firmware option for their platform. I’ve heard indirect bragging from IBM that their systems can be 100% fully open source firmware, no blobs.

Tianocore for OpenPOWER

Andrei Warkentin has ported UEFI to OpenPOWER! Excerpt from readme:

TianoCore UEFI for OPAL/PowerNV (PPC64/PowerPC64 Little-Endian)

This is “UEFI” on top of OPAL firmware. “UEFI” because the specification doesn’t include PowerPC yet (ever?). At some point this experiment will implement reduced-hardware “ACPI” support, mapping the OPAL-based FDT into “ACPI” structures. “ACPI” because it’s also not specced out for PowerPC. It’s getting prototyped on top of QEMU and Skiboot (OPAL firmware).

Why? It’s thought experiment gone too far. In short, there’s IMO value in presenting a common firmware environment shared with other servers (X64, AARCH64). UEFI+ACPI keep the OEMs and IHVs in their comfort zone, and reduce pointless platform boot and configuration variance across different architectures. It also allows plug-in cards to work (assuming EBC ROMs). Petitboot is a nice idea in theory, but in practice it’s probably more suitable to embedded environments rather than whitebox servers that can compete with Intel boxes. UEFI gets us a bootloader environment and device drivers for I/O and booting via storage and networking. OPAL is the abstraction layer for the machine.