PANDA-VM’s LAVA 2.0 released

LAVA (Large-scale Automated Vulnerability Addition) 2.0 is out. This is the PANDA (Platform for Architecture-Neutral Dynamic Analysis) VM LAVA, not the Linaro CI LAVA…

3mdeb: Minnowboard Turbot remote firmware flashing with RTE (Remote Testing Environment)

Minnowboard Turbot remote firmware flashing with RTE (Remote Testing Environment)
April 5, 2018
Arek Cichocki

Work related to a hardware carries some restrictions which don’t occur when working only with a software. One of them is a limited number of devices. This one may cause a problem with a accessibility to the platform. The limited number of users could slow development and testing. What is more work with a hardware requires a minimal knowledge of the theory of circuits and signals to eliminate platform damage by a user. Hardware can be expensive too. Remote Testing Environment project was made to resolve mentioned problems. […]

LAVA 2018.2 released

Neil Williams of Linaro announced the 2018.2 release of LAVA. Here’s 3 changes excerpted from the announcement below:

* Bootloader support changes: Better detect errors in the bootloader – this adds support to distinguish between a bootloader failure and a kernel failure to detect problems when the bootloader tries to start the kernel. This has an important effect on how some test jobs run – see Quiet Kernels below. The parallel change (7a2b3a68 Change the flow of bootloader commands so they are executed individually) supports detecting failures to download artifacts as distinct from failures to execute once downloaded.

* Bootloader action optimisations: To support the better error detection, several bootloader actions have been optimised. This means that different actions may be used, changing the names you may be using for timeouts. e.g.
The timeout name was u-boot-interrupt – now it is bootloader-interrupt. The UI shows you which actions have been assembled into the pipeline.
e.g.: bootloader-interrupt: Wait for prompt Hit any key to stop autoboot (timeout 00:02:00)

* Quiet Kernels: The bootloader support changes wait for an indication that the bootloader has completed and that the kernel has started, by watching for a kernel_start_message. If your kernel is configured to be quiet, then each test job using that kernel *must* clear the kernel_start_message:
   kernel_start_message: ”
In most cases, the test job should not use ‘quiet’ as this hides important debugging information from the kernel boot process.

LAVA 2017.11 released

Neil Williams of Linaro announced the 2017.11 release of LAVA.

2017.11 is the second release on the roadmap to the removal of V1. This is the single largest change ever made to the LAVA packages.
* All dashboard URLs are permanently disabled in lava-server.
* All devices which were not enabled for V2 are now hidden and unusable.
* All V1 code has been removed from lava-dispatcher.
* All V1 documentation has been removed from lava-server-doc.

2017.11 also includes large changes to the packaging of both lava-server and lava-dispatcher. The prompts formerly used to configure a V1 remote worker have been removed.


DebConf, Debian’s annual conference is happening in Cape Town, South Africa. Even if you aren’t in Cape Town this week, the DebConf event team is very good at providing postconference video archives, look for them to be available shortly.

Amongst the many interesting presentations, here’s a few firmware-related presentations to look forward to:

Secure Boot BOF
Ben Hutchings
Secure Boot is a UEFI feature that prevents unsigned boot code from being loaded. Assuming the bootloader checks the signature on the kernel, and the kernel checks the signature on code it itself loads, this chain of trust can be extended quite far into the running system. Unfortunately, the only signing key that is trusted by most implementations is held by Microsoft.
There are 2 major reasons for supporting Secure Boot in Debian:
 * some computers now ship with Secure Boot enabled by default, making it harder to install Debian;
 * while not perfect, it is a technology that can be used to make Debian user safer.
The plan the Ben (bwh) has been hatching is as follows:
 * a minimalistic shim bootloader is signed by Microsoft;
 * the shim load a bootloader that was properly signed by Debian (in the long run, ftpmaster@; right now, it’s bwh’s signing key);
 * the bootloader loads a kernel signed by Debian;
 * the kernel only accepts to load code signed by Debian (securelevel = 1).
The signing process itself uses signature packages, so as not to keep signing keys on the buildds or break reproducibility.
 * no dependency on Microsoft, once the shim is signed (and it should need fixes very seldom);
 * robust process that can take advantage of reproducible builds;
 * gives reasonable guarantees that the running kernel is a legitimate one;
 * trusting only Debian (as opposed to anything Microsoft signs) can easily be achieved by shipping a Debian-signed shim and having the user put the Debian key as the only trusted one.
 * doesn’t protect the userspace (yet!);
 * still vulnerable to somebody with a kernel exploit (but this doesn’t grant persistence) or who can get a bootloader signed by Microsoft.
Help us, fellow Debian hackers! You are our only hope.

Secure Boot for Debian Linux
Ben Hutchings
Three years after a “Plan of action” for Secure Boot support, we’ve had another release without it and there are still many changes required. What is left to do and how will we finish it in time for “stretch”?

Using LAVA for Debian
Neil Williams
How to use LAVA to provide test support on real hardware which can be remote or local to the user.
 * publish local tests from your desk to support testing packages like u-boot.
 * install lava-dispatcher on a machine on your LAN and publish local tests for everyone to view and analyse
 * run CI on the Linux kernel packages on hardware – ramdisk, NFS and SATA media
 * test DI on real hardware (typically ARM).
 * publish local tests of VM images, including live images, and potentially run tests on VM images where appropriate hardware is available.
 * run server-client tests on relevant hardware which cannot be easily performed in sbuild or single VM instances.
 * support for VLAN testing is available although unlikely to be via itself.
 * support for Debian SSO for account creation.
 * XMLRPC and REST API interfaces.

Debugging the IoT
Bdale Garbee Bernelle Verster Andy Simpkins
Panel discussion, aimed at the general public and more technical participants alike. The panel will discuss the open hardware movement, and how it fits in with Smart Homes. It will highlight and discuss the futurology, trends, and challenges. Challenges include security, the role of big vendors, the requirement for a more powerful platform, competing interests and the role of industrial providers. The panel will be hosted by Bernelle Verster, and panelists include Andy Simpkins and others. (Please get in touch if you want to be on the panel too).

Debian on ARM devices
Martin Michlmayr
This talk will cover Debian on ARM devices, including NAS devices, development boards and other devices. The talk will briefly explain how the installer works on ARM from the point of view of a user. It will then cover in detail how Debian on ARM is different to Debian on x86 devices and what infrastructure we created in Debian to support a wide range of ARM devices, such as flash-kernel. Some supported platforms and devices will be covered as well.

Linaro LAVA changes

LAVA is a Continuous Integration tool for testing firmware, pre-OS environment, and embedded OSes, including QEMU-based systems as well as live hardware. Linaro is refactoring the code, which will impact test code and their running validation service, as well as renaming Linaro-Validation to lava-devel. The lava-users and lava-announce lists still exist. Neil Williams of Linaro announced some changes to LAVA, after discussing things at last week’s Linaro Connect. Excerpts of anouncement:

The LAVA dispatcher is being refactored and this had led to advancements and modifications in the lava-server as well as a completely re-written job submission format. LAVA is retaining compatibility *only* with the Lava Test Shell Definitions (the YAML files people are currently using) and there can be no automated way of converting existing JSON job submissions to the new job submission format (which uses YAML to allow for comments, amongst other improvements). The refactoring introduces a lot of benefits, including much more robust communication between the workers and the master, removal of configuration on the workers so that admins only change things in one place, a lot of new methods within the dispatcher to support new types of test and a much cleaner, more modular, codebase for future development. The timetable for these changes is expected to cover most of 2016.

The LAVA developers would ask that everyone running LAVA would subscribe to at least the lava-announce mailing list, to help with the migration to the new support.

Full announcement:

More information:

Linaro’s 96Boards initiative

Earlier this year, ARM’s Linaro created

The 96Boards initiative is designed to offer a single software and hardware community across multiple vendor boards supporting a range of different features. A fixed set of minimum functions including USB, SD, HDMI and standardized low speed and high speed peripheral connectors are provided. Vendors may add customized hardware and feature sets provided the minimum functions are available. We expect this to extend the platform life, increase the market for add-on hardware, and accelerate open source upstreaming of support for new SoC features. The 96Boards standard specification and this website are maintained by the Linaro Community Board Group (LCG). Linaro is a collaborative software engineering organization focused on the ARM architecture. Corporate members of Linaro provide funding and engineers plus direction through various steering committees and resources are split into semi-autonomous groups with their own members.

There are currently two 96Boards specifications for low-cost ARMv7-A and ARMv8-A development boards:
* The Consumer Edition (CE) targets the mobile, embedded and digital home segments.
* The Enterprise Edition (EE) targets the networking and server segments.

They have 3 boards listed currently:
* DragonBoard 410c: Board based on Qualcomm Snapdragon™ 410 processor
* The HiKey Board: Board based on HiSilicon Kirin 6220 processor
* 96Boards UART Serial Adapter: a USB to UART interface to be used with any 96Boards Consumer or Enterprise Edition board.

Marcin Juszkiewicz has a good blog post on 96boards as well:

ARM Devices has a video discussing adding 96boards hardware targets to LAVA, the CI server by Linaro for embedded device testing.

Intel ITS

Intel has a device to help with UEFI testing, the Intelligent Test System (ITS). ITS may have been around for a while, but I just noticed it on the Intel firmware web site. I’m presuming for now that it’s new from IDF, but I may be wrong about that. If you do UEFI testing, you might want to look at this device.

I wonder how it compares to Linaro’s LAVA. LAVA does mostly target ARM devices, but does also target Intel via QEMU, perhaps there are direct Intel targets these days.

New LAVA tool from Collabora

Today, Collabora released ‘lqa’, a new command line tool — and new Python API — for working with LAVA. LAVA is Linaro’s test tool that enables ‘continuous integration’-style testing with embedded devices (including QEMU), to update the firmware and OS, and run tests on the device. The main LAVA interface is a web UI. The tool is mainly intended for embedded development/QA, but is also useful for security researchers. Quoting their announcement on the linaro-validation mailing list:

Collabora has been working on `lqa’, a tool to submit and manage LAVA jobs, which helps to get many of the LAVA job administration and monitoring tasks conveniently done from the command line. `lqa’ brings a new API, lqa_api python module, a complete set of classes to easily interact with LAVA and offering at the same time a clean API on top of which further applications can be built upon (like `lqa’ itself). It has a templating system (using jinja2 package) that allows to use variables in json job files (in future could be expanded to support yaml), specifying their values either from a profile file or directly from the command line making possible the dynamic assignments of template variables during the `lqa’ command execution. The templating mechanism allows to handle groups of jobs, therefore it makes it easier to submit jobs in bulk. `lqa’ also features a flexible profile system (in YAML) which allows to specify a ‘main-profile’ from which further sub-profiles can inherit values, avoiding information duplication between similar profiles. Other of the current features include: Test report generation with the ‘analyse’ subcommand, Polling to check for job completion, All the operations offer logging capabilities, and Independent profile and configuration files.

More Information: