Intel Distribution for Python

Intel has a new Anaconda Python distribution, built with Intel libs to enhance speed/concurrency to help.

It is registration-required freeware if you download it from Intel. It mentions below that  it might also be installable via Anaconda, unclear if registration is required for Anaconda. I’ve not found the source to this project yet, but I could’ve missed it.

A few excerpts from their home page below:

Get up to 97 times faster numerical processing on SciPy stack: NumPy, SciPy, and scikit-learn boosted by the Intel® Math Kernel Library. Speed up data analytics with pyDAAL and parallelize Python workloads with enhanced Intel® Threading Building Blocks (Intel® TBB).

The all-included, out-of-the box distribution accelerates core Python packages including NumPy, SciPy, pandas, scikit-learn, Jupyter, matplotlib, and mpi4py. It integrates the powerful Intel® Math Kernel Library (Intel® MKL), Intel® Data Analytics Acceleration Library (Intel® DAAL) and pyDAAL, Intel® MPI Library, and Intel® Threading Building Blocks (Intel® TBB).

Multithreaded Python workloads can take advantage of multicore architectures from Intel using Intel TBB optimized thread scheduling and efficient communication with the Intel MPI Library.

Installing and managing your Python environment is made easy with the all-in-one installer from Intel or by installing packages from Anaconda Cloud using conda. Third-party packages can be installed with both conda and PIP.

Use the Intel® Distribution for Python powered by Anaconda as a drop-in replacement for your current Python environment to get high performance out of the box. Your Python applications immediately gain significant performance and can further be tuned to extract every last bit of performance using the Intel® VTune™ Amplifier.

The Intel® Distribution for Python is available for free for Python 2.7.x and 3.5.x on OS X*, Windows* 7 and later, and Linux*.

PS: Alas, this does not to appear to be integrated with Intel’s port of CPython to UEFI. 🙂



Marc-Etienne M.Lévei has an IDA shell in IPython! (I wish more security tool  projects would integrate with IPython.)

IPyIDA is a python-only solution to use a IPython console in the context of IDA Pro. It spawns an IPython kernel that you can connect to with ipython console –existing in your shell or by opening a QT Console window in IDA Pro with <Shift-.>. You can then benefit from IPython’s autocompletion, online help, monospaced font input field, graphs, and so on.

UEFI Firmware Parser now in Cheese Shop

The other day I noticed some Github activity for Teddy Reed’s UEFI Firmware Parser, but didn’t notice any formal new announcement. It appears I was not looking in the right place. The parser is now in the official Python Cheese Shop! And it is named “uefi_firmware”, not UEFI Firmware Parser, that explains that comment in the comment log. 🙂 It’ll be nice to have this tool more easily-available in Python. I hope the next time the UEFI Forum updates it’s UEFI port of CPython, they add this module to the UEFI port.

GDB Dashboard: Python visual UI

Modular visual interface for GDB in Python. This comes as a standalone single-file .gdbinit which, among the other things, enables a configurable dashboard showing the most relevant information during the program execution. Its main goal is to reduce the number of GDB commands issued to inspect the current program status allowing the programmer to focus on the control flow instead.

tool: ThunderGate

I just learned about ThunderGate, by Saul St John, The current version is 0.8.499, initial release was 4 months ago. It is a Python RE tool for Apple Thunderbolt Ethernet (and other) controllers, with PCI Option ROM, and UEFI support! I’m excerpting the readme and usage output below, see the URLs for full details, including omitted scary warning disclaimers:

ThunderGate is a collection of tools for the manipulation of Tigon3 Gigabit Ethernet controllers, with special emphasis on the Broadcom NetLink 57762, such as is found in Apple Thunderbolt Gigabit Ethernet adapters. Tigon3 controllers contain a variety of architectural blocks, including a PCI endpoint, an 802.3 media access controller, on-chip ram, DMA read and write engines, nonvolatile storage, and one or more MIPS processors. These features are exposed by ThunderGate through an easy-to-use Python interface, allowing for reverse engineering, development, and deployment of custom firmware and applications. Examples provided include a userspace VFIO tap driver, a firmware application capable of monitoring and manipulating network traffic and host memory, and a PCI option rom containing an EFI boot services driver which can either inhibit the employ or compromise the effectivity of Intel I/O MMU address translation (VT-d). The ThunderGate firmware implements a network protocol allowing for remote control of the device and host system by an Ethernet-connected peer. Currently supported actions include reading and writing from device and host memory, forging network traffic, sending host interrupts, and manipulation of PCI capabilities configuration.

$ py/ -h
usage: [-h] [-v] [-d] [-t] [-s] device
  device        BDF of tg3 PCI device
  -h, –help     show this help message and exit
  -i, –install  install thundergate firmware
  -u, –uio      use uio pci generic interface
  -v, –vfio     use vfio interface
  -d, –driver   load userspace tap driver
  -t, –tests    run tests
  -s, –shell    ipython cli

More Information:

ACPI testing with BITS Python

Recently, Josh Triplett of Intel gave a talk on using BIOS Interface Test Suite (BITS) at LinuxCon North America.

Demystifying ACPI and EFI via Python and BITS

Click to access bits-with-demo.pdf

BTW, Josh also gave this talk at LinuxConNA’15 as well:

Everything’s a File Descriptor

Click to access fd_0.pdf

I think I’ve mentioned BITS in this blog before. But just in case I’ve not, BITS is a powerful, strange set of BIOS diagnostic tools. BITS started as a BIOS-centric tool, but now includes some UEFI support as well. BITS uses the GRUB boot manager as it’s UI, using GRUB menus for different features, see the screenshots page for a better understanding:

BITS also includes a Python interpreter, so you can do interactive Python, or write scripts to test firmware. BITS has interfaces for BIOS, UEFI, and ACPI data.

Jake Edge wrote an excellent follow-up to Josh’s LinuxCON talk, with an article in, discussing BITS’s Python for UEFI and ACPI investigations.

In a talk that could easily be seen as a follow-on to his PyCon 2015 talk, Josh Triplett presented at LinuxCon North America on using Python to explore the low-level firmware of today’s systems. The BIOS Implementation Test Suite (BITS) provides an environment that hearkens back to the days of BASIC, PEEK, and POKE, as he demonstrated at PyCon in Montréal in April, but it is much more than that. In Seattle at LinuxCon, he showed that it can also be used to look at and use the EFI and ACPI code in a system—all from Python.

The article is part of subscriber-only content, and has been ‘leaked’ (see next URL below), and as the link on the page mentions, an occasional leak isn’t too bad, and helps with subscriptions. If you don’t have a LWN subscription, please think about it, they are probably the best news source for low-level Linux technologies. They have a 1-month free trial.

After reading this article, Laszlo Ersek of Red Hat started up a thread with Josh on the QEMU and UEFI dev mailing lists, with some new ways of thinking about using BITS Python for ACPI testing. Lots of good ideas on this thread, if you care about QEMU, ACPI, AML, or ACPICA tools please read the thread: sorry, I’m too lazy to summarize all of the ACPI nuances in the thread, it’s only a few messages.

Using Python to investigate EFI and ACPI
Newsgroups: gmane.comp.emulators.qemu, gmane.comp.bios.edk2.devel

I hope some of the ACPI/AML testing ideas in this thread happen!

More Information:

OpenStack’s hardware introspection service 2.0 released

Dmigtry Tantsur of Red Hat announced version 2.0 of OpenStack’s hardware introspection service was released today on the openstack-announce list.

“This is an auxiliary service for discovering hardware properties for a node managed by OpenStack Ironic. Hardware introspection or hardware properties discovery is a process of getting hardware parameters required for scheduling from a bare metal node, given it’s power management credentials (e.g. IPMI address, user name and password). A special ramdisk is required to collect the information on a node. The default one can be built using diskimage-builder and ironic-discoverd-ramdisk element. Highlights of this release:

* Main Python module was renamed to ironic_inspector
* Client library was split away to a separate project
* edeploy plugin was removed in favor of more generic one called ‘extra_hardware’
* Processing hooks interface was changed
* The way we return API errors was changed to better match Ironic one
* Removed deprecated /v1/discover endpoint
* All options (except for ‘database’) were moved to sections instead of  using ‘discoverd’ for everything
* oslo.db configuration should be used instead of ‘discoverd.database’  option
* keystonemiddleware options should be used instead of reusing ‘ironic’  credentials for checking authentication
* Deprecated ‘authenticate’ opt in favor of ‘auth_strategy’
* Explicit green thread pool is used instead of just launching new threads
* NodeInfo class became more helpful for hooks
* Now it’s possible to hook into processing chain when node is not found
* Inspector no longer checks for Ironic presence on start up as it was  causing problems in real life
* SSL/TLS Support”

More Information:

New LAVA tool from Collabora

Today, Collabora released ‘lqa’, a new command line tool — and new Python API — for working with LAVA. LAVA is Linaro’s test tool that enables ‘continuous integration’-style testing with embedded devices (including QEMU), to update the firmware and OS, and run tests on the device. The main LAVA interface is a web UI. The tool is mainly intended for embedded development/QA, but is also useful for security researchers. Quoting their announcement on the linaro-validation mailing list:

Collabora has been working on `lqa’, a tool to submit and manage LAVA jobs, which helps to get many of the LAVA job administration and monitoring tasks conveniently done from the command line. `lqa’ brings a new API, lqa_api python module, a complete set of classes to easily interact with LAVA and offering at the same time a clean API on top of which further applications can be built upon (like `lqa’ itself). It has a templating system (using jinja2 package) that allows to use variables in json job files (in future could be expanded to support yaml), specifying their values either from a profile file or directly from the command line making possible the dynamic assignments of template variables during the `lqa’ command execution. The templating mechanism allows to handle groups of jobs, therefore it makes it easier to submit jobs in bulk. `lqa’ also features a flexible profile system (in YAML) which allows to specify a ‘main-profile’ from which further sub-profiles can inherit values, avoiding information duplication between similar profiles. Other of the current features include: Test report generation with the ‘analyse’ subcommand, Polling to check for job completion, All the operations offer logging capabilities, and Independent profile and configuration files.

More Information:

Upcoming features in UEFI Python port

Today, on the EDK2-devel mailing list, Daryl McDaniel of Intel gave us a hint about upcoming changes in the UEFI port of CPython 2.7x. I am looking forward to UEFI  ctypes, as well as threading!

More Information, quoting Daryl’s posting:

Later this year I will be committing a port of the ctypes module for EDK II Python.  The built-in edk2 module will also be extended to provide a pointer to the SystemTable which can then be used with the ctypes module to access any of the Boot or Runtime Services as well as loading protocols and accessing their member functions and data. I hope to follow that with some pure Python code that allows direct access to UEFI functionality without the user having to know how to use ctypes.  This is not on the official plan but is just something I would like to do so I can’t give a definite schedule for it. Things that are queued up (in no particular order) are:
    *  command-line switch to force stderr to stdout, similar to 2>&1 redirection.
    *  ctypes for IA32 and X64
    *  threading
    *  4Suite-XML
    * cDeepCopy
    *  zope interface
    *  UEFI wrappers for ctypes