Christoph Hellwig has submitted multiple patches Linux’s existing NVMe support to enable ‘NVMe over Fabrics’ support. I’ve included his initial comments for each of the 4 patches:
general preparation for NVMe over Fabrics support
This patch set adds some needed preparations for the upcoming NVMe over Fabrics support. Contains:
– Allow transfer size limitations for NVMe transports
– Add the get_log_page command definition required by the NVMe target
– more helpers in core code that can be used by various transports
– add some missing constants and identify attributes
NVMe over Fabrics target implementation
This patch set adds a generic NVMe over Fabrics target. The implementation conforms to the NVMe 1.2b specification (which includes Fabrics) and provides the NVMe over Fabrics access to Linux block devices. The target implementation consists of several elements:
– NVMe target core: defines and manages the NVMe entities (subsystems, controllers, namespaces, …) and their allocation, responsible for initial commands processing and correct orchestration of the stack setup and tear down.
– NVMe admin command implementation: responsible for parsing and servicing admin commands such as controller identify, set features, keep-alive, log page, …).
– NVMe I/O command implementation: responsible for performing the actual I/O (Read, Write, Flush, Deallocate (aka Discard). It is a very thin layer on top of the block layer and implements no logic of it’s own. To support exporting file systems please use the loopback block driver in direct I/O mode, which gives very good performance.
– NVMe over Fabrics support: responsible for servicing Fabrics commands (connect, property get/set).
– NVMe over Fabrics discovery service: responsible to serve the Discovery log page through a special cut down Discovery controller.
The target is configured using configfs, and configurable entities are:
– NVMe subsystems and namespaces
– NVMe over Fabrics ports and referrals
– Host ACLs for primitive access control – NVMe over Fabrics access control is still work in progress at the specification level and will be implemented once that work has finished.
To configure the target use the nvmetcli tool from http://git.infradead.org/users/hch/nvmetcli.git, which includes detailed setup documentation.
In addition to the Fabrics target implementation we provide a loopback driver which also conforms the NVMe over Fabrics specification and allows evaluation of the target stack with local access without requiring a real fabric.
Various test cases are provided for this implementation: nvmetcli contains a python testsuite that mostly stresses the configfs interface of the target, and we have various integration tests prepared for the kernel host and target which are available at:
This repository also contains patches from all the series posted today in case you prefer using a git repository over collecting patches.
NVMe over Fabrics RDMA transport drivers
This patch set implements the NVMe over Fabrics RDMA host and the target drivers. The host driver is tied into the NVMe host stack and implements the RDMA transport under the NVMe core and Fabrics modules. The NVMe over Fabrics RDMA host module is responsible for establishing a connection against a given target/controller, RDMA event handling and data-plane command processing. The target driver hooks into the NVMe target core stack and implements the RDMA transport. The module is responsible for RDMA connection establishment, RDMA event handling and data-plane RDMA commands processing. RDMA connection establishment is done using RDMA/CM and IP resolution. The data-plane command sequence follows the classic storage model where the target pushes/pulls the data.
generic NVMe over Fabrics library support
This patch set adds the necessary infrastructure for the NVMe over Fabrics functionality and the NVMe over Fabrics library itself. First we add some needed parameters to NVMe request allocation such as flags (for reserved commands – connect and keep-alive), also support tag allocation of a given queue ID (for connect to be executed per-queue) and allow request to be queued at the head of the request queue (so reconnects can pass in flight I/O). Second, we add support for additional sysfs attributes that are needed or useful for the Fabrics driver. Third we add the NVMe over Fabrics related header definitions and the Fabrics library itself which is transport independent and handles Fabrics specific commands and variables. Last, we add support for periodic keep-alive mechanism which is mandatory for Fabrics.
For more information, see the linux-(nvme,block,kernel) mailing lists.