Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation

[Dan Williams]

[ add linux-cxl for the Linux driver question ]


On Thu, Dec 3, 2020 at 9:10 PM Chris Browy <cbrowy@avery-design.com> wrote:
[..]

I'll let Ben address the other questions...

> 4. What are the userspace system APIs for targeting CXL HDM address domain?
>    Usually you can mmap a SPA if you know how to look it up.

tl;dr: /proc/iomem and sysfs

There are a lot of moving pieces to this question. Once a CXL.mem
device is mapped the driver will arrange for it to appear in
/proc/iomem. If it is persistent memory it will be accessed through a
/dev/pmem or /dev/dax device, and if it is volatile it will show up as
just another "System RAM" range. The System RAM range may end up with
a unique Linux NUMA node id. Alternatively, if the BIOS maps the
device then the BIOS will set up the EFI memory map + ACPI
SRAT/SLIT/HMAT for Linux to interpret.

Once something shows up in /proc/iomem it was traditionally accessible
via /dev/mem. These days though /dev/mem is slowly being deprecated
due to security (see CONFIG_LOCK_DOWN_KERNEL*) and stability (see
CONFIG_IO_STRICT_DEVMEM)  concerns, so depending on your kernel
configuration tools may not be able to use /dev/mem to map the SPA
directly and will need to go through the driver provided interface.

The driver needs to be able coordinate coherent HDM settings across
multiple devices in an interleave, all switches in the path, and the
root bridge(s). So I am expecting a sysfs interface similar to the
nvdimm-subsystem and device-dax-subsystem where there will be
sysfs-files to enumerate the mappings and a provisioning mechanism to
reconfigure HDMs with various interleave settings. For a flavor of
what this might look like see the description of NVDIMM "Region"
devices:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/nvdimm/nvdimm.rst#n469

The difference will be that unlike the nvdimm case where regions are
platform-firmware-defined and static, CXL regions are runtime dynamic.