Introduction

libibverbs is a library that allows programs to use RDMA "verbs" for direct access to RDMA (currently InfiniBand and iWARP) hardware from userspace. For more information on RDMA verbs, see the InfiniBand Architecture Specification vol. 1, especially chapter 11, and the RDMA Consortium's RDMA Protocol Verbs Specification.

Using libibverbs

Device nodes

The verbs library expects special character device files named /dev/infiniband/uverbsN to be created. When you load the kernel modules, including both the low-level driver for your IB hardware as well as the ib_uverbs module, you should see one or more uverbsN entries in /sys/class/infiniband_verbs in addition to the /dev/infiniband/uverbsN character device files.

To create the appropriate character device files automatically with udev, a rule like

KERNEL="uverbs*", NAME="infiniband/%k"

can be used. This will create device nodes named

/dev/infiniband/uverbs0

and so on. Since the RDMA userspace verbs should be safe for use by non-privileged users, you may want to add an appropriate MODE or GROUP to your udev rule.

Permissions

To use IB verbs from userspace, a process must be able to access the appropriate /dev/infiniband/uverbsN special device file. You can check the permissions on this file with the command

ls -l /dev/infiniband/uverbs*

Make sure that the permissions on these files are such that the user/group that your verbs program runs as can access the device file.

To use IB verbs from userspace, a process must also have permission to tell the kernel to lock sufficient memory for all of your registered memory regions as well as the memory used internally by IB resources such as queue pairs (QPs) and completion queues (CQs). To check your resource limits, use the command

ulimit -l

(or "limit memorylocked" for csh-like shells).

If you see a small number such as 32 (the units are KB) then you will need to increase this limit. This is usually done for ordinary users via the file /etc/security/limits.conf. More configuration may be necessary if you are logging in via OpenSSH and your sshd is configured to use privilege separation.

Debugging

Enabling debug prints

Library and providers debug prints can be enabled using the VERBS_LOG_LEVEL environment variable, the output shall be written to the file provided in the VERBS_LOG_FILE environment variable. When the library is compiled in debug mode and no file is provided the output will be written to stderr.

Note: some of the debug prints are only available when the library is compiled in debug mode.

The following table describes the expected behavior when VERBS_LOG_LEVEL is set:

	Release	Debug
Regular prints	Output to VERBS_LOG_FILE if set	Output to VERBS_LOG_FILE, or stderr if not set
Datapath prints	Compiled out, no output	Output to VERBS_LOG_FILE, or stderr if not set

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!