ioctl: defer ioctl version probing until first use

[igaw]

The eager ioctl probing with a dummy command causes the kernel to emit log messages. In addition, probing is performed for every handle, which becomes unnecessarily expensive when iterating over large setups with many fabrics controllers.

Defer ioctl version probing until the first actual command submission. Prefer the 64-bit ioctl opportunistically and fall back to the 32-bit variant only when the kernel returns -ENOTTY.

When --no-ioctl-probing is specified, skip probing entirely and use the 32-bit ioctl directly.

Fixes: #3349 #3348 #3312

[Copilot]

Pull request overview

This PR changes libnvme’s ioctl selection strategy to avoid eager “dummy command” probing (which can trigger kernel log noise) and to reduce per-handle overhead by deferring probing until the first real passthru submission. It opportunistically tries the 64-bit ioctl first and falls back to the 32-bit ioctl only when the kernel returns -ENOTTY; --no-ioctl-probing skips the 64-bit attempt.

Changes:

• Replace per-handle boolean flags (ioctl_admin64, ioctl_io64) with an explicit cached enum ioctl_state for admin and I/O ioctl selection.
• Remove eager ioctl probing during direct-handle open and instead probe lazily on the first command submission.
• Factor direct admin passthru submission into a helper that applies the new probe/cache behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
libnvme/src/nvme/private.h	Introduces enum ioctl_state and stores cached ioctl capability state on the transport handle.
libnvme/src/nvme/lib.c	Removes eager probing during direct open; updates test-handle setup to use the new state enum.
libnvme/src/nvme/ioctl.c	Implements lazy 64-bit-first submission with -ENOTTY fallback and caches the selected ioctl state.

Comments suppressed due to low confidence (1)

libnvme/src/nvme/ioctl.c:247

• Same as the IO passthru path: when ioctl_admin_state is UNKNOWN and probing is enabled, this code may issue the admin command via the 64-bit ioctl, observe -ENOTTY, and then re-issue it via the 32-bit ioctl. Because both submit helpers run submit_entry/submit_exit, callers will see duplicate logging/metrics and an extra failure callback for the probe attempt. Please restructure the probe so that submit_entry/exit are only invoked once for the actual submission.

	if (hdl->ioctl_admin_state == IOCTL_STATE_IOCTL64)
		return libnvme_submit_passthru64(hdl,
				LIBNVME_IOCTL_ADMIN64_CMD, cmd);

	if (hdl->ioctl_admin_state == IOCTL_STATE_IOCTL32 ||
			!hdl->ctx->ioctl_probing)
		goto do_ioctl32;

	err = libnvme_submit_passthru64(hdl, LIBNVME_IOCTL_ADMIN64_CMD, cmd);
	if (err >= 0 || err != -ENOTTY) {
		hdl->ioctl_admin_state = IOCTL_STATE_IOCTL64;
		return err;
	}

	hdl->ioctl_admin_state = IOCTL_STATE_IOCTL32;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.