What Linux' nomodeset Option Really Does

The Linux kernel’s command-line option nomodeset is a workaround to get display output on systems with broken graphics drivers. It has become the first stop-gap solution to many boot-up problems involving graphics output.

Common wisdom is that nomodeset makes Linux use the firmware’s framebuffer by disabling kernel mode setting. But that’s not what happens. I’ll go through the details of nomodeset in this blog post and give some context.

When the user states nomodeset on the kernel’s command line, the kernel will use the device firmware’s framebuffer for graphics output. You can try this easily if you run a recent Linux system. Append an additional nomodeset to the kernel’s command line in the boot loader. Booting the edited entry will give you only rudimentary graphics output with software-only rendering

The graphics device in this case is the scanout buffer that the firmware (or boot loader) already initialized. The Linux kernel treats it as just another regular device. One of the compatible DRM drivers, sich as efidrm or simpledrm, binds to it and marks the device as owner of the scanout buffer’s physical video memory.

Under normal operation, the kernel later loads the hardware’s native DRM driver and binds it to the PCI device that represents the physical graphics card. As part of this process the native driver removes every device that currently owns part of the PCI hardware’s physical video memory. Because the scanout buffer is located somewhere within that range, the native driver removes the earlier device and its driver. After that, it reprograms the hardware and the scanout buffer is gone. This logic has been there for decades in one way or another; even before DRM was a thing.

Setting the nomodeset flag prevents the native DRM driver from registering at all. The driver will not even be initialized. And that means it cannot remove the scanout buffer’s device and the DRM driver bound to it.

If you want to investiate the kernel’s source code, follow the macro drm_module_pci_driver(). Modules that register a PCI driver with this macro invoke drm_pci_register_driver(), which in turn invokes drm_firmware_drivers_only(). Ultimately the state variable video_nomodeset controls registering. This is the state initialized by nomodeset.

This design has a number of implications. The first is that nomodeset never disables kernel mode setting. The initially bound driver, such as efidrm or simpledrm, is a full-featured DRM driver with kernel mode settings. It provides all necessary kernel interfaces like mode setting, page flipping, and GEM memory management for graphics buffers. It can even export those graphics buffers as dma-buf to other devices, or import dma-bufs from other devices.

The one thing this driver cannot do is change the hardware registers. Resolution or the color format of the scanout buffer has been fixed by firmware during boot. All rendering, video processing, and other hardware features has to be done in software, because that’s not covered by the firmware scanout buffer at all.¹ But these limitations are not much different from display panels. Panel devices also provide only a single display mode, color format and no hardware acceleration.

This second point is that without a native DRM driver available for the hardware, the kernel continues to use the DRM driver for the scanout buffer’s device by default. This is how Linux supports arbitrary any display hardware as long as the firmware can initialize it.

Finally, the consequence is that all of user space can rely on DRM to be available on the kernel side. No special cases are necessary for unsupported or broken hardware. Those originally got covered by fbdev and its generic infrastructure; and thus required extra support from user space. Nowadays it’s all been converted to DRM. Removing fbdev from the graphics stack also removed one of the last remaining use cases for Xorg: controlling the fbdev driver. So Xorg has mostly been removed as well.

Getting back to nomodeset, the question is why it is named the way it is, when the option really just means to never replace a firmware’s scanout buffer. The reason is that nomodeset is much older than it’s current incarnation. Before kernel v5.17, DRM drivers tested it with vgacon_text_force() and aborted early when it was set. The VGA text-mode console provided nomodeset on the kernel command line to make the option configurable by the user. In this case it really meant do no kernel mode setting. When we added the ability to keep firmware scanout buffers running, we found that the existing nomodeset already had the necessary semantics. So instead of introducing yet another kernel command-line option, we adopted nomodeset from the VGA console to serve the broader use case.

The VGA console still works as before. With nomodeset given, the kernel not only keeps any firmware scanout buffer, but also keeps a VGA text-mode console if there is any.

And we also updated the old fbdev drivers to work well with all the other components. An fbdev driver that uses the firmware scanout buffer can be removed by a native DRM driver, but a DRM driver that uses the firmware scanout buffer and also be removed by a native fbdev driver. Any combination works. If nomodeset has been given, it also stops any native fbdev driver.

So far so good. Hopefully you found something interesting in this blog post. If you encounter problems with graphics drivers or hardware support, try nomodeset for basic display output.