From 76ac43ad3cb463fadbd706ac052fed7f1c215915 Mon Sep 17 00:00:00 2001 From: Matt Elliott Date: Mon, 18 Nov 2024 09:49:30 -0500 Subject: [PATCH] Add instructions for using udev to manage GPU access (#325) * Added instructions for using udev to manage GPU access * editing * additional edits * update code block syntax * Updated wordlist to resolve linting error * remove duplicate word from .wordlist.txt * spell check does not ignore case :) * Removed "recommended" from the udev section header, moved group membership to be the first method listed * Group Membership -> group membership Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com> * udev Rules -> udev rules (sorry, old habits die hard)) Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com> * Udev -> udev rewording Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com> * Update .wordlist.txt Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com> * Update .wordlist.txt troubleshooting linting failure --------- Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com> --- .wordlist.txt | 3 ++ docs/install/prerequisites.rst | 95 ++++++++++++++++++++++++++++++---- 2 files changed, 87 insertions(+), 11 deletions(-) diff --git a/.wordlist.txt b/.wordlist.txt index 665f5192..8e90b058 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -6,6 +6,7 @@ CTest DHCP dkms dmesg +DRM fw FW FWDEV @@ -32,3 +33,5 @@ vCPU vfio VFIO vhost +udev +Udev diff --git a/docs/install/prerequisites.rst b/docs/install/prerequisites.rst index 39671519..6e04d210 100644 --- a/docs/install/prerequisites.rst +++ b/docs/install/prerequisites.rst @@ -192,11 +192,22 @@ To install for the currently active kernel run the command corresponding to your .. _group_permissions: -Setting permissions for groups +Configuring permissions for GPU access ================================================================ -This section provides steps to add any current user to a video group to access GPU resources. We -recommend using the video group for all ROCm-supported operating systems. +There are two primary methods to configure GPU access for ROCm: group membership or +udev rules. Each method has its own advantages, and the choice depends on your +specific requirements and system management preferences. + +Using group membership +-------------------------------------------------------------------- + +By default, GPU access is managed through membership in the ``video`` and ``render`` groups. +The ``video`` and ``render`` groups are system groups in Linux used to manage access +to graphics hardware and related functionality. Traditionally, the ``video`` group is used +to control access to video devices, including graphics cards and video capture devices. +The ``render`` group is more recent and specifically controls access to GPU rendering capabilities +through Direct Rendering Manager (DRM) render nodes. 1. To check the groups in your system, issue the following command: @@ -204,23 +215,85 @@ recommend using the video group for all ROCm-supported operating systems. groups -2. Add yourself to the ``render`` and ``video`` group using the command: +2. Add yourself to the ``video`` and ``render`` groups: + + .. code-block:: shell + + sudo usermod -a -G video,render $LOGNAME + +3. Optionally, add other users to the ``video`` and ``render`` groups: + + .. code-block:: shell + + sudo usermod -a -G video,render user1 + sudo usermod -a -G video,render user2 + +4. To add all future users to the render and video groups by default, run the following commands: + + .. code-block:: shell + + echo 'ADD_EXTRA_GROUPS=1' | sudo tee -a /etc/adduser.conf + echo 'EXTRA_GROUPS=video' | sudo tee -a /etc/adduser.conf + echo 'EXTRA_GROUPS=render' | sudo tee -a /etc/adduser.conf + +Using udev rules +-------------------------------------------------------------------- +A flexible way to manage device permissions is to use udev rules. They apply system-wide, can be +easily deployed via configuration management tools, and eliminate the need for user group management. +This method provides more granular control over GPU access. + +Grant GPU access to all users on the system +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create a new file ``/etc/udev/rules.d/70-amdgpu.rules`` with the following content: + + .. code-block:: shell + + KERNEL=="kfd", MODE="0666" + SUBSYSTEM=="drm", KERNEL=="renderD*", MODE="0666" + +2. Reload the udev rules: + + .. code-block:: shell + + sudo udevadm control --reload-rules && sudo udevadm trigger + +This configuration grants all users read and write access to AMD GPU resources, +including the AMD Kernel Fusion Driver (KFD) and Direct Rendering Manager (DRM) devices. + +Grant GPU access to a custom group +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create a new group (e.g., ``devteam``): .. code-block:: shell - sudo usermod -a -G render,video $LOGNAME + sudo groupadd devteam - To add all future users to the ``render`` and ``video`` groups by default, run the following commands: +2. Add users to the new group: .. code-block:: shell - echo 'ADD_EXTRA_GROUPS=1' | sudo tee -a /etc/adduser.conf - echo 'EXTRA_GROUPS=video' | sudo tee -a /etc/adduser.conf - echo 'EXTRA_GROUPS=render' | sudo tee -a /etc/adduser.conf + sudo usermod -a -G devteam dev1 + sudo usermod -a -G devteam dev2 + +3. Create udev rules to assign GPU devices to this group: + + Create a file ``/etc/udev/rules.d/70-amdgpu.rules`` with: + + .. code-block:: shell + + KERNEL=="kfd", GROUP="devteam", MODE="0660" + SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="devteam", MODE="0660" + +4. Reload the udev rules: + + .. code-block:: shell -.. tip:: + sudo udevadm control --reload-rules && sudo udevadm trigger - On systems with multiple users, if ROCm is installed system wide, each individual user should be added to the ``render`` and ``video`` groups. +This configuration grants all users in the ``devteam`` group read and write access to AMD GPU resources, +including the Kernel Fusion Driver (KFD) and Direct Rendering Manager (DRM) devices. Disable integrated graphics (IGP), if applicable ================================================================