Performance improvement: only re-render top-level component

[RobinMalfait]

This PR fixes a performance issue where all components using the useIsTopLayer hook will re-render when the hook changes.

For context, the internal hook is used to know which component is the top most component. This is important in a situation like this:

<Dialog>
  <Menu />
</Dialog>

If the Menu inside the Dialog is open, it is considered the top most component. Clicking outside of the Menu or pressing escape should only close the Menu and not the Dialog.

This behavior is similar to the native #top-layer you see when using native dialogs for example.

The issue however is that the useIsTopLayer subscribes to an external store which is shared across all components. This means that when the store changes, all components using the hook will re-render.

To make things worse, since we can't use these hooks unconditionally, they will all be subscribed to the store even if the Menu component(s) are not open.

To solve this, we will use a new state machine and use the useMachine hook. This internally uses a useSyncExternalStoreWithSelector to subscribe to the store.

This means that the component will only re-render if the state computed by the selector changes.

This now means that at most 2 components will re-render when the store changes:

1. The component that was in the top most position
2. The component that is going to be in the top most position

Fixes: #3630
Closes: #3662

Test plan

Behavior before: notice how all Menu components re-render:

Screen.Recording.2025-05-10.at.02.00.21.mov

After this change, only the Menu that was opened / closed will re-render:

Screen.Recording.2025-05-10.at.01.58.51.mov

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
headlessui-react	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2025 0:22am
headlessui-vue	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2025 0:22am