Proposal

The RL environments (e.g., DirectRLEnv) should all inherit from gymnasium.VectorEnv rather than gymnasium.Env.

Motivation

The DirectRLEnv docstring say

This class implements the core functionality for reinforcement learning (RL)
environments. It is designed to be used with any RL library. The class is designed
to be used with vectorized environments, i.e., the environment is expected to be run
in parallel with multiple sub-environments.

While the environment itself is implemented as a vectorized environment, we do not
inherit from :class:gym.vector.VectorEnv. This is mainly because the class adds
various methods (for wait and asynchronous updates) which are not required.
Additionally, each RL library typically has its own definition for a vectorized
environment. Thus, to reduce complexity, we directly use the :class:gym.Env over
here and leave it up to library-defined wrappers to take care of wrapping this
environment for their agents.

This description is true for Gym and Gymnasium < 1.0, however, in v1.0, the problematic functions described (wait and asynchronous updates) were removed from the base class.

Updating to VectorEnv should make it clearer to users that the environments are vectorised and remove the necessity for libraries to implement their own converter functions.
Further, users will be able to use vector wrappers ()with the environments, which isn't possible with gymnasium.Env implementations.

Looking at the code, I believe that minimal code changes will be necessary, as the library already requires Gymnasium v1.2 and we should be able to simplify the implementation to remove components already included in VectorEnv but not in Env

Alternatives

Keep the current solution

Additional context

Add any other context or screenshots about the feature request here.

Checklist

• I have checked that there is no similar issue in the repo (required)