Very low speed when running projects/habitat_ovmm/eval_baselines_agent.py #513
Comments
|
You are sure it is running on the GPU?
… On Aug 7, 2024, at 5:26 AM, Ran Ding ***@***.***> wrote:
Hi, in my machine, I use a P5000 with 16384MiB. I run the code of ovmm projects/habitat_ovmm/eval_baselines_agent.py, but is very slow. It takes 10 min per episode. Is it normal? Thanks.
—
Reply to this email directly, view it on GitHub <#513>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQKNRSCQGFBKUGJXO6DOSTZQIG65AVCNFSM6AAAAABMEJVFYSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2TGMZVHA2DIMA>.
You are receiving this because you are subscribed to this thread.
|
|
Yes, cuda is available, there are some additional warnings when I run this code, is it normal? pybullet build time: Nov 28 2023 23:52:03 |
|
On Aug 7, 2024, at 11:00 AM, Ran Ding ***@***.***> wrote:
Yes, cuda is available, there are some additional warnings when I run this code, is it normal?
Yes, I see the p5000. But I don’t see any evidence of PyTorch using Coda. All of the text below seems to be from OpenGL. I am not saying what is happening, only that I don’t see evidence of it. I could be using the GPU, but we can’t know from the text below.
It is the nature of debugging that something you think is true is not. Finding it is hard and it is best to prove every assumption to yourself.
The simplest thing to do is put some print statements in the code. Some along the line of this maybe.
print(torch.cuda.current_device(),
torch.cuda.device_count(),
torch.cuda.get_device_name(0))
BNut really you should loop through all devices to be 100% correct and general
… pybullet build time: Nov 28 2023 23:52:03
An exception occurred in telemetry logging.Disabling telemetry to prevent further exceptions.
Traceback (most recent call last):
File "/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/iopath/common/file_io.py", line 946, in __log_tmetry_keys
handler.log_event()
File "/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/iopath/common/event_logger.py", line 97, in log_event
del self._evt
AttributeError: _evt
Loading pretrained CLIP
/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525539683/work/aten/src/ATen/native/TensorShape.cpp:3190.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
2024-08-07 19:58:39,056 Initializing dataset OVMMDataset-v0
2024-08-07 19:58:39,591 initializing sim OVMMSim-v0
Renderer: Quadro P5000/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 535.183.01
Using optional features:
GL_ARB_vertex_array_object
GL_ARB_separate_shader_objects
GL_ARB_robustness
GL_ARB_texture_storage
GL_ARB_texture_view
GL_ARB_framebuffer_no_attachments
GL_ARB_invalidate_subdata
GL_ARB_texture_storage_multisample
GL_ARB_multi_bind
GL_ARB_direct_state_access
GL_ARB_get_texture_sub_image
GL_ARB_texture_filter_anisotropic
GL_KHR_debug
GL_KHR_parallel_shader_compile
GL_NV_depth_buffer_float
Using driver workarounds:
no-forward-compatible-core-context
nv-egl-incorrect-gl11-function-pointers
no-layout-qualifiers-on-old-glsl
nv-zero-context-profile-mask
nv-implementation-color-read-format-dsa-broken
nv-cubemap-inconsistent-compressed-image-size
nv-cubemap-broken-full-compressed-image-query
nv-compressed-block-size-in-bits
[19:58:46:474598]:[Warning]:[Sim] Simulator.cpp(508)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID = 0
2024-08-07 19:58:48,281 Initializing task OVMMNavToObjTask-v0
/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
0%| | 0/1199 [00:00<?, ?it/s]Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
[19:58:49:209893]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_gripper_fingertip_right.obj with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file.
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
[19:58:49:213567]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_gripper_fingertip_left.obj with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file.
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
[19:58:49:697287]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_aruco_top_wrist.STL with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file.
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::AssimpImporter::material(): white ambient detected, forcing back to black
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels
Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels
Trade::GltfImporter::material(): property fromMI is an object, skipping
Trade::GltfImporter::material(): property fromMI is an object, skipping
Trade::GltfImporter::material(): property fromMI is an object, skipping
Loading pretrained CLIP
Renderer: Quadro P5000/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 535.183.01
Using optional features:
GL_ARB_vertex_array_object
GL_ARB_separate_shader_objects
GL_ARB_robustness
GL_ARB_texture_storage
GL_ARB_texture_view
GL_ARB_framebuffer_no_attachments
GL_ARB_invalidate_subdata
GL_ARB_texture_storage_multisample
GL_ARB_multi_bind
GL_ARB_direct_state_access
GL_ARB_get_texture_sub_image
GL_ARB_texture_filter_anisotropic
GL_KHR_debug
GL_KHR_parallel_shader_compile
GL_NV_depth_buffer_float
Using driver workarounds:
nv-egl-incorrect-gl11-function-pointers
no-layout-qualifiers-on-old-glsl
nv-zero-context-profile-mask
nv-implementation-color-read-format-dsa-broken
nv-cubemap-inconsistent-compressed-image-size
nv-cubemap-broken-full-compressed-image-query
nv-compressed-block-size-in-bits
—
Reply to this email directly, view it on GitHub <#513 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQKNRTI2WZWCIPXPAPFJWTZQJOD7AVCNFSM6AAAAABMEJVFYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZUGAZDSNJUHA>.
You are receiving this because you commented.
|
|
Thanks Chris for very detailed answer. It seems that I have already used cuda, but still very very slow. print(torch.cuda.current_device(), |
|
Hi Chris, do you know what is the normal speed of running the eval baseline per episode? Thanks! |
|
Same problem here using 8 RTX 3090. In the config GPU 1-7 are used as simulators but nor GPU memory or usage are observed. Seems the simulator is running on CPU. |
Hi, in my machine, I use a P5000 with 16384MiB. I run the code of ovmm projects/habitat_ovmm/eval_baselines_agent.py, but is very slow. It takes 30 min per episode. Is it normal? Thanks.
The text was updated successfully, but these errors were encountered: