Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with OVMM result reproduction #524

Open
Abc12128 opened this issue Nov 21, 2024 · 0 comments
Open

Problems with OVMM result reproduction #524

Abc12128 opened this issue Nov 21, 2024 · 0 comments

Comments

@Abc12128
Copy link

Hello.
I tried to reproduce the results from the original paper on my server and got the following data.
Averaged metrics
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
episode_count: 1199.0
does_want_terminate: 0.4061718098415346
num_steps: 874.3436196830693
find_object_phase_success: 0.6688907422852377
pick_object_phase_success: 0.6205170975813178
find_recep_phase_success: 0.5187656380316931
overall_success: 0.14595496246872394
partial_success: 0.48853211009174313
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
100% | █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ █ | 1199/1199 [12:14:56<00:00, 36.78s/it]
Metrics:
{'episode_count': 1199.0, 'does_want_terminate': 0.4061718098415346, 'num_steps': 874.3436196830693, 'find_object_phase_success': 0.6688907422852377, 'pick_object_phase_success': 0.6205170975813178, 'find_recep_phase_success': 0.5187656380316931, 'overall_success': 0.14595496246872394, 'partial_success': 0.48853211009174313}
But in my results, overall_success is always lower than in the original paper (I chose the baseline configuration in the red box below). I changed --baseline_config_path to rl_agent.yaml and GROUND_TRUTH_SEMANTICS:1) in hssd_eval.yaml.
4c46fe3e45bd04594626c38a5df8868
Do you know the cause of this result?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant