[Algorithm] DPO

[vmoens]

No description provided.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3025

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 7 Unrelated Failures

As of commit 04561bd with merge base 77dbc6c ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t c3f6fe7c25fc2506f819f467cd688e15350f41a048be87d6c166cdc3c2b79306 /exec failed with exit code 1
• Lint / python-source-and-configs / linux-job (gh)
  torchrl/objectives/llm/dpo.py:8:1: F401 'warnings' imported but unused
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 4da82b6df46b0e95de1d3e7f9c600e580e9226cd76a2039f63245b0d09710cbc /exec failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh) (similar failure)

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Continuous Benchmark / CPU Pytest benchmark (gh) (trunk failure)
• Continuous Benchmark / GPU Pytest benchmark (gh) (trunk failure)
  RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor []] is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh) (trunk failure)
  test/test_libs.py::TestGym::test_gym_fake_td[True-False-3-HalfCheetah-v2]
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestActionDiscretizer::test_transform_env[cheetah-SamplingStrategy.RANDOM-False-True]
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.