Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update install_env.sh of Comfy #1170

Merged
merged 14 commits into from
Dec 27, 2024
Merged

Update install_env.sh of Comfy #1170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
75205c7
Update install_env.sh
jackalcooper Dec 26, 2024
fcb1156
"diffusers>=0.22"
jackalcooper Dec 27, 2024
b8e0fa0
fix nexfort deps
jackalcooper Dec 27, 2024
9c9e698
0.22
jackalcooper Dec 27, 2024
9315b2e
Update examples.yml
jackalcooper Dec 27, 2024
3fe3ce6
rm flash
jackalcooper Dec 27, 2024
f6f2866
try fix
jackalcooper Dec 27, 2024
06e1fa6
print cache dir
jackalcooper Dec 27, 2024
ad385d6
Update examples.yml
jackalcooper Dec 27, 2024
8872c33
fix
jackalcooper Dec 27, 2024
330b314
refine
jackalcooper Dec 27, 2024
7f0fe0e
Update examples.yml
jackalcooper Dec 27, 2024
e04c0ae
Update examples.yml
jackalcooper Dec 27, 2024
80a7ad2
try fix
jackalcooper Dec 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 16 additions & 11 deletions .github/workflows/examples.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,14 +173,14 @@ jobs:
should-run-pro:
- ${{ github.event.pull_request.head.repo.full_name == github.repository || github.event_name == 'schedule' }}
image:
- onediff:cu118
- onediff-pro:cu122
- onediff:torch2.3-cuda11.8
- onediff-pro:torch2.3-cuda12.1
test-suite:
- diffusers_examples
- comfy
exclude:
- should-run-pro: false
image: onediff-pro:cu122
image: onediff-pro:torch2.3-cuda12.1
steps:
- name: Login to ACR with the AccessKey pair
if: github.event.pull_request.head.repo.full_name == github.repository
Expand Down Expand Up @@ -283,25 +283,31 @@ jobs:
- run: nvidia-smi
- run: nvidia-smi -L
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip list
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip cache dir
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m oneflow --doctor
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
- name: Install onediff-quant if needed
if: startsWith(matrix.image, 'onediff-pro')
run: |
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install --pre onediff-quant "numpy<2" -f https://oneflow-pro.oss-cn-beijing.aliyuncs.com/onediff-quant/
- name: Pip Install Requirements for ComfyUI & Test
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install --user "huggingface_hub==0.25.0"
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install -r /src/onediff/onediff_diffusers_extensions/examples/requirements.txt
if: matrix.test-suite == 'diffusers_examples'
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install -r tests/comfyui/requirements.txt --user
if: matrix.test-suite == 'comfy'
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install -r ComfyUI/requirements.txt --user
if: matrix.test-suite == 'comfy'
- run: docker exec ${{ env.CONTAINER_NAME }} python3 -m pip uninstall -y transformer-engine
if: matrix.test-suite == 'comfy'
run: |
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install -r tests/comfyui/requirements.txt --user
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install -r ComfyUI/requirements.txt --user
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip uninstall -y transformer-engine
- name: Start ComfyUI Web Service
if: matrix.test-suite == 'comfy'
run: |
docker exec -w /app/ComfyUI -d ${{ env.CONTAINER_NAME }} sh -c "python3 /app/ComfyUI/main.py --gpu-only --disable-cuda-malloc --port 8188 --extra-model-paths-config /src/onediff/tests/comfyui/extra_model_paths.yaml > /app/ComfyUI/onediff_comfyui.log 2>&1"
sleep 30
# print to check if comfy is launched successfully
- run: docker exec ${{ env.CONTAINER_NAME }} ps aux
- run: docker exec -w /src/onediff/onediff_comfy_nodes/benchmarks ${{ env.CONTAINER_NAME }} bash scripts/install_env.sh /app/ComfyUI
if: matrix.test-suite == 'comfy'
Comment on lines +309 to +310
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

Script permissions need to be updated for execution

The installation script (install_env.sh) exists but lacks execution permissions (current: -rw-r--r--). This needs to be addressed for the workflow to execute properly.

  • File: onediff_comfy_nodes/benchmarks/scripts/install_env.sh needs execution permissions (chmod +x)
  • The script's contents look correct and safe:
    • It properly validates input parameters
    • Creates symbolic links for custom nodes
    • Handles CI vs local environment appropriately
    • Installs required dependencies

To fix this, you should add a step to set execution permissions before running the script:

- run: chmod +x onediff_comfy_nodes/benchmarks/scripts/install_env.sh
- run: docker exec -w /src/onediff/onediff_comfy_nodes/benchmarks ${{ env.CONTAINER_NAME }} bash scripts/install_env.sh /app/ComfyUI
🔗 Analysis chain

Verify installation script execution

The installation script is being executed in the ComfyUI environment. Let's verify its contents and execution permissions.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check if the installation script exists and has proper permissions
ls -l onediff_comfy_nodes/benchmarks/scripts/install_env.sh

# Check the script's contents for potential issues
cat onediff_comfy_nodes/benchmarks/scripts/install_env.sh

Length of output: 1229

- name: Test ComfyUI
if: matrix.test-suite == 'comfy'
run: |
Expand All @@ -317,7 +323,6 @@ jobs:
# run_comfy_test "/share_nfs/hf_models/comfyui_resources/workflows/deep-cache.json" 600
# run_comfy_test "/share_nfs/hf_models/comfyui_resources/workflows/deep-cache-with-lora.json" 800
# run_comfy_test "workflows/text-to-video-speedup.json" 5000
docker exec -w /src/onediff/onediff_comfy_nodes/benchmarks ${{ env.CONTAINER_NAME }} bash scripts/install_env.sh /app/ComfyUI
docker exec -w /src/onediff/onediff_comfy_nodes/benchmarks ${{ env.CONTAINER_NAME }} bash scripts/run_all_tests.sh || {
echo "Test fails! print the ComfyUI logs..."
docker exec onediff-test cat /app/ComfyUI/onediff_comfyui.log
Expand Down Expand Up @@ -355,15 +360,15 @@ jobs:
run: docker exec -w /src/onediff/onediff_diffusers_extensions ${{ env.CONTAINER_NAME }} bash examples/unet_save_and_load.sh --model_id=/share_nfs/hf_models/stable-diffusion-xl-base-1.0
- if: matrix.test-suite == 'diffusers_examples'
run: |
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install --user scikit-image "numpy<2"
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install --user scikit-image "numpy<2" "diffusers==0.23" peft "huggingface_hub==0.25.0" "transformers>=4.28.1"
docker exec -e ONEFLOW_MLIR_ENABLE_INFERENCE_OPTIMIZATION=0 ${{ env.CONTAINER_NAME }} python3 -m pytest -v onediff_diffusers_extensions/tests/test_lora.py --disable-warnings

- name: Install Requirements for WebUI
if: matrix.test-suite == 'webui'
run: |
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip config set global.extra-index-url https://pypi.tuna.tsinghua.edu.cn/simple
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install --user pytorch-lightning gradio==3.41.2 diskcache gitpython pytorch_lightning==1.9.4 scikit-image jsonmerge pillow-avif-plugin torchdiffeq torchsde clean-fid resize-right lark tomesd blendmodes facexlib opencv-python==4.8.0.74 piexif inflection ftfy regex tqdm pydantic==1.10.13
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip install --user pytorch-lightning gradio==3.41.2 diskcache gitpython pytorch_lightning==1.9.4 scikit-image jsonmerge pillow-avif-plugin torchdiffeq torchsde clean-fid resize-right lark tomesd blendmodes facexlib opencv-python==4.8.0.74 opencv-python-headless piexif inflection ftfy regex tqdm pydantic==1.10.13
docker exec ${{ env.CONTAINER_NAME }} python3 -m pip uninstall -y transformer-engine

- name: Prepare environment for WebUI
Expand Down
7 changes: 3 additions & 4 deletions onediff_comfy_nodes/benchmarks/scripts/install_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,9 @@ if [ "$CI" = "1" ]; then
echo "Detected CI environment. Skipping local environment-specific dependencies."
else
echo "Detected local environment. Installing local environment-specific dependencies."
pip install -r $CUSTOM_NODES/ComfyUI_InstantID/requirements.txt
pip install -r $CUSTOM_NODES/PuLID_ComfyUI/requirements.txt
python3 -m pip install --user -r $CUSTOM_NODES/ComfyUI_InstantID/requirements.txt
python3 -m pip install --user -r $CUSTOM_NODES/PuLID_ComfyUI/requirements.txt
Comment on lines +25 to +26
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add error handling for requirements installation.

The requirements installation commands should include error handling to ensure the script fails gracefully if the requirements files are missing or if installations fail.

Consider this improvement:

-  python3 -m pip install --user -r $CUSTOM_NODES/ComfyUI_InstantID/requirements.txt
-  python3 -m pip install --user -r $CUSTOM_NODES/PuLID_ComfyUI/requirements.txt
+  for node in "ComfyUI_InstantID" "PuLID_ComfyUI"; do
+    req_file="$CUSTOM_NODES/$node/requirements.txt"
+    if [ -f "$req_file" ]; then
+      echo "Installing requirements for $node..."
+      python3 -m pip install --user -r "$req_file" || {
+        echo "Failed to install requirements for $node"
+        exit 1
+      }
+    else
+      echo "Warning: Requirements file not found for $node"
+    fi
+  done
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
python3 -m pip install --user -r $CUSTOM_NODES/ComfyUI_InstantID/requirements.txt
python3 -m pip install --user -r $CUSTOM_NODES/PuLID_ComfyUI/requirements.txt
for node in "ComfyUI_InstantID" "PuLID_ComfyUI"; do
req_file="$CUSTOM_NODES/$node/requirements.txt"
if [ -f "$req_file" ]; then
echo "Installing requirements for $node..."
python3 -m pip install --user -r "$req_file" || {
echo "Failed to install requirements for $node"
exit 1
}
else
echo "Warning: Requirements file not found for $node"
fi
done

fi

echo "Installing common dependencies..."
pip install websocket-client==1.8.0 numpy==1.26.4 scikit-image -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install nexfort
python3 -m pip install --user nexfort websocket-client==1.8.0 numpy==1.26.4 scikit-image
1 change: 1 addition & 0 deletions onediff_diffusers_extensions/examples/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ diffusers[torch]==0.19.3
onediff
chardet
opencv-python==4.8.0.74
opencv-python-headless
2 changes: 0 additions & 2 deletions tests/diffusers-docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,6 @@ services:
volumes:
- $HOME/test-container-cache-${CONTAINER_NAME}/dot-cache:/root/.cache
- /share_nfs:/share_nfs:ro
- ${SDXL_BASE}:/app/ComfyUI/models/checkpoints/sd_xl_base_1.0.safetensors:ro
- ${UNET_INT8}:/app/ComfyUI/models/unet_int8/unet_int8:ro
- $PWD:/src/onediff
working_dir: /src/onediff
restart: "no"
Loading