Could you provide a way to run with multiple GPUs?

[FranciscoPark]

export OMP_NUM_THREADS=16 export CUDA_VISIBLE_DEVICES=0,1,2,3 python run_src/do_generate.py \ --model_ckpt meta-llama/Llama-3.1-8B \ --dataset_name GSM8K \ --note tensor_parallelism \ --num_rollouts 16 \ --api vllm \ --model_parallel \ --tensor_parallel_size 4

Im trying to use vllm with 4GPUs. Is there anything else I should change?

[LordEdison]

same problem😭