Copy does not multi thread blob container creation

[TylerBurnett]

I am working on a use case where I need to replicate 75'000 containers that contain data from one blob storage account to another.
When trying to do this copy, the scanning itself (Which I have noticed to include pre-creating the containers on the dest) takes upwards of 70 hours. Watching the logs, I have noticed there is no threading that occurs in the scanning phase for containers which accounts for this massive slowdown.

Copying the data itself after the scan only takes 20 minutes. Are there any alternatives or settings I could try to speed this up?

[dphulkar-msft]

Hello @TylerBurnett ,

You can try using the below alternative approaches.

1. **Pre-create the Containers: ** Utilize a custom script or tool (e.g., Azure CLI) to create all 75,000 destination containers in parallel before initiating the data copy. This can help eliminate delays caused by container creation during the transfer process.

2. **Copy Containers Individually: ** Develop a script to copy data from one container at a time using AzCopy.

[TylerBurnett]

Thanks for ideas, I ended up going with a concept similar to what you've noted where I parallelized the individual container copies. That said this comes with a variety of drawbacks which are not as ideal.

Is there any chance, this could be added to the backlog for prioritisation? I know this isn't the first or last time someone will encounter this issue.

Thanks,
Tyler

[dphulkar-msft]

Could you share any challenges or drawbacks you encountered while copying individual containers?
Your insights could be valuable for us.