Shortcutting upload of new container

[ruckc]

So I am trying to splice together new container from existing layers in harbor.

I have a requirement to combine a software container by adding a rather large data layer (30GB) to the software container. Given that the software will change due to software changes, library upgrades, etc... we recognize that the software components will evolve much faster than the data, and we want to avoid transferring and storing the rather large amount of data in each new container.

Ideally, the deployment platforms (docker/kubernetes) only have to pull the updated software layers, but not the data layer. Additionally, when our DevOps builds an updated software container, it doesn't have to download the data layer, assemble the new container, and reupload the same 30gb.

So I've got two containers stored, one with the software and all of it's layers, and one with just a single layer with the 30gb of data.

I am pulling down the two manifests/configs of both containers and splicing them together (appending their layers, history, rootfs). I can upload the config, but the manifest fails because the "blob unknown to registry". I've tried the HEAD endpoint /v2/{image}/blobs/{digest} and it returns 404 unknown.

Is there a way for it to recognize it already has the layer blobs from the other images in the project?