Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish images for service container #15

Open
TonyCTHsu opened this issue Nov 19, 2024 · 3 comments · May be fixed by #34
Open

Publish images for service container #15

TonyCTHsu opened this issue Nov 19, 2024 · 3 comments · May be fixed by #34

Comments

@TonyCTHsu
Copy link
Contributor

Why

Pulling images from ghcr.io is more reliable for automations in Github Actions. It also avoid hitting the rate limit threshold for Dockerhub.

Here is a list of images we used from https://github.com/DataDog/dd-trace-rb

  • datadog/agent
  • ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:v1.12.0
  • elasticsearch:8.1.3
  • memcached:1.5-alpine
  • mongo:3.6
  • mysql:8.0
  • opensearchproject/opensearch:2.8.0
  • postgres:9.6
  • starburstdata/presto:332-e.9
  • redis:6.2
@lloeki
Copy link
Contributor

lloeki commented Nov 20, 2024

Essentially this would make this repo and its ghcr.io/datadog/images-rb registry a mirror, which is fine and in scope.

A corresponding directory structure and image naming scheme should be devised, e.g quick thinking:

src/mirror/
  hub.docker.com/     # is that it? or rather some `docker.io`, or just `docker`, or nothing?
    datadog/
      agent/
        latest/       # last component is the default base tag
          Dockerfile  # can contain `append-tags: 1.2.3`, but then should strip away the `latest` special case tag?
    elasticsearch/
      8.1/
        Dockerfile
    ...
  ghcr.io/
    datadog/
      dd-apm-test-agent/
        v1.12/
          Dockerfile
  ...

Tooling (docker.rake) might need to be slightly updated. CI workflow will probably need an overhaul and made more generic with reusable parts, which is on the table anyway for sustainability.

@TonyCTHsu
Copy link
Contributor Author

TonyCTHsu commented Nov 21, 2024

A corresponding directory structure and image naming scheme should be devised

Do we really need this? I think it is quite straightforward with a YAML as https://github.com/DataDog/images/blob/master/mirror.yaml defined them.

images:
  - source: "docker.io/library/postgres:9.6"
    dest:
      repo: "postgres"
      tag: "9.6"

Not sure how it is done or if we can leverage the existing https://github.com/DataDog/images to do this (we might just need a simplified versions of it).

From what I understand, in order to build a mirror, we need a source and a destination and do something like:

docker pull docker.io/repository/image:tag
docker tag docker.io/repository/image:tag ghcr.io/USERNAME/IMAGE:TAG
docker push ghcr.io/USERNAME/IMAGE:TAG

Here is something interesting about https://github.com/DataDog/images

  • Rakefile to automate docker build
  • mirror.lock.yaml with sha for checksum?
  • Seems only publishing registry.ddbuild.io

@lloeki
Copy link
Contributor

lloeki commented Nov 26, 2024

I think it is quite straightforward with a YAML

images:
  - source: "docker.io/library/postgres:9.6"
    dest:
      repo: "postgres"
      tag: "9.6"

is literally equivalent to:

# src/mirror/postgres/9.6/Dockerfile
FROM postgres:9.6

except the former is is Yet Another Format only consumable by some tool (for which you say "Not sure how it is done"), and the latter is both immediately usable by both our rake task and a raw docker command, and extensible should we need to add a patch or something.

I'm thinking that we shouldn't even try to claim it's a 1:1 mirror† image: all a consumer should care about might be what the image provides.

† hence the point of having the source in the path otherwise a consumer can't distinguish between docker.io/library/postgres (implicitly postgres), some.other/place/postgres and yet.another/place/postgres.

@lloeki lloeki linked a pull request Nov 26, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants