Sidecar S3 Configuration "reading meta file failed" error

[berkayalan]

Hi,

I would like to deploy thanos sidecar to ecs as a task. I created an S3 bucket for it : com-thanos-data-bucket .

This is my dockerfile:

FROM thanosio/thanos:v0.37.2

COPY objstore.yml /objstore.yml

USER root

RUN sed -i "s/%%%THANOS_BUCKET_ALIAS%%%/com-thanos-data-bucket /g" /objstore.yml 

ENTRYPOINT ["/bin/thanos", "sidecar"]

CMD ["--prometheus.url=https://prometheus.myendpoint.tech", \
"--objstore.config-file=/objstore.yml", \
"--log.level=debug"]

And this is also my objstore.yml file:

type: S3
config:
  bucket: "%%%THANOS_BUCKET_ALIAS%%%"
  endpoint: "s3.eu-central-1.amazonaws.com"
  region: "eu-central-1"

But I got these errors:

level=warn msg="reading meta file failed, will override it" err="failed to read data/thanos.shipper.json: open data/thanos.shipper.json: no such file or directory"

level=warn err="read dir: open ./data: no such file or directory"

I think my objstore can't connect to S3. What am I doing wrong? I checked my ECS role and it has permission to access S3.

Thanks in advance.

[dosubot[bot]]

Hello, @berkayalan! I'm here to help you out with your bug. Let's squash it together! 🤖

The error messages you're encountering suggest that the Thanos Sidecar is unable to find the thanos.shipper.json file and the ./data directory. Here are some steps to resolve these issues:

1. Ensure the Data Directory Exists: The --tsdb.path flag, which defaults to ./data, specifies where the Thanos Sidecar expects to find the TSDB data and the thanos.shipper.json file. Make sure that this directory exists and is correctly mounted in your ECS task. You might need to create this directory or adjust your ECS task definition to mount the correct volume.

2. Check File Permissions: Ensure that the ECS task has the necessary permissions to read and write to the ./data directory. This includes both the ECS task role and the file system permissions within the container.

3. Verify S3 Configuration: Although the error messages are related to local file paths, double-check your S3 configuration in objstore.yml. Ensure that the bucket name and endpoint are correctly set and that the ECS task has the necessary IAM permissions to access the S3 bucket. The required actions typically include s3:ListBucket, s3:GetObject, s3:DeleteObject, and s3:PutObject ^[1].

4. Debugging: Since you have set the log level to debug, review the logs for any additional information that might indicate other issues with the setup.

By addressing these points, you should be able to resolve the "reading meta file failed" error and ensure that your Thanos Sidecar can connect to the S3 bucket as expected.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}