Dataset appears invalid after push to local MinIO repo

[sergiimk]

After successfully pushing a dataset via SmTP to kamu node, the dataset cannot be viewed in Web UI and appears to be in invalid state.

Steps to reproduce

1. Checkout https://github.com/kamu-data/kamu-deploy-example repo.
2. Update environments/minikube/environment.yaml to disable Jupyter and Superset - we don't need them
3. Setup the node:

minikube start --cpus no-limit --memory no-limit
./sync.sh
./port-forward.sh

4. In another console run:

kamu init

# Login with Github account
kamu login http://localhost:4200

# Just getting any example dataset
kamu pull "s3://datasets.kamu.dev/odf/v2/example/ca.bankofcanada.exchange-rates.daily" --no-alias

# Observe data is present
kamu tail ca.bankofcanada.exchange-rates.daily

# Push dataset to local node (update with your minikube IP and github name)
kamu push ca.bankofcanada.exchange-rates.daily --to odf+http://192.168.49.2:30201/sergiimk/ca.bankofcanada.exchange-rates.daily

5. Open http://localhost:4200/
6. Attempt to see the dataset
7. Observe the error in GQL response:

errorKind: "INTERNAL_ERROR"
errorMessage: "Object Store error: Object at location fed01ee573bae82531d0abd13d6fc0ce943419261102bfd722e6eb71f3330734dff52/data/f1620200b3c622f9eb6b1a17e5425c06a631e8b9775070c361b97e07ddafb84858e7d not found: Client error with status 404 Not Found: No Body"

8. Inspect the minio bucket:

AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=password123 aws --endpoint-url http://192.168.49.2:30401 s3 ls "s3://datasets/fed01ee573bae82531d0abd13d6fc0ce943419261102bfd722e6eb71f3330734dff52/"

9. Observe that the data/ is completely missing - the dataset push resulted in invalid state!

Expectations

• Web UI
  • does not truncate an error
  • makes error easy to copy
• Node
  • Investigate
    • why our tests are not catching this error
    • why push returns a successful response while data is not pushed
  • Ensure error handling is sound
  • Fix the push logic itself
• The fix is reflected in https://github.com/kamu-data/kamu-deploy-example repo

[s373r]

The bug report is closed as the issues on the kamu-deploy-example side have been fixed.

Several tasks are extracted as stand-alone tasks:

• S3: configurable server proxy mode kamu-cli#909
• Easy-to-copy error messages kamu-web-ui#467