Problem downloading from prod-lads S3 bucket

[bertcoerver]

Hello,

I'm not sure if this is the right place for this question, if so please point me to a better place to ask it.

Until a few days ago, this script below was working fine. Now I'm getting a "Acces Denied" error. The script first requests a token to access S3 (similar to visiting https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials in a browser). That seems to be working fine. Then I set up GDAL to ingest the token and in this example I finally try to get some metadata info from the file (using gdal.Info).

Any ideas why this is not working (anymore) or where I can find more information on why I'm being "Denied Access"? I'm located in Europe (does that matter?).

Thanks!

import requests
import base64
from osgeo import gdal
gdal.UseExceptions()

def get_token(un, pw):

    s3_endpoint = "https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials"
    
    login_resp = requests.get(s3_endpoint, allow_redirects=False)
    login_resp.raise_for_status()

    auth = f"{un}:{pw}"
    encoded_auth  = base64.b64encode(auth.encode('ascii'))

    auth_redirect = requests.post(
        login_resp.headers['location'],
        data = {"credentials": encoded_auth},
        headers= {"Origin": s3_endpoint},
        allow_redirects=False
    )
    auth_redirect.raise_for_status()

    final = requests.get(auth_redirect.headers['location'], allow_redirects=False)
    results = requests.get(s3_endpoint, cookies={'accessToken': final.cookies['accessToken']})
    results.raise_for_status()

    return results.json()

test1 = '/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2022061.1130.002.2022269172704.nc'
test2 = '/vsis3/prod-lads/VNP03IMG/VNP03IMG.A2022061.1130.002.2022061184041.nc'

un, pw = <my earthdata login>
creds = get_token(un, pw)

gdal_config_options = {
    "AWS_ACCESS_KEY_ID": creds["accessKeyId"],
    "AWS_SESSION_TOKEN": creds["sessionToken"],
    "AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
}

for k, v in gdal_config_options.items():
    gdal.SetConfigOption(k, v)

info = gdal.Info(test1)

Some more info on the error

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>43WPRPC643QM199Z</RequestId>
<HostId>
wRyoUYOZ56804o9TXxdh/+QyWLf1C16QkICGZk5A1P5RFOXrka7HFtG0RmqsL1dENezqyv9h9sw=
</HostId>
</Error>

[jhkennedy]

@bertcoerver S3 Direct access only works in-region, so your code needs to be running in AWS in the same region as the data, which is typically the us-west-2 (Oregon) region:
https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentialsREADME

Are you running the code in us-west-2? You can hit the credentials endpoint from anywhere and get the access keys, but when you try and access the data out of region, you'll get that access denied error.

[bertcoerver]

Hi @jhkennedy, thanks for your reply! Thats very strange, because I’ve been running my code locally (on my laptop), from France, without any VPN, for about the last ~10 months I think.

So that was not supposed to work?

Guess I’ll have to look into other options then, like this opendap or something.