I wrote back to the latest thread with a development build that uses GDAL's logging handler instead of the one that we've been using. A big apparent advantage of GDAL's handler is that they've (optionally) enabled their own internal debug logging to be exposed through the python logging, just at the logging.DEBUG level.

Observation: When converting a UTF-16-encoded bytestring to str, you get a similar error.

>>> len('foo'.encode('utf-16').decode('utf-8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The forums posts linked have a variety of different specific exceptions. Maybe this is related to one or more of them?

dba6ab4 tests this suspicion by examining the file encoding of any files provided by the user in an args dict.

After some digging into the python documentation and some trial-and-error, I believe this exception is being caused by how python is handling the user's default locale and filesystem encoding. The solution (PR coming soon) is to force python to use UTF-8 strings. Since GDAL expects UTF-8 filepaths, this is the most natural solution and should guarantee that we are working with UTF-8 paths at all times.

A few further notes from discovery and repair, for future us:

• The user's workspace was in their home directory
• The user's home directory contained the character ö.
• The exception reported (UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xf6 in position 60) matches the byte value for ö in the Latin-1 encoding, so we can assume that the user's locale or filesystem encoding is latin-1.
• Python was reading the user's filesystem encoding (on a Windows computer) as utf-8, so we can conclude that the user's locale is something other than UTF-8, C or POSIX (unlikely because it's a windows computer). The user's locale must therefore be causing some low-level files to be encoded as latin-1, which then causes errors in GDAL which expects UTF-8.
• Forcing the python application to use UTF-8 Mode resolved the issue on the user's computer.

For anyone experiencing this issue on InVEST < 3.13.1 (I'm guessing that this will be the next release), setting the environment variable PYTHONUTF8=1 for all users and then relaunching the workbench should resolve the issue.

This environment variable will be necessary for at least the next 4 years, until Python 3.15 makes UTF-8 mode the default.

OK, I don't have a solution, exactly, but I have been able to reproduce this on a Windows VM using Windows 10 insider edition, from the UK and German versions of Windows. The errors are the same in both, but the UK one is much easier for me to navigate :)

Here's what I've gathered so far:

• In both cases, the python-detected default encoding is cp1252, which is normal for Windows outside the US. The migration to UTF-8 has been slow.
• The root cause of the exception is a UnicodeDecodeError that is functionally equivalent to:

  >>> 'ötzi'.encode('cp1252').decode('utf-8')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 0: invalid start byte
  

• Because the actual reported exception is a SystemError, we can infer that the true exception is being called from the python C API.
• Because this is being reported in our GDAL logging handler, I am inferring that the GDAL SWIG wrapper is mishandling a string.

Furthermore:

• These exceptions only happen when the application has been installed on Windows for this one user. Installing for all users (which puts the application in C:\Program Files) makes the exception go away.
• When InVEST is installed for all users but the workspace is within the user's home directory, the errors are absent. This makes me think that InVEST's paths are fine, and that there's something happening within GDAL causing this error.

Workaround

Install InVEST for all users at install time.

This is on hold because I haven't had a chance to dig into it for a while, and is mitigated by being sure to install InVEST for all users on the computer.

I haven't thought about this issue in about a year, and in that time I seem to have deleted the Windows VM I was using for testing. But looking back on this issue, this is really looking like a core GDAL issue and not something that's strictly wrong with InVEST. If this is true, then we should be able to reproduce this by:

1. Creating a VM using Windows 11 Insider Edition (UK)
2. Creating a user with ö in the username
3. Installing gdal and python in a mamba environment within the user's home directory
4. Trying to ReadAsArray() a raster
5. Expect failure.

If this still fails, it'd be a great minimum, reproducible example for the GDAL devs.