Can't load glTF URIs with unicode characters

[lilleyse]

According to the glTF URIs section:

Paths with non-ASCII characters MAY be written as-is, with JSON string escaping, or with percent-encoding; all these options are valid. For example, the following three paths point to the same resource:

{
   "images": [
       {
          "uri": "grande_sphère.png"
       },
       {
           "uri": "grande_sph\u00E8re.png"
       },
       {
           "uri": "grande_sph%C3%A8re.png"
       }
   ]
}

But when I try to use GltfReader to load a glTF with a URI to 🐶.bin I get a runtime error:

std::runtime_error caught: GET `🐶.bin` failed: Couldn't resolve host name

🐶.gltf.zip

This may be a limitation in uriparser which only parses RFC 3986 URIs which are ASCII-based. It does not handle RFC 3987 IRIs which allow Unicode characters verbatim.

[kring]

Your explanation makes sense to me @lilleyse. I wonder if there's a better library for our purposes than uriparser. I couldn't find one when I originally selected it for cesium-native, but that was over four years ago now.

[azrogers]

I've investigated a few alternatives to uriparser that each come with their own drawbacks:

• skyr-url is a more modern C++ URL parser as compared to uriparser's C API. It implements the WhatWG URL specification, which covers both RFC 3986 and RFC 3987 URLs. However, skyr-url seems to be unmaintained (last commit December 2023, compared to yesterday for uriparser). It also fails for URLs that uriparser can parse. For example, one of the test cases includes the URLs "test" and "buffer". skyr-url, perhaps correctly, believes these URLs to be invalid. We could probably work around this somehow, but that's a pain.
• uriparser has wchar_t equivalents to all of its methods - uriParseSingleUriW instead of uriParseSingleUriA, etc. But not only would we need to convert to and from UTF-16 in every method call, copying the string multiple times (and how do we convert when the string/wstring conversion classes in C++ are deprecated?) - but I'm not even sure it would support RFC 3987 if we used the wide equivalents.
• ada seems to be the current state-of-the-art in URL parsing. It supports the WhatWG specification and unicode, and is the URL parser used by Node.js among others. However, it seems to do things just slightly differently. Take a look at some of the failed test cases in the branch I pushed where I was trying it out. This is after a number of workarounds I implemented to try to get the tests passing. Some of these will probably not cause much of an issue - the difference between https://example.com/?some=parameter and https://example.com?some=parameter is probably meaningless to most web servers - but some of them, like not escaping colons in the pathname, could be a big problem.

Also, perhaps the biggest issue, neither alternative to uriparser has equivalents to uriUriStringToWindowsFilenameA, uriUnixFilenameToUriStringA, etc. These methods would have to be reproduced ourselves. They're not too complicated - about fifty or sixty lines of C - but it's yet another headache to deal with that makes switching just a bit more tricky.

Instead of an alternative parser, I'm going to try to see if I can work around the lack of unicode support in uriparser. While 🐶.bin might not parse, %f0%9f%90%b6.bin probably will.

[kring]

Thanks for investigating @azrogers. ada looks really promising, and may be worth the switch even if it causes some pain. There is a lot of dodgy stuff in our Uri.cpp due to limitations in uriparser or just general incompleteness. For example, take a look at the TODO and terrible implementation of addQuery, even though we use it all over the place. I'm inclined to think that test failures caused by switching are much more likely to be bad tests than problems with ada.

The lack of the filename conversion functions would indeed be a hassle, but I suspect they're actually quite simple when based on a URL library with a less old-school interface.

Considering how fundamental URLs are to what we do, I think this is well worth another day or three of effort. Do you think making the switch to ada is possible in that kind of time frame?

[azrogers]

It should be, as long as we're ok with changing around some of the tests where necessary. I can put some more time into it today once I finish catching up on reviews.