As a user, I want the PDS4 properties to be case agnositics

[tloubrieu-jpl]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

API user

💪 Motivation

...so that I can handle the case anyhow and avoid having silent errors when the case is not right.

📖 Additional Details

Actually this is a bug in the current API version where the case is turn to lower case mostly but the q= critiria need to have the exact PDS4 . notation case. I believe it used to work but I don't remember how.

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

No response

[al-niessner]

@tloubrieu-jpl

There is a utility function (model.SearchUtil) that converts . to _ hide the elasticsearch specialness of . -- I do not know if opensearch has the same specialness but given its roots probably. Another utility function does the reverse hiding (mostly) the . and _ translations from the end user.

These functions do not change the case of what is being searched. Part of the reason why is that if we lower cased everything from the user, then the db would have to all be lower case too (opensearch is not case agnostic when it comes to field names). If they were all set to lower case in opensearch, then the user would get them returned as lower case. As I remember that was an undesirable consequence. If you had a table of correct casing for each all lower case field names, it could be undone. However who and how are you maintaining that table? You also have to fix the problem of coexisting field names such as Document and document.

All that said, if it is possible to make opensearch case agnostic to field name case via a global configuration, then that should be done. When searching, case sensitivity of the value can be set but not the field name.

Another option would be to use an edit estimate to get it to select a near field given the edit distance - another way to correct case on the fly -- but it will also change 'cace' to 'case' which may not be desirable or have unwelcome consequences like a successful return when the user was expecting an error.

[tloubrieu-jpl]

Thanks @al-niessner , I believe this worked before. I don't know when and how we lost this feature, maybe when we switched to opensearch... Certainly we did not have to maintain a table to match case sensible properties with lower case properties, and I think I would rather give up that requirement than doing that.

[tloubrieu-jpl]

It might also be that harvest was loading the fields in lower case and stop doing that at some point. Or I just dreamed but the reason I am thinking it used to work is that the notebook I was testing the API with, worked 1 or 2 years ago, then worked again recently except for the case of the properties in the q parameter.

[al-niessner]

@tloubrieu-jpl

Not hallucinating (proof you are not an AI bot). We did make changes for a short time to have the fields be case agnostic but changed back because of the problem of returning wrong case to user in JSON response because all lower case in elasticsearch at the time. I think we did just with the test data at the time. It was short lived. #154 was part of it,

[al-niessner]

@tloubrieu-jpl

May have been changes in how we defined the fields. As text they are case insensitive. Keywords are case sensitive. See:
https://stackoverflow.com/questions/51069213/is-there-a-way-to-make-elasticsearch-case-insensitive-without-altering-the-exist

[tloubrieu-jpl]

Thanks @al-niessner for pointing that ticket #154 out.

I will close this ticket as wontfix, and open 2 other tickets:

• bug on field case in responses field case in response and query have mismatched cases #362
• a could have ticket with the suggestion for fields which are not found but are close to a different field (e.g. with different case): As a user, I want to get an error 400 and relevant suggestions when the properties I am using in the q and fields do not exist in the registry #363