Sorting on additional events attributes

[dennisoelkers]

Events should be sortable on additional attributes, which do not allow sorting yet:

On regular events page:

• Description
• Event Definition
• Aggregation Time Range?

On security events page:

• Description
• Type

[dennisoelkers]

Sorting based on Event Definition is most probably not possible with pagination. The event definition of an event is stored in MongoDB, while the event itself is stored in OpenSearch. We would need to join the two, which does not work with pagination.

[dennisoelkers]

Sorting based on Description is not possible out of the box, because the message field is text and not keyword by default (this means it is optimized for full text search instead of sorting/aggregating). We could enable fielddata on it, but that would increase memory usage significantly.

[Emy-01]

@dennisoelkers would we still need pagination if we limit the results so that we only show a defined number of messages? I think this was a conversation you had with @tellistone.
If we do that, what’s the ideal number of messages we could show to allow the sorting?
Also, in case we do an asynchronous job -and not sure if this is right but- wouldn’t this mean that at the end we get already all the results and therefore we wouldn’t need to perform additional fetching?

[coffee-squirrel]

Related: #17437

[dennisoelkers]

@Emy-01: I am not sure I get the questions correctly. We always need to some kind of pagination, because we don't know how many events we will get back. Under any circumstance we can assume that there will be more than 25/50/100 events in the system, so optimizing for the corner case that there is only a single page of events in order to be able to sort in-memory based on the event definition title does not appear to useful here.
Also, what do you mean with asynchronous job? We are not doing anything asynchronous here when fetching events.