ML Generation on Output

[tastyfrankfurt]

Hey Guys,

Just wondering if anyone has looked into including ML pipeline to output models for querying the data. This would be really good to do as some of the wildcard searching can lead to very expensive searches on platforms. Anyway keen to here if this is on the road map or something we could look at putting in, or allowing someway to plugin to existing outputs to support this???? Good example would be say the OpenSearch output.

Thanks

Jail

[nasbench]

I guess this is more a question for the sigma library pySigma. I'll transfer it there.

In the meantime can you elaborate more on this with maybe just a rough example?

I'll cc @thomaspatzke maybe he is interested in such a thing.

[thomaspatzke]

There was a similar discussion some time ago and the result was, that in contrast do pattern matching, expression of ML models in a generic way is quite hard due to the different capabilities of the backend systems that finally should do the search. In the end the idea was discarded as too challenging and too biased towards specific implementations to be a good fit in Sigma.

Nevertheless, I'm always open for good ideas to implement it. As usual in Sigma, a solution must not aim 100% coverage of a use cases someone could imagine. Restricting on a specific part could possibly solved in a good way and provide value for everyone.

[tastyfrankfurt]

Yes agreed, there are many implementations and very technical details to do so, perhaps a similar approach like the 'pySigma Backend Cookiecutter Template' could be useful here, that way people could feed the pipeline data into a module that has been built run ML pipeline processes. 2 i can think of would be opensearch, elasticsearch and spark. again the consumer of the code would need the compute in order for the ML development to occur. Sorry if that doesnt make sense and again might be putting that into the hard problem to solve bucket.

Thanks for your answer above.