Object counting in Elixir with ML

[ndrean]

I want to go further with Elixir and ML and start a repo where you do real-time object counting in Elixir.

What I understood:

1. firstly use a convolution neural-network (like almost all image detection model),. The YOLOv8 model for object detection. This model seems the current state-of-the-art model. It is . You apply it to each selected video frame.
2. then we want to estimate how the detected object is progated to the next frame. You may use a Kalman filter or linear velocity.
3. it is then followed by a "reconciliation" process where the futur position - as computed from the previous frame - is compared to the current detection. It uses the Mahalanodis distance and the cosine distance for appearance similarity to discriminate targets. This is know an DeepSORT.

For the moment, all this remains largely unclear. I have no idea how how to use YOLO in Elixir, how to define boxed around detected objects (Javascript?), nor if the Kalman, Mahalanodis algorithms are implemented in Elixir and how to handle and link the data and state.

[LuchoTurtle]

This seems like an interesting idea that I also found nowhere in Elixirland.
It seems that real-time counting (either be it through a video file or webcam) is really facilitated when using Python, for example (https://github.com/afsanamimii/Real-Time-Object-Counting).

The closest I found was from this video -> https://www.youtube.com/watch?v=SfnbE51wRII.
They use https://github.com/cocoa-xu/evision, a binder from OpenCV to Elixir. Maybe it's a good starting point?

[ndrean]

Ok, of course some bibliographic work before. I continue the first observation round 😀

Firstly, a blog post than nicely explains what I was thinking of

Then what I found for Elixir:

• blog post Object detection with Yolo
• YT video object detection with Yolo
• your video -> Evision

Then some first thoughts.

For the video source, I will keep things simple: just be able to process an archived video. The ultimate extension to handle any live stream would probably mean using webRTC but I would keep this for the desert.

Image manipulation, you have a few libraries:

• evision niffing OpenCV,
• vix niffing libvips,
• Image, wrapping Vix,
• stb_image.

Who does what and is good at?
I want to capture a video stream , extract frames - say 1 out of N or every t ms - and produce an Nx tensor from a frame ... to be able to run the object detection and object tracking calculations. But can we run such calculations so fast??

Then if we use any video device, then we probably need this to happen server side and not only use requestVideoframeCallback in a browser context. Evision seems a good candidate.

A Livebook is probably the best format to experiment this. TODO

[ndrean]

I document the slow progress while trying to follow this video.

RUST preamble.

I had to remove the current RUST I had installed via brew and used the recommended RUST installer. This can save you some time....

Export model to ONNX

From the YOLO family, I just picked the first pre-trained model from the YOLOv8 family: YOLOv8n.

If I understood correctly, you have the ONNX model and the ONNX runtime. The first can be transcribed via AxonOnnx whilst the second run via Ortex. It seems (?) much easier to produce an ONNX runtime and run it via ortex than to produce an ONNX format from a - say- PyTorch model and load it into Axon.

You can use a Python library to export the pre-trained model (.pt) into ONNX format.

yolo export model=yolo8n.pt format=onnx imgsz=640

All this is just to catch-up the first minute of the video. I now have a "yolonv8.onnx" file.
Now we can start and see what we can do with this.

Load the model In a Livebook.

• I firstly used AxonOnnx as shown in the video. Note that you need to get "main" from github. However, the instruction AxonOnnx.import(path_to_onnx) failed because of "unsupported "Resize".
• I then added the ortex deps. => Unable to compile ortex in a Livebook due to the mentioned issue.

Conclusion: NaN

Load via a Mix project.

• I add the ortex package. I modify the "deps/ortex/lib/ortex/util.ex" module as per the previous issue. This compiles and Ortex.load("yolov8n.onnx") seems to load a model. Note that default size is 640x640. I can even Nx.Serving it. OK.
• AxonOnnx also fails for the same reason. The model seems to expect 640x640 though...!??

Evision

Leave the default flag EVISION_PREFER_PRECOMPILED=true, otherwise it does not compile in a Livebook.

First conclusion

I can't AxonOnnx an ONNX model (?) but I can Ortex it. However, he seems able to do this in the video....

YOLOv8 License

This is why I previously (#111) asked some questions about licenses.
It is distributed under the GNU Affero General Public License v3.0: https://opensource.org/license/agpl-v3

The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version.

BibTeX citation

@software{yolov8_ultralytics,
author = {Glenn Jocher and Ayush Chaurasia and Jing Qiu},
title = {Ultralytics YOLOv8},
version = {8.0.0},
year = {2023},
url = {https://github.com/ultralytics/ultralytics},
orcid = {0000-0001-5950-6979, 0000-0002-7603-6750, 0000-0003-3783-7069},
license = {AGPL-3.0}
}

[ndrean]

Update on one (minor?) point: I am able to run WebRTC with an Elixir Channel. 🚀

What's the point? Well, it is not obvious at all to do 🥵, at least for me.

but most importantly, WebRTC streams images, can share screens, so coupled with Evision, this is a first step to progress as now I have material to test the rest.

It could be interesting to have a specific repo in DWYL on how to do this. Below is the repo. It may be need a bit more Dwyl style for the Readme.
I will maybe try to test this on Fly, and I guess I will be disappointed for STUN/TURN reasons, but let's see.

https://github.com/ndrean/RTC