Table of Contents
If the Wii taught us anything, it’s that flailing your arms around is fun. So, we’ve built a pipeline that tracks your hand and arm movements in real-time, converting your moving limbs into precise game controls!
This project is not only a fun way to play games, but it also has a wide range of applications in various fields:
-
🎨 Gesture-controlled digital painting or air-drawing.
-
🧠 Gamifying controlled arm and hand movements are great for physical therapy exercises, especially in patients recovering from injuries.
-
🤖 Control robots or drones with the flick of a wrist!
Why Osu! though?
Because if our algorithm can handle tracking hyper-precise arm movements at ridiculous speeds, it can handle pretty much anything. Also, Osu! is a fun game that we're all abysmal at, so we figured we'd make it a little easier for ourselves.
To train our model, we used Roboflow, a platform that allows us to easily train models on custom datasets. We combined two datasets hosted on Roboflow, the Palm Detection Dataset and a Hand Pose Dataset, to create a dataset that could be used to train a model to detect hands and their poses, specifically fists and open palms. We then used Roboflow to train a model on this dataset, achieving an mAP of 0.995 over 200 epochs.
We used this model to construct a custom workflow that could be used to detect hands and their poses in real-time. This workflow was then integrated into a Streamlit web app that could be used to play osu! with hand gestures.
We used Streamlit to provide users with an easy way to launch and interact with our application, offering a clear and elegant showcase of our project. Streamlit enabled us to seamlessly present key features, our team, tech stack, gameplay instructions, and data visualizations that were critical in selecting the best model version. Its simplicity allowed us to create a clean and intuitive UI, ensuring a smooth user experience while maintaining a professional and polished look.
To determine the best version of our model, we conducted a comprehensive data collection process and leveraged Matlab for in-depth analysis. This allowed us to rigorously compare performance metrics across different iterations, ensuring that we selected the optimal model with confidence while minimizing lag. Our use of Matlab's powerful analytical tools enabled us to gain precise insights, driving the accuracy and reliability of our final choice.
To optimize code performance while ensuring high model confidence, we developed and tested various attributes across multiple models. These optimizations included frame splitting, which allowed us to evaluate specific intervals for more efficient resource usage, and smoothing techniques to gradually adjust movements, minimizing abrupt or jittery transitions. We also incorporated calibration to fine-tune sensitivity, enhancing user experience and reducing the need for excessive movement. Additionally, we implemented threading to process frames concurrently, significantly improving both performance and responsiveness.
Cindy Li (autoclicker, model analytics, optimizations) - [email protected]
Cindy Yang (Roboflow workflow, model integration, tuning) - [email protected]
Elise Zhu (CV pipeline, Streamlit app, design) - [email protected]
Selina Sun (data generation, model training, Streamlit app) - [email protected]