This repo documents my demo project during my stint with the Insight program at Toronto.
LimeLight is a deep-learning based privacy tool that automatically and selectively blurs faces in videos.
Check out LimeLight's website!
Video content creators, such as social media vloggers and news television companies, have to comply with data privacy laws like the General Data Protection Regulation (GDPR) in Europe.
Publishing a video online requires getting consent from members of the public who appear in the video. Alternatively, their faces have to be anonymized manually.
Both of these solutions are impractical and time-consuming.
LimeLight was developed on Python 3.6.
To install all related dependencies:
pip install -r requirements.txt
To use LimeLight from the command line, enter:
python limelight/anonymize_vid.py <path_to_video> --known-faces-loc=<faces_directory>
where
<path_to_video>contains the path to the video to be anonymized<faces_directory>is a directory containing.jpgimages of faces of the subject. These faces will not be blurred.
--dst: Location to save processed video. Defaults to the same folder aspath_to_video.--mark-faces: Mark detected faces with bounding boxes.
Original YouTube source video can be found here.
LimeLight comprises of two main parts, Face Detection and Face Recognition, as shown in the schematic above. For this project, the focus of my work is on improving the Face Detection aspect using deep learning, since the Face Recognition uses a state-of-the-art face embedder and runs relatively quickly on dlib.
A pre-trained RetinaNet model in Keras that has been re-trained to detect faces using transfer learning. The model was re-trained on the WIDER FACE dataset.
Performed using a Siamese Network with FaceNet as the feature extractor / embedder. The face_recognition and dlib modules were used to implement Face Recognition.
Given the use case, failing to detect a face (false negative) is more undesirable than misclassifying an object as a face (false positive). A performant solution detect as many faces as possible and also run close to, if not in, real-time.
As such, the metrics pertinent to this use case are recall and processing speed.
The following image shows three different iterations of my face detection models and their results.
Note: The base model uses the Viola Jones object detection framework, implemented in OpenCV.
Through a series of optimizations, the processing speed of LimeLight's face detection was improved 20-folds to 5.9fps.
The processing speed could be improved further through some optimizations described in the Future Work Section.
The following optimizations could be attempted to improve LimeLight — I might just try them out after the Insight program.
- A triage network — skips face detection if the subsequent frame is similar
- Network pruning
- Network quantization
- WIDER FACE dataset, available at here.
@inproceedings{yang2016wider,
Author = {Yang, Shuo and Luo, Ping and Loy, Chen Change and Tang, Xiaoou},
Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
Title = {WIDER FACE: A Face Detection Benchmark},
Year = {2016}}