Facial recognition and head coverings
Facebook’s Casual Conversations dataset
Facebook recently published “Casual Conversations”- 45,000 short videos with paid actors as a new benchmark with diversity along speakers’ age, gender, and skin color (Facebook classifies these speakers on the Fitzpatrick scale rather than race and ethnicity).
While downloading I was listening to a NewNaratif podcast about hijab rules in Singapore, and wondered whether Facebook included any hijab-wearing women in a diversity dataset. To make the question applicable to a wider audience, my research question is: how many speakers in the dataset wore a hat or head covering? What types are included in the dataset?
My thought is that hats can confuse facial recognition software, and you might want to know whether the dataset includes men and women wearing a variety of hats.
Facebook’s paper does not mention ‘hat’, ‘hair’, ‘covering’, or ‘hijab’, even though two hats and a hair accessory are visible in their announcement:
I asked Cristian Canton, Head of Facebook’s AI Red Team for more info.
I downloaded 3 parts of the dataset (out of 76 parts, 90-100GB each). Of the 116 speakers, four women had a head wrap, and two women with partial hair cover appear in the two-person scenes.
As far as I know, Facebook made no restrictions on head coverings:
Does this data exist somewhere else?
On hats: you can search for datasets including occlusion or disguised / costumed faces. Due to the sensitivity of face images, especially faces scraped from the internet, you may need an institution or legal representative to sign for access to a dataset.
On masks: obviously there’s been increased interest in facial recognition and masks since 2020. One of the first large datasets appeared in late March 2020 github.com/X-zhangyang/Real-World-Masked-Face-Dataset
On hijabs and other Muslim women’s wear: Universiti Teknologi Malaysia researchers published Occluded Face Detection, Face in Niqab Dataset in 2019. I haven’t been able to access this paper.
Researchers at Khatam University and Sharif University of Technology in Tehran published Partially Covered Face Detection in Presence of Headscarf for Surveillance Applications based on office security videos.
Universiti Malaysia researchers posted Skin Color Pixel Classification for Face Detection with Hijab and Niqab in 2017, which is about isolating the face pixels when someone wears a neutral or ‘skin color’ covering.
Updates?
This article was posted in April 2021. For any updates on head coverings in facial recognition, see this GitHub Readme.
