Facial recognition (FR) is a controversial technology. People's privacy, attribution & freedoms are often questions when FR is advertised as being used to identify someone, such as transiting through airports, online via social media, or CCTV tracking at venues. However, like any technology, the intent is what really matters as this technology is publically available for developers to build solutions around.
In this article, we're going to focus on how FR can be used to support missing person investigations. Trace Labs (https://www.tracelabs.org), NCPTF (https://ncptf.org) & many other organizations are doing amazing work around the world to help law enforcement & families, and often during missing person investigations an investigator may come across multiple profiles, sometimes dozens and photos that you think are associated with your subject but you are uncertain if it really is the subject. This article focuses on simple methods to support the verification process.
This is where FR comes into play. We can leverage artificial intelligence to support our investigations using simple, easy to use & free tools. To be clear, we are not trying to use FR for scale & reverse locating other profiles here, such as tools like Primeyes, we are using it specifically to support positive identification of a subject against another image.
Why?
Verification. Providing leads for law enforcement as part of crowdsourced information, or for law enforcement specifically who are tasked with conducting the investigations themselves, verification is always of extreme importance. We can use technology to support that verification process. Importantly, this is only a very small part of the process, but it can create efficiencies when trying to positively identify a subject against a social media profile.
Tools
Amongst others, Google, Amazon & Microsoft all offer free tools to "sample" FR technology. These tools are enough for us to achieve our tasks that we're covering in this article. However, they differ in capabilities. For example, Google performs "Face Detection" but Amazon & Microsoft provide "Face Comparison" capabilities. This is important to note as we are focusing on facial comparisons.
Microsoft Cognitive Services
The first tool we have available is the Microsoft Face Verification tool. Available here: https://azure.microsoft.com/en-au/services/cognitive-services/face/#demo
As you can see, the tool is very user friendly & provides a simple way to compare images to provide a confidence score. The benefit of Microsoft's tool is that it does not require a login to access.
Amazon Rekognition
To access the Amazon technology, you will need to login to the AWS Console (after creating an account) at https://aws.amazon.com/console then search for "Rekognition" & access the free sample tool there.
Once in the tool, click the "Try Demo" then "Face Comparison" link on the left-hand navigation pane, then upload your source image & comparison image.
Whilst more restrictive to access quickly, FR technology & the datasets that were used as part of the machine learning (ML) process differ from company to company. Therefore, it is beneficial to use multiple platforms to add another layer of verification to the process for comparative analysis.
Use Case
Let's apply this to a real missing person case to see some of the utility. We will look at the missing person case related to Logan Shearburn (https://www.namus.gov/MissingPersons/Case#/69693?nav)
We are assuming that investigators will search across prominent social media platforms to identify associated profiles. In this example, we will search on Instagram & identify that there are multiple profiles that could be related to the subject. Each of these may have valuable clues & associates which feed into the information chain for further analysis.
Here we can see a number of profiles. Some with clear profile images that we could associate without the requirement for FR, but some have obscured images or no profile image where we need to compare a picture of the known subject with images from the profile itself to create a positive identification (based on a confidence score, noting nothing is exact with FR).
Profile 1
Looking at the first profile, the images available have subjects with different haircuts, different facial expressions, different view angles & different facial hair. A person trained in image analysis may have no problem identifying feature matches, however, if the investigator does not have that expertise or access to those resources this is where we can use technology to assist without needing strong coding skills.
When comparing the profile image itself we can see that FR does not provide a match for the subject's photo. However, if we go 1 layer deeper & use a photo from a post on the profile, we get a 98.9% confidence score (using Amazon) that the profile is related to the subject.
It's important to note when we tried the same comparison using Microsoft Cognitive Services, it did not provide a match at all. This highlights the requirement to use and compare multiple tools, don't simply stop after a failed attempt.
Let's now look at a profile that has an obfuscated person in the profile picture, and the only facial photo available as a post within the profile is very dark & hard to identify.
Profile 2
Here we will see no match using the profile image itself, but we get a high confidence score for an image within the profile. Importantly, we could brighten up the image with some image editing tools, however, not everyone is comfortable with doing this so we are focusing this article on lowering the barrier to entry for using FR and therefore will use what's available directly in front of us.
As we can see above, even with a very dark image, the FR technology gives us some level of assurance that we are on the right path in verifying this person is associated with this profile.
Conclusion
This article was written to provide simple methods & freely available points-&-click tools to conduct FR without any deep technical understanding. Specifically, we wanted to explore methods to verify a subject against identified social media profiles as part of an investigation. An important learning point is that different tools will provide different results, and false positives are inevitable. Being diligent & trying multiple platforms is key.
For greater depth on the subject of FR & particularly building tools, understanding AI, and how it can be done at scale, there are some really good articles available. Below are some that might be of interest for further investigation:
How does facial recognition work (Norton): https://us.norton.com/internetsecurity-iot-how-facial-recognition-software-works.html
Who-Where-WhomWith(WWWW): A Facial Recognition Tool for Image-based Data Gathering and Graph Analysis (Lorenzo Romani): https://medium.com/analytics-vidhya/who-where-whomwith-wwww-a-facial-recognition-tool-for-image-based-data-gathering-and-graph-dd8f2b13c279
Facial Recognition with Python and Elasticsearch (Lorenzo Romani): https://medium.com/@lorenzoromani/facial-recognition-with-python-and-elasticsearch-quick-tutorial-85cd02fe903d