Back to all articles

What Is Facial Recognition – And How Sinister Is It?

Source: Guardian design Photograph: Guardian design/Getty

As a surveillance technique it’s ubiquitous, but is still a political, legal and ethical conundrum.

What is facial recognition?

Facial recognition technology has spread prodigiously. It’s there on Facebook, tagging photos from the class reunion, your cousin’s wedding and the office summer party. Google, Microsoft, Apple and others have built it into apps to compile albums of people who hang out together.

It verifies who you are at airports and is the latest biometric to unlock your mobile phone where facial recognition apps abound. Need to confirm your identity for a £1,000 bank transfer? Just look into the camera.

New applications crop up all the time. Want to know who’s at the door? A video doorbell with facial recognition will tell you, provided you’ve uploaded a photo of the person’s face. Other systems are used to spot missing persons and catch slackers who lie about the hours they spend in the office. Advertisers, of course, are in on the act. Thanks to facial recognition, billboards can now serve up ads based on an estimate of your sex, age and mood.

That sounds quite Big Brother. Is it a surveillance tool?

Sometimes, yes. China uses facial recognition for racial profiling and its tracking and control of the Uighur Muslims has been roundly condemned as a shameful first for a government. Its cameras also spot and fine jaywalkers, verify students at school gates, and monitor their expressions in lessons to ensure they are paying attention.

Russia has embraced the technology too. In Moscow, video cameras scan the streets for “people of interest” and plans have been mooted to equip the police with glasses that work the same way.

There have been reports that Israel is using facial recognition for covert tracking of Palestinians deep inside the West Bank. Meanwhile, in Britain, the Metropolitan and South Wales police forces have trialled facial recognition to find people in football and rugby crowds, on city streets, and at commemorations and music festivals. Taylor Swift even installed the tech at a gig in California to weed out stalkers.

Shops are increasingly installing the technology to deter and catch thieves. Next year, it will make its Olympic debut in Tokyo.

How did it get everywhere?

Advances in three technical fields have played a major part: big data, deep convolutional neural networks and powerful graphics processing units or GPUs.

Thanks to Flickr, Instagram, Facebook, Google and others, the internet holds billions of photos of people’s faces which have been scraped together into massive image datasets. They are used to train deep neural networks, a mainstay of modern artificial intelligence, to detect and recognise faces. The computational grunt work is often done on GPUs, the superfast chips that are dedicated to graphics processing. But there is more to it than flashy new technology. Over the past decade in particular, facial recognition systems have been deployed all over the place, and the data gathered from them has helped companies hone their technology.

How does it work?

First off, the computer has to learn what a face is. This can be done by training an algorithm, usually a deep neural network, on a vast number of photos that have faces in known positions. Each time the algorithm is presented with an image, it estimates where the face is. The network will be rubbish at first, like a drunk playing pin the tail on the donkey. But if this is done multiple times, the algorithm improves and eventually masters the art of spotting a face. This is the face detection step.

Next up is the recognition part. This is done in various ways, but it’s common to use a second neural network. It is fed a series of face pictures and learns – over many rounds – how best to tell one from another. Some algorithms explicitly map the face, measuring the distances between the eyes, nose and mouth and so on. Others map the face using more abstract features. Either way, the network outputs a vector for each face – a string of numbers that uniquely identifies the person among all the others in the training set.

In live deployments, the software goes to work on video footage in real time. The computer scans frames of video usually captured at crowd pinch points, such as entrances to football stadiums. It first detects the faces in a frame, then churns out vectors for each one. The face vectors are then checked against those for people on a watchlist. Any matches that clear a preset threshold are then ranked and displayed. In the British police trials a typical threshold for a match was 60%, but the bar can be set higher to reduce false positives.

It’s not the only way the police use facial recognition. If a suspect has been picked up, officers can upload their mugshot and search CCTV footage to potentially trace the suspect’s movements back to the scene of a crime.

How accurate is it?

The best systems are impressive. Independent tests by the US National Institute of Standards and Technology (Nist) found that between 2014 and 2018 facial recognition systems got 20 times better at finding a match in a database of 12m portrait photos. The failure rate fell from 4% to 0.2% over the period, a massive gain in accuracy that was largely down to deep neural networks. Nist says the networks have driven “an industrial revolution” in facial recognition.

But such great performance depends on ideal conditions: where a crisp and clear mugshot of an unknown person is checked against a database of other high-quality photos. In the real world, images can be blurred or poorly lit, people look away from the camera, or may have a scarf over their face, or be much older than in their reference photo. All tend to reduce the accuracy.

The technology has a terrible time with twins, the Nist tests found, with even the best algorithms muddling them up.

What about bias?

Bias has long plagued facial recognition algorithms. The problem comes about when neural networks are trained on different numbers of faces from different groups of people. So if a system is trained on a million white male faces, but fewer women and people of colour, it will be less accurate for the latter groups. Less accuracy means more misidentifications and potentially more people being wrongly stopped and questioned.

Last year, the American Civil Liberties Union (ACLU) found that Amazon’s Rekognition software wrongly identified 28 members of Congress as people who had previously been arrested. It disproportionately misidentified African-Americans and Latinos. Amazon said the ACLU had used the wrong settings.

Police trials have highlighted further shortcomings of facial recognition. A Cardiff University review of the South Wales trials found that the force’s NEC NeoFace system froze, lagged and crashed when the screen was full of people and performed worse on gloomy days and in the late afternoon because the cameras ramped up their light sensitivity, making footage more “noisy”.

During 55 hours of deployment the system flagged up 2,900 potential matches of which 2,755 were false positives. The police made 18 arrests using the system, but the Cardiff report does not state whether any of the individuals were charged.

The Welsh trial highlighted another challenge for facial recognition: lambs. Not farmyard fugitives but people on the watchlist who happen look like plenty of other people. While scanning the crowds at Welsh rugby matches, the NeoFace system spotted one woman on the South Wales police watch list 10 times. None of them were her.

Source: Cisco

Who has the technology?

Tech firms around the world are developing facial recognition but the US, Russia, China, Japan, Israel and Europe lead the way. Some nations have embraced the technology more readily than others.

China has millions of cameras connected to facial recognition software and Russia has declared hopes to expand its own surveillance networks. In Europe, as elsewhere, facial recognition has found its way into shops to spot thieves and into businesses to monitor staff and visitors, but live face recognition in public spaces is still mostly at trial stage.

In the US, the police typically use facial recognition to search CCTV footage for suspects rather than scanning live crowds. But it is becoming more pervasive. A 2016 report from Georgetown Law’s Center on Privacy and Technology found that half of all Americans are in police facial recognition databases, meaning that algorithms pick suspects from virtual line-ups of 117 million mostly law-abiding citizens. […]

What about other biometrics?

While facial recognition technology has drawn huge attention, the police and other organisations are looking carefully at new biometrics, beyond fingerprints and DNA, that uniquely identify people.

Skin texture analysis is said to overcome problems with facial recognition caused by different expressions and partially covered faces by analysing the distance between skin pores. It has not been tested much, but developers claim it could be good enough to distinguish between twins.

Another biometric that interests the police, because it can be done at a distance without a person’s cooperation, is gait analysis. As the name suggests, the algorithms identify people by the unique style of their stride, reflecting differences in anatomy, genetics, social background, habits and personality.

With vein recognition, optical scanners are used to map blood vessels in the hand, finger or eye. Because our veins are buried beneath the skin, the scanners are considered hard to fool. Fujitsu’s PalmSecure system uses vein maps to monitor the to and fro of employees at various businesses.

Speaker recognition is already used by banks and the HMRC to confirm people’s identities and its use is spreading. Unlike speech recognition, which translates sounds to words, speaker recognition detects the unique acoustic patterns created by an individual’s vocal tract and their speech habits. […]

Full article posted here by Ian Sample at Guardian.

If you enjoyed reading this article, you might find this post interesting too.

Follow us on our LinkedIn and Twitter to stay up to date with the latest news in the digital data-driven ecosystem

Digital analytics, optimisation, data science or programmatic expert, and looking for a job?

Check out our latest live vacancies here

Digital agency looking to expand your team with top-tier talent? 

Send us your jobs here

Get in contact with us!