The ethical implications of using AI

Software and technology companies use artificial intelligence (AI) to make processes and applications easier for everyone involved. You can open your phone just by looking into the camera. You can call out to a device to give you the weather report. It can redirect you when traffic is getting too heavy in a certain area. You can resolve customer service issues without waiting on hold. In the best examples, AI cuts down employees’ workloads and improves the user experience by limiting inefficient human interactions. But there is a flipside to reducing the human element: unintentional bias.

All AI is built from data — whether loaded into it or gathered by it over time. Where, when and how that data is digested by the AI is written into the machine learning algorithms by computer programmers. They take ideas for programs, processes and apps and figure out what information is needed to make that idea come to fruition. So to limit human-to-human interactions on the back end, they have to be heavily involved on the front end, and that is where unconscious bias comes in.

In all that we do, we are limited by our knowledge and background in how we frame things. That is not to say efforts are not made to include other perspectives, but that, too, is dependent on how wide the net is cast in gathering those additional viewpoints.

For example, facial recognition continues to increase in popularity. As mentioned earlier, it’s used to open phones, but it’s also being used in such ways as allowing entry into secure locations and in uncannily identifying people on social media. While that may raise some privacy concerns, it is really in how law enforcement is using it that is prompting even more serious questions.

Facial recognition can yield incredible results when, say, trying to track someone’s movements through security camera footage. Those results can also be incredibly flawed. The problem is the programs that are being employed to perform facial recognition services are limited to the data they are being fed. One study looked at the datasets being supplied by a leading company in the field to organizations that were building facial recognition software. The analysis found that in that widely distributed dataset only 40% of the images were of women and only 1% were images of people over the age of 60. It was found to have an overrepresentation of men ages 18 to 40, but an underrepresentation of people with darker skin.  

A 2019 federal study by the National Institute of Standards and Technology (NIST) found similar issues among the majority of face recognition algorithms. The study examined how 189 software algorithms from 99 developers executed the two most popular applications of facial recognition on more than 18 million images of about 8.5 million people. Overwhelmingly, U.S.-developed algorithms resulted in higher rates of false positives for women over men; for Asians, African Americans and Indigenous groups, with respect to race; and for the elderly and children, with respect to age.

The study also discovered that some algorithms developed in Asian countries resulted in fewer false positives between Asians and Caucasians, which could mean “the location of the developer as a proxy for the race demographics of the data they used in training … is potentially important to the reduction of demographic differentials due to race and national origin.”

Joy Buolamwini, an AI researcher and founder of the Algorithmic Justice League, published a study that revealed the vast gender and racial biases found in the facial recognition software sold by Amazon, IBM and Microsoft. Because of her report, all three took their software off the market. Like the NIST study, her findings indicated the majority of facial recognition worked successfully for white males most of the time, but not for people of color or women.

The ramifications of the biases have resulted in at least three separate cases of Black men being arrested for serious crimes they didn’t commit. Even though the arrests were eventually thrown out, there is no way to measure the psychological and emotional toll of their experiences. All three have filed lawsuits for their wrongful arrests.

But facial recognition isn’t the only form of AI the criminal justice system is using.

In some states, judges can use programs that can predict recidivism (i.e., the likelihood someone will commit another crime) when determining a defendant’s sentence. One of the most widely used tools for this draws from 137 data points when creating its prediction. In a 2016 ProPublica analysis, researchers looked at more than 7,000 individuals arrested in Broward County, Florida, between 2013 and 2014, and revisited the individuals two years later to see how many had re-offended, which was the same benchmark used in the algorithm.

Only 20% of those who had been predicted to commit violent crimes had done so. Furthermore, the study found the algorithm more often incorrectly predicted Black defendants were at a higher risk of re-offending (45%) than white defendants (23%). Conversely, white defendants were nearly twice as often (48%) mislabeled as a lower risk to re-offend than Black defendants (28%).

In fact, when controlled for prior crimes, future recidivism, age and gender, Black defendants were 77% more likely to receive higher risk scores than their white counterparts.

The ProPublica analysis found that even though the questions asked to gather the data were not deliberately biased, they were inherently biased, skewing the predictions in a way that was beneficial to violent and nonviolent white defendants.

The scores are generally meant to be just one factor when establishing a sentence, but it has been used by a judge to actually overturn a plea deal for a longer sentence.

Interestingly, the founder of the company who developed the recidivism algorithm focused on in the ProPublica study said the program wasn’t designed to be used for sentencing. It was meant to be used to help cut down on crime. The founder said this when he was called as a witness at an appeal for a defendant whose sentence was influenced by his rating, which said he was a high risk for violent recidivism.

When looking at these two applications of using machine learning and AI in criminal justice, it is evident that more perspectives and more representation need to be at the table when these systems are developed.

Still, AI has been used to genuinely solve bias issues.

When it snowed in Sweden in the past, the main roads and the highways would be cleared first, and then sidewalks and bike paths. Most countries that get snow operate this way.

One Swedish city conducted a study from a gender perspective and found that three times as many pedestrians were injured from icy conditions than motorists. What does that have to do with gender? A larger percentage of women were being injured since they walk, cycle and use public transportation more than men, who typically drive to their work.

At no extra cost to the municipality, the city reordered how snow was removed, so walkways to daycares were given priority, then sidewalks to larger office buildings that had a more female-dominated workforce (e.g., hospitals and municipal facilities), followed by bike paths and sidewalks to schools and, finally, roadways.

Incorporating gender-equality standards resulted in fewer injuries to residents and, in turn, a reduction in medical expenses, which benefits everyone since Sweden offers government-funded healthcare to its citizens.

Bringing machine learning and AI into our lives isn’t all positive or negative. Sure, society is reaping plenty of benefits from properly designed AI programs, but until inclusion expands among developers and takes into account more diverse viewpoints and experiences, the damage caused by deficient AI programs will undermine any of the benefits.