The Myth of the Impartial Machine
The following post is a summarized version of the article that appeared in Parametric Press, issue 01. The full article is available here.
From voice assistants to image recognition to fraud detection to social media feeds, machine learning (ML) and artificial intelligence (AI) are increasingly important to society. The two fields have made enormous strides in recent years, thanks to gains in computing power and the so-called information explosion. Billed as being more accurate, more consistent, and more objective than human judgment, the promises and expectations of ML and AI have never been greater.
But there have been well-documented instances where these algorithms have produced disturbing results. Facial analysis programs were found to have an 20 to 34 percent error rate when trying to determine the gender of African American women, compared with an error rate of less than 1 percent for white men. If data-trained models are supposed to be objective and impartial, how did these algorithms go so wrong? Can such bias be fixed?
At the Urban Institute, we are using machine learning to classify text, create novel data extracted from news articles, and generate synthetic, anonymized public use files from confidential data, to name a few examples. It is critical that we understand how bias is introduced — and addressed — in our work so we can provide the best evidence for policymakers, researchers, and practitioners on the ground.
Sources of bias
Bias can be introduced and amplified during the machine learning process in two key ways.
1) Biased data
Data that are nonrepresentative and biased
Data are biased when they do not represent the population being modeled, which can happen through sampling errors and nonsampling errors.
1. Sampling errors occur when the selected sample does not accurately reflect the composition of the broader population the data were intended to model.
2. Nonsampling errors arise from many different sources. Typical examples include poorly phrased data-collection questions, web-only data collection that excludes people without easy access to the internet, overrepresentation of people who feel strongly about a subject, and responses that may not reflect true opinions.
Data that are representative but still biased
Even if a dataset is representative of its target population, it may still be biased to the extent that it reflects any historical social bias. A recruiting algorithm once used by Amazon was found to disproportionately favor male candidates. The data used to train the algorithm were based on resumes collected over 10 years. Because the tech industry was historically dominated by men, men submitted a majority of the resumes. Therefore, even though the dataset was “representative” of the historical applicant pool, it captured the inherent gender bias and passed it on to the recruiting algorithm.
2) Bias amplified through algorithms
Machine learning algorithms themselves may amplify bias if they make predictions that are more skewed than the training data. Such amplification often occurs through two mechanisms: incentives to predict observations as belonging to the majority group and runaway feedback loops.
Algorithms incentivized to predict observations to belong to the majority group
In order to maximize predictive accuracy when faced with an imbalanced dataset, machine learning algorithms are incentivized to put more learning weight on the majority group, thus disproportionately predicting observations to belong to that majority group.
Suppose we have an image-classification algorithm tasked with identifying the gender of a person in a given image. The training dataset contains 10 images of people cooking, each image showing either a man or a woman in the kitchen.
Suppose the training dataset contains eight images of women cooking and two images of men and that the model’s predictive accuracy is 70 percent. When these data are fed through the model, the model correctly labels the gender of six photos of women and one photo of a man while mistaking the gender of the remaining three photos (a 30 percent error rate).
But if the model were to predict that every image is of a woman, it would lower its error rate to 20 percent because it has correctly labeled all the photos of women while only mistaking the gender of the two photos of men. In pursuit of higher accuracy, the model has amplified the bias from 80 percent women in the training data to 100 percent women in the model prediction.
The GIF below shows what happens to the model’s error rate at different levels of accuracy. When accuracy is low, the model has a tendency to amplify bias, whereas when accuracy is high, the model is not incentivized to predict that all the images are of women. If you want to play around with the settings yourself to see how different levels of bias and accuracy affect the model’s predictions, please visit the original article.
Runaway feedback loops
In the previous image-classification example, bias amplification stops at the model prediction stage. But in machine learning models where the prediction is fed back into the model as inputs for the next round of predictions, bias can be amplified further in the form of a feedback loop.
Consider a predictive policing algorithm used to determine the optimal allocation of police forces across a city. Suppose the city has two precincts, A and B, and one policeman, John. The city wishes to use historical data on past crime occurrences to determine which precinct to dispatch John to.
The goal is to send John to each precinct in proportion to the frequency of crimes that occur in each precinct. If 40 percent of crimes occur in precinct A, the algorithm should send John to patrol A 40 percent of the time.
Let’s say the city experienced 22 crimes in the past year, with 12 of those occurring in precinct A and 10 in precinct B, as shown below.
The predictive algorithm uses these historical data to determine where to send John. The likelihood the algorithm will dispatch John to precinct A is in proportion to the share of all crimes that occurred in A. In this case, this probability is 12 in 22, or 54.5 percent. The probability the algorithm will choose to send John to precinct B is 10 in 22, or 45.5 percent. Let’s say the algorithm decides to dispatch John to patrol precinct A. While he’s there, John encounters three crimes and logs them in the system. The updated data now show a total of 15 past crimes in A and 10 in B. But because no one was sent to precinct B, crimes that occurred in B on the same day were not captured in the data.
The next day, when the algorithm decides where to send John, the probability of sending John to A would be 15 in 25, or 60 percent, and the probability of sending him to B would be 10 in 25, or 40 percent. The algorithm now has an even higher likelihood of sending John to precinct A as a result of its decision to assign John to A the day before. When the model chooses to send John to precinct A, more instances of crime will be logged for precinct A, while crimes that occur in B are ignored and remain uncaptured in the data. Over time, the data will become increasingly distorted, with the gap between the two precincts’ observed crime ratio and the actual crime ratio widening because of the model’s decisions.
The GIF below shows one way these algorithmic assignments might play out. In this example, precinct A initially has four crimes logged in the data while precinct B has two. See what happens over the next 14 days as the model uses these data to dispatch John.
Feedback loops are especially problematic when subgroups in the training data exhibit large statistical differences (e.g., one precinct has a much higher crime rate than others). A model trained on such data will quickly “run away” and make predictions that fall into the majority group only, thereby generating increasingly lopsided data that are fed back into the model. Even when subgroups are statistically similar, feedback loops can still lead to noisy and less accurate predictions. To play around with how different crime rate settings affect the model’s tendency to run away, please visit the original article.
Bias in data and in algorithms are interrelated
There is a clear relationship between bias in data and bias in algorithms. When an algorithm is fed training data where one group dominates the sample, it is incentivized to prioritize learning about the dominant group and overpredict the number of observations that belong to the dominant group. This tendency is exacerbated when the model’s predictive accuracy is relatively low. Conversely, if the data were balanced, the model would have nothing to gain by overpredicting the dominant group.
Bias can also be perpetuated through a feedback loop if the model’s own biased predictions are repeatedly fed back into it, becoming its own biased source data for the next round of predictions. In the machine learning context, it’s no longer just garbage in, garbage out — when there’s garbage in, more and more garbage may be generated through the ML pipeline if potential sources of bias are not monitored and addressed.
How to tackle bias
How does one actually tackle bias in the ML pipeline? Although a suitable fix depends on each specific circumstance, here are some ways to reduce bias in machine learning.
1) Debiasing the data
One key to debiasing the data is to ensure that a representative sample is collected in the first place. Bias from sampling errors can be mitigated by collecting larger samples and adopting data collection techniques such as stratified random sampling.
Bias from nonsampling errors is more varied and harder to tackle, but one should still strive to minimize these kinds of errors through means such as proper training, establishing a clear purpose and procedure for data collection, and conducting careful data validation.
What about data that are representative but reflect bias inherent in the population? For datasets that contain minority groups, one can oversample those minority groups to create a more balanced training dataset. For datasets that may contain biased associations, one can first quantify and remove any biased associations from the dataset before proceeding to the model training and prediction stages.
2) Debiasing the algorithm
In addition to debiasing the data, one can apply model interventions to directly offset an algorithm’s tendency to amplify bias. One intervention method is to impose model constraints that specify the statistical distribution of predictions. Researchers have been able to reduce bias amplification in image-labeling algorithms (PDF) by nearly 50 percent by adding model constraints that required the proportion of images predicted to be male versus female to fall within 5 percent of the ratio observed in the training dataset. This approach can be particularly useful for reducing bias amplification when using imbalanced training datasets.
Another particularly useful intervention method for combating feedback loops is imposing sampling rules that specify how outputs generated from model predictions should be fed back into the algorithm. Going back to the predictive policing example, one can impose a sampling rule such that the more likely police are sent to a particular precinct, the less likely data observed from those assignments are incorporated into the algorithm.
3) Evaluating performance against metrics other than just accuracy
Considering the performance of a model against metrics other than accuracy is important. Examples of alternative metrics include the false positive rate and the false negative rate.
One can then apply model constraints to make the algorithm satisfy a fairness rule. Common rules (PDF) include the following:
1. Predictive parity: Predictions are made without considering characteristics such as gender and race.
2. Well-calibration: In situations with more than one predicted outcome (e.g., risk scores on a scale of 1 to 9 instead of high versus low risk), the proportion predicted to be in a particular class is the same across groups for every possible score value.
3. Error rate balance: Performance measures such as false positive rates or false negative rates are required to be equal across groups.
4) Going beyond the data or algorithm
Beyond tackling bias in data or algorithms, some companies have begun to adopt broader measures, such as establishing ethical AI guidelines, where minimizing bias is included as part of a company’s overarching AI objectives.
Another measure is to conduct third-party algorithm audits to ensure each AI product satisfies a company’s AI guidelines. An even broader measure that targets the human source is to promote diversity in the AI and ML workforce. Although these measures alone do not eliminate bias completely from machine learning, they do reflect how companies and the broader AI and ML community are increasingly aware of the need to address bias as such technologies become more widely used.
Conclusion
Challenges to addressing bias in machine learning
Although the previous section mentioned approaches to mitigate bias in ML, truly eliminating such disparities is more challenging. Even with perfectly representative data, if the underlying systems being modeled are themselves unjust, then the model results will still end up reflecting these biased behaviors.
Another challenge is pinpointing exactly what a “fair” outcome means. “Fairness” has many definitions, yet proving all these definitions simultaneously is mathematically impossible. Trying to be fair in one way necessarily means being unfair in another way.
What can be done to fix the situation?
Lest the situation seem hopeless, companies and organizations can take steps to improve the results of their modeling efforts. One such step is to make sure the group working on ML problems is diverse enough to represent as many viewpoints as possible. In addition, employees should undergo bias training to increase awareness of how their own assumptions of the world influence their work.
Another important point is being transparent and open about exactly what a ML model does, how it arrives at its results, and what accuracy metrics it is optimizing. Finally, acknowledging that ML is not a magical solution to all the world’s problems but, like any other tool, has its limitations and weaknesses, will help maintain a more realistic perspective on what these models can (and cannot) achieve.
Tackling bias in machine learning is a challenging problem, and there are no simple solutions or clear-cut answers. But it is imperative that practitioners do work to address bias because it imposes serious costs on both individuals and society as a whole by unfairly denying people opportunities while perpetuating stereotypes and inequities. ML and AI will indeed change the world; let’s make sure they change the world for the better.
-Alice Feng (Urban Institute)
-Shuyan Wu (State of Rhode Island)
