Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor

“Like earlier technological innovations in poverty management, digital tracking and automated decision-making hide poverty from the professional middle-class public and give the nation the ethical distance it needs to make inhuman choices: who gets food and who starves, who has housing and who remains homeless, and which families are broken up by the state. We manage the individual poor in order to escape our shared responsibility for eradicating poverty.”

Eubanks argues that maintaining personal distance is a way to avoid contemplating labor history and welfare rights. In a world of limited and insufficient resources, humans would prefer to remain in ignorance rather than be forced to recognize the “inhuman choices” they must make.

“Though most poorhouse stays lasted less than a month, elderly and disabled inmates often stayed for decades. Death rates at some institutions neared 30 percent annually. Poorhouse proponents reasoned that the institution could provide care while instilling moral values of thrift and industry. The reality was that the poorhouse was an institution for producing fear, even for hastening death.”

The image of the poorhouse has the dramatic irony of attempting to be a moral “correction” in which conditions were anything but moral. The introduction of “thrift and industry” was merely a smokescreen to shift blame onto the poor, satiating middle class worries by contributing to the myth of the chronically and blameful poor.

“Hysteria about welfare costs, fraud, and inefficiency increased as the 1973 recession took hold…Elected officials and state bureaucrats, caught between increasingly stringent legal protections and demands to contain public assistance spending, performed a political sleight of hand. They commissioned expansive new technologies that promised to save money by distributing aid more efficiently. In fact, these technological systems acted like walls, standing between poor people and their legal rights. In this moment, the digital poorhouse was born.”

Eubanks strengthens the metaphor of the digital poorhouse by alluding to the history of actual poorhouses: The new “walls” surrounding the poor are automated technical systems.

“Mailing Date: 3/26/08

Dear SOPHIE STIPES

MA D 01 (MI)

Your MEDICAID benefits will be discontinued effective APRIL 30, 2008 due to the following reason(s):

-FAILURE TO COOPERATE IN ESTABLISHING ELIGIBILITY

-FAILURE TO COOPERATE IN VERIFYING INCOME SUPPORTING LAW(S) OR REGULATION(S):

470IAC2.1-1-2

Important: If you believe you may be eligible for Medicaid benefits under another category and have more information about your case, please contact us at the number listed at the top of this notice within ten days (13 days if this notice is received by mail) of the date of this notice.”

The language of the notice dehumanizes the recipient. Its form is disrespectful, curt, and unpleasant: Writing in all-caps has long been the visual equivalent of shouting, while short imperative sentences read like barked orders. Its content is similarly aggressive, the accusatory language preemptively assuming bad intent on the part of the recipient, while the stressful time constraints it imposes with no regard to potential difficulties of receiving mail shift responsibility onto the recipient.

“Millions of copies of drivers’ licenses, social security cards, and other supporting documents were faxed to a centralized document processing center in Grant County; so many of them disappeared that advocates started calling it ‘the black hole in Marion.’ Each month the number of verification documents that vanished—were not attached properly to digital case files in a process called ‘indexing’—rose exponentially. According to court documents, in December 2007 just over 11,000 documents were unindexed. By February 2009, nearly 283,000 documents had disappeared, an increase of 2,473 percent. The rise in technical errors far outpaced increased system use.”

Eubanks uses hard numbers to really drive home the failure of the new automated system. Alluding to the digital gap as a “black hole” where data “vanishes” strengthens the mysterious authority of digital technocracy as means of profiling the poor.

“One of the great victories of the welfare rights movement of the 1960s and ‘70s was the redefinition of welfare benefits as the personal property of the recipient, rather than as charity that can be bestowed or denied on a whim. Activists successfully challenged inequitable access to public assistance by appealing decisions and demanding access to administrative law procedures known as fair hearings.”

The appeal to property law worked, but unfortunately also further entrenched moralistic ideas of personal responsibility rather than the more collective approach to community support.

“Ronald Reagan’s 1976 stump speech about the lavish lifestyle of ‘welfare queen’ Linda Taylor was intended to make the face of welfare both Black and female. ‘There’s a woman in Chicago,’ he said during the New Hampshire Republican presidential primary contest. ‘She has 80 names, 30 addresses, 12 Social Security cards and is collecting veterans’ benefits on four non-existing deceased husbands. She’s got Medicaid, getting food stamps and she is collecting welfare under each of her names. Her tax-free cash income alone is over $150,000.’ Ms. Taylor was eventually charged with using 4 aliases, not 80, and collecting $8,000, not $150,000, but Reagan’s overblown claims found fertile ground, and the image of the welfare queen has remained central to our country’s understanding of public assistance.”

The wild exaggeration in President Reagan’s speech drew on and further influenced myths about the extent of welfare fraud. His language is hyperbolic as he inflates numbers to achieve a rhetorical effect for the audience: outrage and anger at the fabricated specter fraud.

“Automated decision-making can change government for the better, and tracking program data may, in fact, help identify patterns of biased decision-making. But justice sometimes requires an ability to bend the rules. By removing human discretion from frontline social servants and moving it instead to engineers and private contractors, the Indiana experiment supercharged discrimination.”

Eubanks undermines the assumption that history always corrects itself, arguing that this is just a moral salve for people to avoid actively fixing a bad situation. Justice requires corrective action, not passivity.

“But when federally funded low-income housing was proposed to replace what had been demolished, white middle-class Angelenos vigorously resisted. Calling a plan to construct 10,000 affordable public units a ‘Red Plot to Control L.A. Housing,’ opponents blocked the creation of Elysian Park Heights, a racially integrated public housing complex, and organized to have the City Housing Authority investigated by the House of Un-American Activities Committee on charges of Communism.”

This quote highlights two key historical insights: that racism motivated the current housing crisis, and that political campaigns papered over any whiff of overt racism. Eubanks’s inclusion of these historical moments is indicative of a political history that prioritized avoiding issues for short-term approval rather than long-term strategies to address problems.

“His tent is immaculate. There are crates with OxyClean, laundry detergent, and a bottle of bleach. Science fiction novels, a copy of It Can’t Happen Here by Sinclair Lewis, and a copy of the progressive magazine In These Times sits on his air mattress. He’s trying to stay healthy, so he’s switched to diet drinks, and there are maybe a half-dozen two-liter bottles scattered around: diet cranberry, Mountain Dew, Gatorade. Some sport a black-marker ‘X’ on their twist top: they might contain rum or act as makeshift midnight latrines.”

Eubanks takes care to portray Gary Boatwright as outside the common stereotypes of unhoused people. She stresses the cleanliness of his tent, the fact that he reads insightful social commentary, and his interest in health and wellness. Her goal is to build empathy between her readers and this man by illustrating the ways in which he is like them. Tidy, scholarly, and engaged, he is quite different from the image of the poor and unhoused politicians demonize.

“But in South LA, a high VI-SPDAT score is a catch-22. There is very little permanent supportive housing in the area, so Pathways clients have to go through a second interview with the housing authority to determine if they are able to live independently in private housing. A high VI-SPDAT score might qualify a Pathways client for a Section 8 voucher. But it can also be an indicator that he is too vulnerable to live on his own.”

LA applicants must walk the tricky tightrope of the VI-SPDAT, stripping away their agency and making it clear that their attempts to get assistance are coerced performances.

“There is a long history of social services and the police collaborating to criminalize the poor in the United States…In the absence of strong data protection rules, it seems likely that coordinated entry’s electronic registry of the unhoused will be used for similar purposes. Outstanding warrants for status crimes provide justification for dragnet searches. Mobile and administrative data can turn any street corner, any tent encampment, or any service provider into a site for a sting operation.”

Eubanks references the obscured workings of the coordinated entry system in LA. Addressing the inequities of the system requires active rather than passive citizen participation: in this case, the introduction of strict data protection laws. Ironically, the harvesting of data has become more of an intrusion than literal policing.

“The problem is not that the city lacks adequate data on what kind of housing is needed to address the homelessness problem. Rather, poor and working-class people and their allies may not be able to overcome explicit political resistance from organized elites. The proponents of the coordinated entry system, like many who seek to harness computational power for social justice, tend to find affinity with systems engineering approaches to social problems.”

Eubanks points out that we haven’t corrected the problem of poverty not because it’s so tricky to do, but because “organized elites” have no reason to care about those languishing in digital poorhouses. The perception that there aren’t enough resources to go around tamps down the imperative to share.

“And the next time he takes the VI-SPADT, he will likely score lower. The model counts prison as housing. The system will see him as less vulnerable, and his prioritization score will slip even lower. He’ll stay trapped, too vigorous for intervention and too marginal to make a go of it without support.”

The automated system that regulates Gary Boatwright’s housing cannot truly assess his needs. Instead, using public services—or being forced to use them while incarcerated—becomes a deciding factor in future policing and access to services.

“Even with recent clarifications that the harm must be ‘serious,’ there is considerable room for subjectivity in what exactly constitutes neglect or abuse. Is spanking abusive? Or is the line drawn at striking a child with a closed hand? Is letting your children walk to a park down the block alone neglectful? Even if you can see them from the window? The first screen of the list of conditions classified as maltreatment in KIDS illustrates just how much latitude call screeners have to classify parenting behaviors as abusive or neglectful.”

This lack of authoritative and well-informed data allows too many outside variables to influence the results of poverty profiling tools. Despite technological advancements, human judgment is still necessary for fair assessment—though the new tools view human decision-making as disruptive to the process.

“Call referral is a deeply problematic proxy for maltreatment. It can be easily manipulated. CYF’s own research shows that it creates nearly all the racial disproportionality in the county’s child protective system. In other words, the activity that introduces the most racial bias into the system is the very way the model defines maltreatment. This easily gameable, discriminatory variable threatens to reverse all of the extraordinary work Cherna and his team have done.”

By neglecting to include racial bias in their model, researchers repeated the mistake that plagues inadequate technological systems as a part of the digital poorhouse. Flawed data gives flawed results and cannot be the solution to problems.

“Our denial runs deep. It is the only way to explain a basic fact about the United States: in the world’s largest economy, the majority of us will experience poverty. According to Mark Rank’s groundbreaking life-course research, 51 percent of Americans will spend at least a year below the poverty line between the ages of 20p and 65. Two-thirds of them will access a means-tested public benefit: TANF, General Assistance, Supplemental Security Income, Housing Assistance, SNAP, or Medicaid. And yet we pretend that poverty is a puzzling aberration that happens only to a tiny minority of pathological people.”

Eubanks admits that our inability to address poverty is partially due to our propensity to ignore it. By shifting blame onto the poor, stigmatizing them as inherently puzzling and “pathological,” the middle-class can imagine that they can avoid making the “choices” the poor must have made to end up in their unenviable position.

“When the digital poorhouse simply bars access to public benefits, as in Indiana, it is fairly easy to confront. But classification and criminalization work by including poor and working-class people in systems that limit their rights and deny their basic human needs. The digital poorhouse doesn’t just exclude, it sweeps millions of people into a system of control that compromises their humanity and their self-determination.”

This specificity towards those who are included versus excluded in the expanding digital poorhouse helps bolster Eubanks’s theory that the digital poorhouse is being built around everyone in society, entrapping them by limiting their personal authority and self-determination.

“Prediction, unlike classification, is intergenerational. Angel and Patrick’s action will affect Harriette’s future AFST score. Their use of public resources drives Harriette’s score up. Patrick’s run-ins with CYF when Tabatha was a child will raise Harriette’s score as an adult. Angel and Patrick’s actions today may limit Harriette’s future, and her children’s future. The impacts of predictive models are thus exponential. Because prediction relies on networks and spans generations, its harm has the potential to spread like a contagion, from the initial point of contact to relatives and friends, to friends’ networks, rushing through whole communities like a virus.”

The simile comparing risk assessment models to a contagion clarifies Eubanks’s argument that the exponential data harvesting done by profiling tools has the potential to unleash chaos that we cannot yet predict.

“Forensic anthropologists still routinely find skeletons in poorhouse burying grounds that show evidence of being tampered with: saw marks on femurs and pelvic bones, skulls with tops that lift off like lids. Yesterday, we experimented on the corpses of the poor; today, we tinker with their futures.”

Modern readers are shocked by the medical experimentation conducted on the poor in previous centuries but know little about the digital experimentation inflicted on the poor today. By drawing this comparison between the physical and digital poorhouses, Eubanks drives home the terrifying consequences at stake.

“[Administrators and data scientists] obliquely accuse their subordinates, often working-class people, of being the primary source of racist and classist outcomes in their organizations. Then, managers and technocrats hire economists and engineers to build more ‘objective’ systems to root out the human foibles of their economic inferiors. The classism and racism of elites are math-washed, neutralized by technological mystification and data-based hocus pocus.”

Despite their claims of scientific objectivity, the technological systems in place operate almost magically, curtained off from observation. This undercuts political claims that numbers are neutral—in reality, they reflect the biases and blind spots of those who created the algorithms that produce them.

“It would stand us all in good stead to remember that infatuation with high-tech social sorting emerges most aggressively in countries riven by severe inequality and governed by totalitarians. As Edwin Black reports in IBM and the Holocaust, thousands of Hollerith punch card systems—an early version of computer software—allowed the Nazi regime to more efficiently identify, track, and exploit Jews and other targeted populations.”

By alluding to WWII, Eubanks draws a connection between the increase of fascist policing and the feigned ignorance of politicians. This historical reference may feel overblown, but it effectively illustrates the weight of Eubanks’s argument for readers.

“The most important step in dismantling the digital poorhouse is changing how we think, talk, and feel about poverty. As counterintuitive as it may sound, the best cure for the misuse of big data is telling better stories. But our vision has been radically limited by the narrow frame that has evolved for talking about poor and working class people.”

The book’s conclusion condenses its argument: The poorhouse was a means of isolating parts of the population so that others could ignore poverty; the digital poorhouse accomplishes this same function today.