Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

O’Neil is a popular blogger who writes about the risks of big data applied to populations seeking credit, being evaluated for parole, seeking jobs, trying to get into college, being evaluated for effectiveness as teachers, and so on. The biggest takeaway from this short book, which is well worth reading, is that big data’s often fatal flaw is lack of feedback. Really effective models assist their users only if they neither over-include nor under-exclude, like Moneyball or 538’s models or even Google’s search engine. Which means that you follow up on people you predicted would succeed and people you predicted would fail, and if you missed their performance you try to update the model. But algorithms that deny people credit show more neither follow up on them nor leave their subsequent performance unaffected by the operation of the model—a person who can only get credit at 18% a month will predictably be more likely to default than someone who got 2%. Likewise, teacher evaluation algorithms don’t have independent measures to cross-check; a teacher fired for being ineffective doesn’t get fed back into the system if she leaves for another school system that doesn’t use the same metrics and then wins Teacher of the Year. Moreover, current models often take as predictors things that correlate with being poor and nonwhite, rather than treating those disparities as problems that need to be ameliorated by social and public policy. show less

I, too, was a mathematician once, but I lost my faith. Ms. O'Neil still seems to have much of hers, for an Occupier. I kept thinking she was naive that this stuff is fixable, but I may just be naively paranoid.

That algorithms can be biased was not a surprise to me. Logic itself can be biased because it is dependent on language which, like many WMDs, is a black box. It is full of proxies. Take the term "criminal" which (like "terrorist") brands the one so called as an evil doer. And it's measurable by determining if one has been convicted of anything. You can read The New Jim Crow: Mass Incarceration in the Age of Colorblindness and discover that going to jail is part of systemic racism but when you hear the word "criminal" or "convict", show more you usually don't think much further.

Or consider how we think about patents. It's supposed to protect inventors and most people think that's what it's doing (and perhaps much of the time it is) but it has become a weapon of big companies to keep small ones from entering a field.

This is all before mathematics enters into it.

When reasoning is done by a computer, it's faster, bigger, and less reflective. You might think the marketplace can fix this. A business that does less error-prone reasoning will make more money and prevail. Ms. O'Neil tries to explain why it doesn't work this way but I don't think she does that good a job of it. At least she tries.

The problem is that even with a sloppy algorithm, you can make up the difference on volume. In the world of actual weapons of mass destruction, you could nuke entire countries and win the war. Most of the people you killed are "innocent" except in the sense of having been born into the wrong nation. Businesses using "approximate" algorithms will succeed and success is self-validating. On an individual level we say "life is unfair" but on a corporate level the winners selected by the marketplace are seen as deserving their success if they haven't broken any laws which anyone noticed. (And if someone noticed? Google "war on whistleblowers.")

Ms. O'Neil thinks that this is just the birth throes of a new technology and in the future, we'll look back at it like we do at sweat shops and child labor, excesses that we have managed to overcome. I am less confident of our rosy future. I'm more like Chris Hedges (one of whose books I read right before this one) who sees morality not as a dimension of society that progresses like technology and science, but one which remains at pretty much a constant level. I see the technology as multiplying our moral failings in a way that will be difficult to correct. (I am reminded of those who believe that those who fear global warming are underestimating man's ability to solve problems and some yet unforeseen discovery will come along when we need it, and we can just ignore it for now.)

Currently, outside of dystopian science fiction, people see apps as a universal good and trust our cyber-overlords. Then, some may find themselves among the collateral damage, and if sufficiently hurt, will discover no one will want to listen to them. Isn't it an axiom of capitalism that, with some exceptions we are safe to ignore, the poor deserve to be poor? show less

A pretty thought-provoking read, even if you're already familiar with the idea. She unearths so many ways that algorithms affect our lives, without us even realizing it, and her central thesis is strong & clear: These things can be made to help us, or to harm us, but they are not neutral. She advocates powerfully for transparency and oversight (typically in the form of auditing) in the process, which really would benefit all stakeholders. And she reminds us again & again that we need to constantly consider our true priorities. When we don't think carefully about it, we will often tend to value monetary gains over justice & fairness. But if we take the time to assess what we're doing, it's usually possible to alter that calculus, and use show more algorithms in ways that suit our true societal values.

Definitely recommend this. show less

In the two years since Cathy O'Neil published this study of the danger posed by mathematical models—how their supposed neutrality merely reflects the biases of those who create them while creating closed-system feedback loops that warp society in unjust ways—it hasn't become any less relevant. In fact, not only does the book remain relevant, but subsequent revelations—most prominently those about Facebook's involvement with the 2016 US elections—show that O'Neil may even have been too cautious in what is a trenchant and prescient critique.

This was a frustrating book for me.
On the positive side, it does a really good job of describing the ways that data and algorithms can be misused or abused, either deliberately or unintentionally, leading to flawed conclusions and damaging outcomes.
On the other hand, the tone of the book (as encapsulated in the subtitle) implies that it is the use of the data itself that is the problem, rather than the business practices that use them.
Just as fire can be used to warm a home or burn it down, all technology is associated by risk and the need to regulate its use. But one gets the sense that the author would rather have us shiver in the dark than take advantage of the opportunity to illuminate the darkness.
So the end result is a show more condemnation of not only the bad decisions and structural inequities inherent to our current under-regulated capitalist economy and associated sociopathic values, but also a rejection of some of the technologies that could be used to address these issues if we so choose. show less

This book shows the hidden ways in which the use of "Big Data" is much more far-reaching and harmful than expected. Big data refers to the massive amount of information now available because of computers that is collected and analyzed and sold to third parties.

In particular, as the author demonstrates convincingly, applications of Big Data “punish the poor and the oppressed in our society, while making the rich richer.” She paints a sobering picture.

The author calls the mathematical models employing Big Data and used to such harmful effect “Weapons of Math Destruction” or WMDs.

In WMDs, she explains, “poisonous assumptions . . . camouflaged by math go largely untested and unquestioned.” They create their own toxic feedback show more loops, and, to an extent which shocked me, guide decisions in a large variety of areas ranging from advertising to prisons to healthcare to hiring and firing decisions. Most importantly, because they rely on esoteric mathematical models, no matter that they are many times based on biased and/or erroneous premises:

“They’re opaque, unquestioned, and unaccountable, and they operate at a scale to sort, target, or ‘optimize’ millions of people.”

The goal is always profit, but what is lost is fairness, the recognition of individual exceptions, and simple compassion and humanity, adding to the inequality gap, not to mention downward spirals for some unfortunate victims from which it is almost impossible to escape.

I am not at all well-versed in math, but the author manages to explain how all this works without requiring that one understand specific algorithms. She provides examples from the worlds of teacher evaluations, hiring decisions generally, advertising, insurance, police programs, college admissions, lending and credit evaluation, and political targeting.

One of the saddest chapters (and they are all sad, unfortunately) is about the many for-profit universities (Trump University comes to mind) that specifically target people in great need, selling them overpriced promises of success. Her quotes from the marketing materials of these places are horrifying. They look for individuals who are “isolated,” with “low self esteem” who have “few people in their lives who care about them” and who feel “stuck.” She shows how they use google searches, residential data, and Facebook posts, inter alia, to find “the most desperate among us at enormous scale”:

“In education, they promise what’s usually a false road to prosperity, while also calculating how to maximize the dollars they draw from each prospect. Their operations cause immense and nefarious feedback loops and leave their customers buried under mountains of debt.”

The chapter on the way the “stop and frisk” policing operates is also very depressing; and in truth we have seen the tragic results in city after city.

The fact is, the whole book is rather a downer, albeit an important one. Although O’Neil cites a few programs that have used Big Data to help people rather than to enrich a few and oppress the rest, can one really think that “moral imagination” can take precedence over prejudice and greed? Personally, I’m not so sure. The author provides ideas about how to change (and importantly, regulate) uses of Big Data, but she is more optimistic than I am, ending on a positive note:

“We must come together to police these WMDs, to tame and disarm them. My hope is that they’ll be remembered, like the deadly coal mines of a century ago, as relics of the early days of this new revolution, before we learned how to bring fairness and accountability to the age of data. Math deserves much better than WMDs, and democracy does too.”

Evaluation: I hope this important book gets a lot of attention. My husband always makes the argument about privacy concerns that what do we care if we’ve done nothing wrong? This book shows how, astoundingly, that isn’t enough to stop Big Data from hurting us in many aspects of our lives. It is a critical lesson for today’s world, and the world of our children. show less

Putting you in your place

We model everything now. Teacher evaluations, job applicants, credit applications, online purchasing, voting patterns, crime – pretty much anything you can think of is modeled in some opaque black box of unaccountable algorithms. They are so inherently faulty, discriminatory and racist as to be shameful. They cut off the poor from keeping up, and provide the wealthy with all kinds of advantages. Data Scientist Cathy O’Neil says in Weapons of Math Destruction: ”Big Data processes codify the past. They do not invent the future.”

Something as simple as a zip code can tell a system what kind of neighborhood you live in, and make assumptions. Search history, social media activity, purchase record – all show more contribute to an instant decision that you are worthy or not. These values are plugged in to school applications, job applications, and personal evaluations such as HR records, personality tests and even dating sites. Even purged, forgiven, and expired details remain active. Police model neighborhoods. They harass residents for every little thing in poorer neighborhoods, while giving a free pass to wealthier ones, where crimes are far bigger, but mostly white collar. Only ten states have outlawed the use of credit checks on job applications. For shopping at downscale stores, credit cardholders had their limits slashed, making them poorer and making them poorer risks – as in higher interest rates. It is computer models that schedule shifts, without concern for the needs of the employee in terms of child care, time off between shifts, or advance notice. Managers are paid to optimize revenue per hour worked, so memos from above go unheeded.

That models are often incorrect, badly designed, misinformed and misconstrued, means that people are denied service, or not hired, or outright fired. But there’s always someone else behind them, so it’s just the cost of doing business. “Unfairness is the black stuff belching out of the smokestacks. It’s an emission, a toxic one,” O’Neil says. We are all just collateral damage.

One insurance company instantly evaluates whether a customer is likely to shop around. If it judges not, it charges them more. It actually has 100,000 microsegments (buckets) depending on instant customer scores. In Florida, a driver with a clean record but a poor credit score pays $1552 more for insurance than a driver with a high credit score and a drunk driving conviction. Shopping sites won’t offer you a discount if you are already logged in. Payday loans and for profit schools prey on the disadvantaged and the desperate, extracting billions from them. The games are endless.

WMD is extremely fast paced, fact packed, and depressing. It has come to the point that machines dictate who may have a successful life, right out of the gate. Initiative, courage, creativity, drive, human kindness – don’t enter into it. We are all typecast by Big Data - assigned values mathematically that can stymie a life. There is no appeal. There isn’t even any knowing. The poor get poorer. The rich find the new era refreshing.

And of course, none of this is transparent. Customers cannot arrive at these prices, these decisions or these scores themselves. It’s all in the math, manipulating us. And yet, 73% of Americans believe search engine results are “accurate and impartial”. 62% believe Facebook posts their submissions to everyone. Nothing could be farther from the truth. Worse, data banks draw on each other, multiplying their errors, sometimes creating completely false profiles of a person, who then cannot get a job, rent an apartment or buy a car.

O’Neil says she is outraged by her own profession. You will be too.

David Wineberg show less