Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

A nice overview of the misuses and inherent biases of many different decision-making tools based on "big data". O'Neil dubs these biased algorithms "Weapons of Math Destruction", and provides many examples of the negative effects of these tools, in particular, of the disenfranchisement, social immobility, credit traps, price gouging, and other nastiness done to poor people in the name of efficiency and profit. Luckily for the poor, these algorithms are also increasingly affecting middle class people in similar ways, as companies insinuate they way into people's driving habits, recreational choices, "wellness", psychological state, and political relevance.

She calls for careful ethical consideration and assessment of these toxic show more algorithms, but I was left feeling that the political will to regulate these tools is unlikely to be there for us, especially as these tools are exceptionally helpful for businesses to maximize their profits at the expense of employees and customers alike. Add in her observations that many of the worst offenders either actively help the richest Americans or can be avoided by the application of a bit of money, and it's hard to see any hope of change until such time as some sufficiently horrible event occurs that affectis a broad enough range of people to force change. (Just kidding, obviously, as the 2008 financial crisis would have seemed like just the sort of thing to make the government punish offenders and regulate the financial industry, but that certainly didn't happen.) show less

Weapons of Math Destruction shows us how mathematical modelling can perpetuate institutional inequalities under the guise of sound science. Each of the 10 chapters focuses on exposing an industry's unfair modelling practices. Practices that handicap the poor and marginalized unfairly. The list goes from usual suspects like online advertising and US justice system, to education and the job market. It's a warning against techno-solutionism, with concrete examples of corporations using algorithms that trade consumer privacy for profit. It's timely, important, and should be read widely.

Cathy points out three definitive factors of a "WMD":
1. Opacity
Are the inner workings of the system transparent? Are the outputs consistent across the board? show more For example, the difference between NYT influencing electorates and FB influencing electorates is that NYT's output is public and same for everyone. This ensures accountability and ability to appeal.
2. Damage
What is the maximum damage ? Are there feedback loops that self-satisfy the model's assumptions? For example, credit scoring models that unfairly give black people lower scores could put them in a circle of poverty and unemployment, further driving their credit score down.
3. Scale
What is the reach of the model? For example, the Reagan administration's 1983 report A Nation at Risk enabled nation-wide scoring of high-school teachers.

The book really shines in the final chapters when addressing the instruments employed by credit scoring and insurance firms. The final chapter on social media and elections is prescient. Interesting bits there include Solomon Messings at FB, Epstein-Robertson on GOOG, and Simulmedia bringing profiling to TV.

One quibble with the book would be the chapter on online advertising. The book fails to mention a wealth of research on online trackers and FTC filings about grotesque violations of the right to privacy by big-tech companies. It failed, atleast for me, to underscore the extent and impact of the online tracking problem. Another such blindspot would be the work done on recidivism models after Pro Publica's investigation. I suppose the broader issue is that Cathy did not go into depth when I think it could've helped, but I'm sure that was a conscious choice to keep the book popularly readable. One inexcusable omission, however, was overlooking deanonymization, linkage, or membership inference attacks. "Anonymization saves" is perhaps the most widely-held privacy-related misconception in the general public and definitely one of the most powerful big data attacks, and this book does itself a disservice by not bringing it up even once.

Some interesting bits:
1. The "cost of privacy" has become actualized: Those turning off the surveillance-boxes in their car may be eligible for higher premiums on their auto-insurance.
2. The Personalization Paradox: Credit scores don't just score us on our actions but on our socio-economic class which may disadvantage us unfairly. Personalization leads to more accurate scores, but its costs us our privacy.
3. Credit scores in hiring: Credit scores are actively used as a proxy for morality, from hiring to dating.
4. Case-studies for Right to be Forgotten: Multiple case-studies of mistaken identity and people seeking to have their repaid debts "forgotten" by credit-scorers.
5. Douglas Merrill & ZestFinance: Scary show less

Weapons of Math Destruction is one of those books that makes me want to buy a case and send a copy to every person I know. Upton Sinclair once wrote, “It is difficult to get a man to understand something, when his salary depends on his not understanding it.” That is true, but there are exceptional people like mathematician Cathy O’Neil who when they come to understand the pernicious effects of what they do, quit their jobs and dedicate themselves to raising the alarm and working to counter the damage.

Weapons of Math Destruction are defined by O’Neil as those algorithms and analytics that are used to make many of the daily decisions that affect people’s lives. She takes care to clarify #NotAllAlgorithms. What makes an algorithm show more a WMD is when the factors that comprise the algorithm are opaque, unknown to the people affected by it and often even by the people using it. It operates on a large scale, affecting large swaths of people. Worse, it is unaccountable. It does not gather data to see if it makes the right decisions, it does not self-correct. It just runs and lives are changed and no one is the wise to its operations. And most of the time, these algorithms make the poor poorer, the rich richer, and make life harder for those whose lives are already hard.

These algorithms are with us everywhere we turn. They evaluate our teachers, determine prison sentences and parole releases, determine whether we are hired, promoted, fired, insured, surveilled, and how much we pay for things. E-scores even determine how long you wait for customer service, a good score guiding you to a human and a poor one shunting you off to call center hell. Data that shows you don’t shop around results in insurance companies charging you hundreds of dollars more, never offering the discounts they will offer to people whose scores say they comparison-shop. Even in politics, your cookies reveal data that has a politician’s front page show you different pictures and issues than they show your neighbor.

When governments make it illegal for companies to use data like credit scores or race, they turn to other options like e-scores, an unregulated mass of data collection used to create profiles of all of us, sold and used without our knowledge. These e-scores could include false data from other people with similar names and yet result in higher car insurance, worse commercial offers from retailers, higher prices on cars, and a raft of other things.

I loved Weapons of Math Destruction. It is even-handed, pointing out that many of these processes began with good intentions. O’Neil shows that quantitative analysis can be used for good and gives examples. She is not anti-math nor anti-analytics. She wants them to be more transparent and be open to correction. People need to know when algorithms are used against them. We know about credit scores, but there’s so many others, hidden behind “Intellectual Property” rights that allow companies to hide what factors influence our scores. Often these factors have nothing to do with us individually, but with people who seem, on the surface, like us.

Read this book! It is accessible, explaining in easy-to-understand terms exactly what is happening and how it affects us. Accessibility does not sacrifice rigor and a full 30% of the book is devoted to citations that sustain her argument. It is urgent, because every day algorithms encroach on our lives even more. It is fascinating, full of shocking and surprising stories of people whose lives are changed by this, that, or the other score. It has useful tips, such as clearing your cookies before shopping so you are not logged in will result in more discount offers. Most of all, read it because there is really are secret forces, operating under the surface, affecting our lives from birth to death, at work, at home, at play. They will only gain power so long as we are unaware of them.

I received a review copy of Weapons of Math Destruction from Blogging For Books
http://tonstantweaderreviews.wordpress.com/2016/12/16/weapons-of-math-destructio... show less

Summary: An insider account of the algorithms that affect our lives, from going to college, to the ads we see online, to our chances of getting a job, being arrested, getting credit and insurance.

Big Data is indeed BIG. Mathematical algorithms shape who will see this post on their Facebook newsfeed. If you go to Amazon or another online bookseller, algorithms will suggest other books like this one you might be interested in. Have you seen all those ads about credit scores? They are more important than you might imagine. Algorithms used by employers and insurance companies determine your employability and insurability in part through these scores. Far more than another credit card (bad idea, by the way) or a mortgage are on the line. show more These algorithms seem objective, but how they are formulated, and the assumptions made in doing so mean the difference between useful tools that benefit people, and "black boxes" that thwart the flourishing of others, often unknown to them.

Cathy O'Neil should know. A tenure track math professor, she made the jump to Wall Street and became a "quant" who helped develop mathematical algorithms and witnessed, in the crash of 2008, the harm some of these caused. And she began to notice how algorithms often painfully impacted the lives of many others. She describes how a teacher was fired because of the weighting of performance scores of a single class, despite other evaluations finding her an excellent teacher (afterwards it was found that there were a high number of erasures on tests for students who would have been in her class the previous year, suggesting these had been altered to improve scores).

As she looked at the algorithms responsible for such injustices, she came to dub them "Weapons of Math Destruction" or WMDs and she identified three characteristics of these WMDs:

Opacity: those whose lives are affected by them have no idea of the factors and weighting of those factors that contributed to their "score".

Scale: how widely an algorithm is applied across industries and sectors of life can affect how much of one's life is touched by a single formula. For example, the FICO scores mentioned above affect not only credit, but the ability to get a job, the cost of auto insurance, and your ability to rent an apartment.

Damage: WMDs can reinforce other factors perpetuating a cycle of poverty, or incarceration.

She also shows that what makes these algorithms destructive is the use of proxy measurements. For example an employer may not know directly how savvy someone is as a marketer, and so they use a "proxy" measurement of how many Twitter followers that person has. Or age is used as a proxy for how safe a driver one is. For a group, the proxy may work well, and be utterly inaccurate for an individual that falls within that proxy group.

Then in successive chapters, she chronicles some of the ways WMDs operate in different parts of life. She discusses the U.S. News & World Report college rankings, and the use of algorithms in admissions processes. Social media uses algorithms to target advertising, which means some will see ads for for-profit schools and payday lenders, and others for upscale furnishing or Viagra, based on clicks, likes, searches, and comments. Policing strategies, including locations for intensified "stop and frisk" policing are shaped by another set of algorithms. Algorithms to filter resumes may use scoring algorithms that discriminate by address and psych exam algorithms may render others unemployable in a certain industry. Scheduling algorithms may promote efficiency at the expense of the ability of workers to sleep on a regular schedule, or arrange childcare, or work enough hours to qualify for health insurance. Algorithms sometimes shut people out from credit or low cost insurance when in fact they are good risks. She concludes by showing how algorithms determine ads and news we see (and don't see). In an afterword she explores the flaws in algorithms revealed on the election of Donald Trump (algorithms, for example predicted Clinton would easily win Michigan and Wisconsin, where consequently she did not campaign, and lost by small margins).

In her conclusion, she makes the case not only for a code of ethics for mathematicians but also argues that regulation and audits of these algorithms are necessary. The value assumptions, as well as the mathematical methods of many algorithms are flawed, and yet opacity means those whose lives are most affected don't even know what hit them.

She helps us see both the sinister and useful side of these algorithms. They may reveal where a pro-active intervention may save a family from descending into family violence, or provide extra assistance to a child in danger of falling behind in a key subject. Or they may be used to invade personal rights, or even to perpetuate socio-economic divides in a society. The reality is that the problem is not the math but the old GIGO problem (garbage in, garbage out). The values and assumptions of the humans who devise the formulas and weightings of values and the use of proxies determine what may be destructive outcomes for some people. Yet it can be hidden behind an app, a program, an algorithm.

The massive explosion in storage capacities, processing speeds, and the way our interests, health status, travel patterns, spending patterns, fitness, diet and sleep habits, our political inclinations and more may be tracked via our online and smartphone usage makes O'Neil's warning an urgent one. We create mountains of data that may be increasingly mined by government and private interests. Perhaps as important as asking whether this will be governed in ways that contribute to our flourishing, is whether we will be alert enough to care.

____________________________

Disclosure of Material Connection: I received this book free from the publisher. I was not required to write a positive review. The opinions I have expressed are my own. show less

I, too, was a mathematician once, but I lost my faith. Ms. O'Neil still seems to have much of hers, for an Occupier. I kept thinking she was naive that this stuff is fixable, but I may just be naively paranoid.

That algorithms can be biased was not a surprise to me. Logic itself can be biased because it is dependent on language which, like many WMDs, is a black box. It is full of proxies. Take the term "criminal" which (like "terrorist") brands the one so called as an evil doer. And it's measurable by determining if one has been convicted of anything. You can read The New Jim Crow: Mass Incarceration in the Age of Colorblindness and discover that going to jail is part of systemic racism but when you hear the word "criminal" or "convict", show more you usually don't think much further.

Or consider how we think about patents. It's supposed to protect inventors and most people think that's what it's doing (and perhaps much of the time it is) but it has become a weapon of big companies to keep small ones from entering a field.

This is all before mathematics enters into it.

When reasoning is done by a computer, it's faster, bigger, and less reflective. You might think the marketplace can fix this. A business that does less error-prone reasoning will make more money and prevail. Ms. O'Neil tries to explain why it doesn't work this way but I don't think she does that good a job of it. At least she tries.

The problem is that even with a sloppy algorithm, you can make up the difference on volume. In the world of actual weapons of mass destruction, you could nuke entire countries and win the war. Most of the people you killed are "innocent" except in the sense of having been born into the wrong nation. Businesses using "approximate" algorithms will succeed and success is self-validating. On an individual level we say "life is unfair" but on a corporate level the winners selected by the marketplace are seen as deserving their success if they haven't broken any laws which anyone noticed. (And if someone noticed? Google "war on whistleblowers.")

Ms. O'Neil thinks that this is just the birth throes of a new technology and in the future, we'll look back at it like we do at sweat shops and child labor, excesses that we have managed to overcome. I am less confident of our rosy future. I'm more like Chris Hedges (one of whose books I read right before this one) who sees morality not as a dimension of society that progresses like technology and science, but one which remains at pretty much a constant level. I see the technology as multiplying our moral failings in a way that will be difficult to correct. (I am reminded of those who believe that those who fear global warming are underestimating man's ability to solve problems and some yet unforeseen discovery will come along when we need it, and we can just ignore it for now.)

Currently, outside of dystopian science fiction, people see apps as a universal good and trust our cyber-overlords. Then, some may find themselves among the collateral damage, and if sufficiently hurt, will discover no one will want to listen to them. Isn't it an axiom of capitalism that, with some exceptions we are safe to ignore, the poor deserve to be poor? show less

A pretty thought-provoking read, even if you're already familiar with the idea. She unearths so many ways that algorithms affect our lives, without us even realizing it, and her central thesis is strong & clear: These things can be made to help us, or to harm us, but they are not neutral. She advocates powerfully for transparency and oversight (typically in the form of auditing) in the process, which really would benefit all stakeholders. And she reminds us again & again that we need to constantly consider our true priorities. When we don't think carefully about it, we will often tend to value monetary gains over justice & fairness. But if we take the time to assess what we're doing, it's usually possible to alter that calculus, and use show more algorithms in ways that suit our true societal values.

Definitely recommend this. show less

Finding a book authored by an Ivy League mathematician that is both interesting and relatable to non-mathematicians is always a nice surprise. The information O'Neil provides on often-unregulated data brokering and predictive software programs is fresh, crucial, and backed up by many examples of actual and potential problems (as well as assorted proposed solutions) for American consumers and business leaders of all types. Even if number crunching isn't your bag, I recommend reading it to gain awareness of new computational technology trends that directly impact your wallet--and your privacy--in invisible ways. Should definitely be on educator/criminal justice/finance major reading lists. I'd probably have rated it higher, but she lost a show more couple notches with me for obvious lapses in political impartiality in a book devoted to quantitative mathematical science. show less