The Alignment Problem: Machine Learning and Human Values

Well it took me a while to work my way through this book. Part of the sales promotion suggests that if you only read one book on AI, it should be this one. And yes, I've read a whole host of books recently about the issues of algorithms that have been trained on one set of limited data....(eg white American Males) and then applied the learnings via AI to the population at large....inevitably there are real problems. This book seems to go into the issue in great depth.
Putting it my my own terms, the book is really about getting AI to align with what we really want it to do........Or to deliver the outcomes that we would like it to deliver. And, it seems that this is really difficult. It's often difficult to spell out exactly what we show more want. It's even more difficult to anticipate what direction AI might take as it follows instructions exactly .......frequently delivering results that we didn't anticipate or desire.
I was intrigued by the fact that when some version of AI was trained on the language spoken in 2016 it ran into some problems because the language had changed by 2017....and even more so in 2018.
I actually wrote a Masters thesis on the Philosophy of values and I'm rather aware that the terminology around values, is a "mess". But Christian appears to be oblivious to this and plunges into the field without ever defining (or seeming to understand) exactly what he or others take "values" to be.....though there are some allusions there.
There is a lot of material: a lot of detail and, for my own learning, I've extracted some key notes as follows:
"Because the system transformed the words it encountered into numerical representations called vectors, Google dubbed the system “word2vec,” and released it into the wild as open source.....Because word2vec made words into vectors, it enabled you to do math with words.....For instance, if you typed China + river, you got Yangtze. If you typed Paris − France + Italy, you got Rome. And if you typed king − man + woman, you got queen.
An initial look at the data [on crime convictions and re-offence] suggested something might be wrong.....From the crystal ball of 2016, they also knew that Fugett, the 3/10 risk, went on to be convicted of three further drug offenses. Over the same time period, Packer, the 10/10 risk, had a clean record.....A statistical analysis appeared to affirm that there was a systemic disparity. The article ran with the logline “There’s software used across the country to predict future criminals. And it’s biased against blacks.”
The real game he and his fellow researchers are playing isn’t to try to win boat races; it’s to try to get increasingly general-purpose AI systems to do what we want, particularly when what we want—and what we don’t want—is difficult to state directly or completely.
This is a book about machine learning and human values: about systems that learn from data without being explicitly programmed, and about how exactly—and what exactly—we are trying to teach them.....The field of machine learning comprises three major areas:
In supervised learning, the system is given a series of categorized or labeled examples—like parolees who went on to be rearrested and others who did not—
And in reinforcement learning, the system is placed into an environment with rewards and punishments—like the boat-racing track with power-ups and hazards—and told to figure out the best way to minimize the punishments and maximize the rewards.....There is a growing sense that more and more of the world is being turned over, in one way or another, to these mathematical and computational models.....They are steadily replacing both human judgment and explicitly programmed software of the more traditional variety......This is happening not only in technology, not only in commerce, but in areas with ethical and moral weight.
In recent years, alarm bells have gone off in two distinct communities. The first are those focused on the present-day ethical risks of technology. If a facial-recognition system is wildly inaccurate for people of one race or gender but not another, or if someone is denied bail because of a statistical model that has never been audited and that no one in the courtroom —including the judge, attorneys, and defendant—understands, this is a problem.....we will find ourselves more and more often in the position of the “sorcerer’s apprentice”: we conjure a force, autonomous but totally compliant, give it a set of instructions, then scramble like mad to stop it once we realize our instructions are imprecise or incomplete—lest we get, in some clever, horrible way, precisely what we asked for.
How to ensure that these models capture our norms and values, understand what we mean or intend, and, above all, do what we want—has emerged as one of the most central and most urgent scientific questions in the field of computer science.......It has a name: the alignment problem.....This is a story in three distinct parts.
1. Part one explores the alignment problem’s beachhead: the present-day systems
2. Part two turns the focus to reinforcement learning, as we come to understand systems that not only predict, but act;
3. Part three takes us to the forefront of technical AI safety research,
I. Prophecy
In the summer of 1958, a group of reporters are gathered by the Office of Naval Research in Washington, D.C., for a demonstration by a twenty-nine-year-old researcher at the Cornell Aeronautical Laboratory named Frank Rosenblatt. Rosenblatt has built something he calls the “perceptron,” and in front of the assembled press corps he shows them what it can do.
The basic training procedure for the perceptron, as well as its many contemporary progeny, has a technical-sounding name—“ stochastic gradient descent”—.....Where there is a difference between what you wanted and what you got, then figure out in which direction (“ gradient”) to adjust each weight—Move each of them a little bit in the appropriate direction (“ descent”). Pick a new example at random, and start again. Repeat as many times as necessary......This is the basic recipe for the field of machine learning—Minsky and Papert show, with the stiff formality of mathematical proof, that there are seemingly basic patterns that Rosenblatt’s model simply will never be able to recognize....“There had been several thousand papers published on perceptrons up to 1969,” says Minsky. “Our book put a stop to those.”). ....By 1973, both the US and British governments have pulled their funding support for neural network research......By the 1980s it became understood that networks with multiple layers (so-called “deep” neural networks) could, in fact, be trained by examples just as a shallow one could.“I now believe,” admitted Minsky, “that the book was overkill.”
By the 1990s, LeCun’s networks were processing 10 to 20% of all checks in the United States.
As Hinton would later summarize, “Our labeled datasets were thousands of times too small. [And] our computers were millions of times too slow.” Both of these things, however, would change......In 2007, Princeton professor Fei-Fei Li used Amazon Mechanical Turk to recruit human labor, at a scale previously unimaginable, to build a dataset that was previously impossible. It took more than two years to build, and had three million images, each labeled, by human hands.......Only later, in the mid-2000s, did it come to be appreciated that the GPU could do a lot more than light and texture and shadow. It turned out that this hardware, designed for computer gaming, was in fact tailor-made for training neural networks.
Krizhevsky with the neural network trained in his bedroom—its official name is “SuperVision,” but history will remember it simply as “AlexNet”—made half as many errors as the model that came in second.
{With image recognition software] The album cover was a selfie of Alciné and a friend of his. Alciné is Haitian-American; both he and his friend are Black. “Gorillas,” it said....Within two hours, Google + chief architect Yonatan Zunger reached out. “Holy fuck,” he wrote. “This is 100% Not OK.”.....Alciné knew the issue wasn’t a biased algorithm. (The algorithm was stochastic gradient descent, just about the most generic, vanilla, all-purpose idea in computer science:...“It’s not even the algorithm at fault. It did exactly what it was designed to do.” The problem, of course, with a system that can, in theory, learn just about anything from a set of examples is that it finds itself, then, at the mercy of the examples from which it’s taught.
Douglass wrote. “It seems to us next to impossible for white men to take likenesses of black men, without most grossly exaggerating their distinctive features.”....The average white photographer does not know how to deal with colored skins and having neither sense of the delicate beauty or tone nor will to learn, he makes a horrible botch of portraying them.”
“Though the available academic literature is wide-ranging, it is surprising that relatively few of these scholars have focused their research on the skin-tone biases within the actual apparatuses of visual reproduction.”.....For decades, film manufacturers and film developers used a test picture as a color-balance benchmark. This test picture became known as the “Shirley card,”....The chemical processing of film was tuned accordingly, and as a result cameras simply didn’t take good photos of Black people. “A good VCR person will have a color girl stand in front of the cameras and stay there while the technicians focus on her flesh tones to do their fine adjustments to balance the cameras. This color girl is always white.”)....In time Kodak began using models of more diverse skin tones. “I started incorporating black models pretty heavily in our testing, and it caught on very quickly,” recalls Kodak’s Jim Lyon. “I wasn’t attempting to be politically correct. I was just trying to give us a chance of making a better film, one that reproduced everybody’s skin tone in an appropriate way.”
All machine-learning systems, from the perceptron onward, have a kind of Shirley card at their heart: namely, the set of data on which they were trained....Unfortunately, it’s true by definition that there is always proportionately less data available about minorities. This means that our models about minorities generally tend to be worse than those about the general population.”.....Once developed, a biased system has the potential for widespread impact. If the system becomes a standard in the field, the bias becomes pervasive.”
A major movement in rooting out bias, then, is trying to better expose, and better understand, the training datasets behind major academic and commercial machine-learning systems......One of the more popular public-domain databases of pictures of faces, for instance, is what’s known as the Labeled Faces in the Wild (LFW)....In 2014, Michigan State’s Hu Han and Anil Jain analyzed the dataset and determined it was more than 77% male, and more than 83% White.....There are more than twice as many images of George W. Bush in the LFW dataset as there are of all Black women, combined.......[We ought to feel scepticism] when a company announces that their system is, say, “99% accurate”: Accurate on what? Accurate for whom?
Was there a better way? There was, and it came in the form of what are called “distributed representations.” The idea was to try to represent words by points in some kind of abstract “space,” in which related words appear “nearer” to one another. A number of techniques emerged over the 1990s and 2000s for doing this, but one in particular in the past decade has shown exceptional promise: neural networks. ....When our model guesses wrong, we’ll adjust the coordinates of our word representations to slightly nudge the correct word toward the context words in our mathematical space and slightly nudge any incorrect guesses away.......You can do no more than set up this kind of prediction objective, make it the job of every word’s word vectors to be such that they’re good at predicting the words that appear in their context or vice-versa—you just have that very simple goal—and you say nothing else about how this is going to be achieve.......And out come these word vectors that are just amazingly powerful at representing the meaning of words and are useful for all sorts of things. .......Word-embedding models like these, including Google’s word2vec and Stanford’s GloVe, subsequently became the de facto standard for computational linguistics, undergirding since roughly 2013 almost every application that involves computer use of language, be it ranking search results, translating passages from one language to another, or analyzing consumer sentiment in written reviews.....The embeddings, simple as they are—seemed to capture a staggering amount of real-world information.....You could, for instance, simply add two vectors together to get a new vector, and search for the nearest word.....Czech + currency = koruna......And you could subtract words, too. This meant—incredibly—you could produce “analogies” by getting the “difference” between two words and then “adding” it to a third.....Berlin − Germany + Japan = Tokyo....Unfortunately, that wasn’t all the vectors captured. They contained stunning gender biases.....“However, none of these papers have recognized how blatantly sexist the embeddings are and hence risk introducing biases of various types into real-world systems.”....A system naïvely using word2vec, or something like it, might well observe that John is a word more typical of engineer résumés than Mary. And so, all things being equal, a résumé belonging to John will rank higher in “relevance” than an otherwise identical résumé belonging to Mary.
The problem with machine-learning systems is that they are designed precisely to infer hidden correlations in data.
“We’re thinking, how do we define the best thing?” says Bolukbasi. “They said, ‘Sociologists can’t define what is good.’ As an engineer you want to say, ‘Okay, this is the ideal, so this is my target, so I’m just going to make my algorithm until I reach that target.’ Because it’s involved so much with people and culture and everything, you don’t know what’s optimal. You can’t optimize for something. It’s very hard actually in that sense.”
As the team wrote, “One perspective on bias in word embeddings is that it merely reflects bias in society, and therefore one should attempt to debias society rather than word embeddings. However, . . . in a small way debiased word embeddings can hopefully contribute to reducing gender bias in society......A classic test of unconscious bias in humans used in the social sciences is the “implicit association test,” where subjects will see a sequence of words and are asked to press a button anytime the word belongs to either of two different categories: for instance, a flower (e.g., “iris”) or something pleasant (e.g., “laughter”). It sounds simple enough, and it is; the story is not in accuracy, but in reaction time.
The distance between embeddings in word2vec and other widely used word-embedding models uncannily mirrors this human reaction-time data. The slower people are to identify any two groups of words, the farther away those word vectors were in the model.
The model’s biases, in other words, are, for better or worse, very much our own.
The more strongly a word representation for a profession skews in a gender direction, the more overrepresented that gender tends to be within that profession......Baseline aside, however, there is a consistent trend across time that shows the gender bias in word embeddings for professions moving in lockstep with the change in the workforce itself.
By looking at texts across time, they found a wealth of narratives reflecting social change.
There is a broad assumption underlying many machine-learning models that the model itself will not change the reality it’s modeling. In almost all cases, this is false......Indeed, uncareful deployment of these models might produce a feedback loop from which recovery becomes ever more difficult or requires ever greater interventions.
In 1969 a Scottish-born statistician named Tim Brennan was working for Unilever in London,
“I had a values crisis,” Brennan tells....he noticed that Unilever, had spent more money studying the packaging for its “Sqezy” than the entire British government had spent on literacy.
As the era of the personal computer dawned, the use of statistical models at all points in the criminal justice system, in jurisdictions large and small, exploded. In 1980, only four states were using statistical models to assist in parole decisions. By 1990, it was twelve states, and by 2000, it was twenty-six......In 2001, the state of New York began a pilot program using COMPAS to inform probation decisions......The Times urged wider acceptance of risk-assessment tools in parole......Then—abruptly—the tone changed.......From there the coverage in 2017 only got bleaker—in May, “Sent to Prison by a Software Program’s Secret Algorithms”.........Angwin stayed at the paper for fourteen years—from the dot-com crash through the rise of social networks and smartphones—.....Angwin set about trying to find the most consequential, and overlooked, decisions being made based on data. She landed at criminal justice. Statistical risk assessments, COMPAS and others, were rapidly being adopted in hundreds of jurisdictions:.....not just for parole, but for pretrial detention, bail, and even sentencing. “I was shocked, actually,” she says. “I realized that our whole country was using this software. . . . And then what I found even more shocking was that none of them had independently been validated.”.....The more Angwin learned about risk-assessment models, the more concerned she became......What resulted was the piece that Angwin and her team published in May 2016. Titled “Machine Bias,” it ran with the logline “There’s software used across the country to predict future criminals. And it’s biased against blacks.”
When people were asking for privacy, they were actually worried about somebody using their data in the wrong way. It wasn’t so much about hiding the data at all costs but preventing harm from the way that data was used......Over time, the public discussion shifted from privacy to fairness, and everything that used to look like a privacy problem suddenly looked like a fairness problem.”.......The company or organization behind a model typically defends its model by showing that it “doesn’t use race as an attribute,” or is “race-blind.” This seems intuitive enough—how can something be discriminatory toward a particular group if it doesn’t know who is in that group to begin with?....Simply removing the “protected attribute” is insufficient. As long as the model takes in features that are correlated with, say, gender or race, avoiding explicitly mentioning it will do little good.....Omitting the protected attribute makes it impossible not only to measure this bias but also to mitigate it.....“The most robust fact in the research area,” Hardt says, “is that fairness through blindness doesn’t work.
Chouldechova’s analysis lands exactly in the same place: A tool that is calibrated, she writes, “cannot have equal false positive and negative rates across groups, when the recidivism prevalence differs across those groups.”.....“So you just can’t have it all,” she says. “It’s a general principle......As Sam Corbett-Davies explains, “There isn’t a world in which Pro-Publica couldn’t have found some number that was different that they could call bias. There’s no possible algorithm—there’s no possible version of COMPAS—where that article wouldn’t have been written.”
However, even those who emphasize the importance of calibration think that it alone isn’t enough. As Corbett-Davies says, “Calibration, though generally desirable, provides little guarantee that decisions are equitable.”.....I ask Julia Angwin what she herself makes of the storm of theoretical results that her article prompted, and of the ultimate impossibility of doing what her team seemed to demand—namely, to make a tool both equally calibrated and with an equal balance of false positives and false negatives.....What I’m really happy about is no one knew that that was a question until we came up with it.
A group of criminal justice scholars, write. “We are being presented with the chance of a generation, and perhaps a lifetime, to reform sentencing and unwind mass incarceration in a scientific way and that opportunity is slipping away because of misinformation and misunderstanding about [statistical risk-assessment models].
One of the most important things in any prediction is to make sure that you’re actually predicting what you think you’re predicting. This is harder than it sounds....If a baby lion, let’s say, were repeatedly misidentified by human volunteers as a cat, it would become part of a system’s training data as a cat—and any system labeling it as a lion would be docked points and would have to adjust its parameters to correct this “error.”
One often talks in shorthand of predicting recidivism itself, but that’s not what the training data captures. What the training data captures is not re-offense, but rather re-arrest and reconviction. This is a potentially crucial distinction.....A person who commits crimes in an area that is less aggressively policed, or who has an easier time getting their charges dropped, will be tagged by the system as someone who did not recidivation.
The system begins to sculpt the very reality it is meant to predict. This feedback loop, in turn, further biases its training data.
The locations that are flagged for targeted policing are those that were, by our estimates, already over-represented in the historical police data.” ...“Self-reported marijuana use rates among young Black males and young White males are roughly the same. But the arrest rates for marijuana-related crimes are two and a half to five times higher among Black young males.”...A 2018 investigation by the New York Times found that Black residents of Manhattan were fifteen times more likely than White residents to be arrested on marijuana charges, despite similar rates of use. ....If differential enforcement emboldens the overlooked group more than it deters the scrutinized group, it may only make the problem worse. Alexandra Chouldechova explains: “If you think about it from that perspective, then you’re saying, Okay this particular population, maybe they’re less able to provide for themselves: they actually maybe have lower risk, but higher needs.” Maybe they need day care for their children on their court date or a ride to court—not detention. As it turns out, simply reminding people about their court date can significantly improve their rate of appearance....Unfortunately, many risk assessment tools, unlike COMPAS, conflate a prediction of failure to appear with a prediction of criminal re-offense.
A machine-learning model, trained by data, “is by definition a tool to predict the future, given that it looks like the past. . . . That’s why it’s fundamentally the wrong tool for a lot of domains, where you’re trying to design interventions and mechanisms to change the world.”
Ernest Burgess, writing in 1937—“The time has arrived in Illinois, in my judgment,” he wrote, “to stop tinkering with parole as an isolated part of our penal problem. What is required is a major operation which involves a complete reorganization of the prison system of the state.”
About 10% of pneumonia patients were ultimately dying in USA—and so correctly identifying which patients were at greatest risk would translate fairly straightforwardly into lives saved.
Rich Caruana ‘s group was tasked with building a machine-learning model to predict patient survival rates that could help the hospital triage new patients.....“Even small improvements in predictive performance for prevalent and costly diseases, such as [pneumonia], are likely to result in significant improvements in the quality and efficiency of healthcare delivery.
They deployed one of the simpler models that his neural net had so handily beaten. Here’s why: The correlation that the rule-based system had learned, in other words, was real. Asthmatics really were, on average, less likely to die from pneumonia than the general population. But this was precisely because of the elevated level of care they received. “So the very care that the asthmatics are receiving that is making them low-risk is what the model would deny from those patients,” Caruana explains. “I think you can see the problem here.” A model that was recommending outpatient status for asthmatics wasn’t just wrong; it was life-threateningly dangerous. ....“I said, what I’m worried about is things that the neural net has learned that are just as risky as asthma but the rule-based system didn’t learn.”
It’s often observed in the field that the most powerful models are on the whole the least intelligible, and the most intelligible are among the least accurate.
“I want to do machine learning for health care. Neural nets are really good, they’re accurate; but they’re completely opaque and unintelligible, and I think that’s dangerous now.
A generalized additive model is a collection of graphs, each of which represents the influence of a single variable. For instance, one graph might show risk as a function of age, another would show risk as a function of blood pressure, a third would show risk as a function of temperature or heart rate, and so forth.....These individual one-variable risks are then simply added up to produce the final prognosis. In this way it is more complex by far than, say, a linear regression but also much more interpretable than a neural net....The generalized additive model turns out to be just as accurate as his old neural net, and far more transparent.
I looked at it, and I was just like, ‘Oh my—I can’t believe it.’ It thinks chest pain is good for you. It thinks heart disease is good for you. It thinks being over 100 is good for you. . . . It thinks all these things are good for you that are just obviously not good for you.”...Again it was precisely the fact that these patients were prioritized for more intensive care that made them as likely to survive as they were.
“Everyone is committing these mistakes,” he says, “just like I have committed them for decades, and didn’t know I was doing it.”....Many are finding themselves uncomfortable with how little they know about what’s actually going on inside those models.
The defense community has found itself increasingly thinking about what an automated battlefield might look like—what risks and questions surround the idea of ever more autonomous weapons.
“I’d learned a bit about machine learning, and how some of the best methods don’t really lend themselves to being transparent or interpretable,” he says, “and then I came across this. In the earlier drafts of the GDPR, it was much more explicit. . . . They said people should have the right to ask for an explanation of algorithmically made decisions.”.....Getting intelligible explanations out of a deep neural network is an unsolved scientific problem.
In 1954, Robyn Dawes was an undergraduate philosophy major at Harvard, specializing in ethics. His thesis—“ A Look at Analysis”—investigated whether, and to what degree, moral judgments were rooted in emotion. He then moved into Psychology and looked at some older work by Ted Sarbin. ...Sarbin looked at predictions of academic performance for incoming freshmen at the University of Minnesota. The “actuarial” model was a simple linear regression to predict their college GPA from just two data points:....The human predictions were made by experienced clinical psychologists who had access to these two data points, plus additional tests, an eight page dossier, notes from a colleague’s interview, and their own firsthand impression of the student....Sarbin found no measurable difference between the two predictions. If anything, the actuarial model was more accurate.....the human counsellors made their predictions chiefly on the basis of class rank and test scores—the very same data used in the regression model. They just weren’t as consistent or finely tuned in how they weighted it.....Sarbin’s conclusion was that the time-intensive effort spent in conducting interviews was a waste.....Thirty years after Sarbin’s original paper, and many dozens of studies later, he concluded, “A search of the literature fails to reveal any studies in which clinical judgment has been shown to be superior to statistical prediction when both are based on the same codable input variables” (emphasis mine).....Even if a model was trained only to mimic a single expert’s judgments, it still outperformed the expert themselves!..Given the complexity of the world, why on earth should such dead-simple models—a simple tally of equally weighted attributes—not only work but work better than both human experts and optimal regressions alike?
Despite the enormous complexity of the real world, many high-level relationships are what is known as “conditionally monotone”—they don’t interact with one another in particularly complex ways. Regardless of whatever else might be happening with a person’s health, it’s almost always better if that person is, say, in their late twenties rather than their late thirties.
Second, there is almost always error in any measurement. For theoretical as well as intuitive reasons, the more error-prone a measurement is, the more appropriate it is to use that measurement in a linear fashion......Dawes’s point. As he wrote: “The linear model cannot replace the expert in deciding such things as ‘what to look for,’ but it is precisely this knowledge of what to look for in reaching the decision that is the special expertise people have.” ....It was Dawes’s conclusion that human expertise is characterized by knowing what to look for—and not by knowing the best way to integrate that information....“The whole trick is to know what variables to look at,” he wrote, “and then to know how to add.”
Rudin and her colleagues published a paper in 2018 showing that they could make a recidivism-prediction model as accurate as COMPAS that could fit into a single sentence: “If the person has more than three prior offenses, or is an 18-to-20-year-old male, or is 21-to-23 years old and has two or more priors, predict they will be rearrested; otherwise, not.”
So simple models, made from hand-selected high-level variables, perform about as well as more complex models—sometimes better—and consistently as well as or better than human experts.
As it happens, finding optimal simple rules is not for the faint of heart. In fact, it requires tackling an “intractable,” or “NP-hard” problem: a thicket of complexity in which there is no straightforward means of obtaining the guaranteed best answer.....Given tens of thousands of patient records, each with dozens or perhaps hundreds of different attributes—age, blood pressure, etc.—how do you find the best simple flow chart for diagnosis?
Rudin let her algorithm, called Bayesian Rule Lists, loose on a set of 12,000 patients, to pore over some 4,100 different properties for each—every drug they were taking, every health condition they had reported—to make the best possible scoring system. She then compared her own model to both CHADS2 and CHA2DS2-VASc against held-out portions of that same dataset......The results showed a marked improvement over both CHADS2 and CHA2DS2-VASc.....In a subsequent project, Rudin and her PhD student Berk Ustun worked with Massachusetts General Hospital to develop a scoring system for sleep apnea,
Even into the twenty-first century, it was not uncommon for practitioners to simply come up with an ad hoc model based on their own intuition. This is sometimes derisively referred to as the “BOGSAT method”: a bunch of guys sitting around a table....First, the model showed—contrary to received wisdom and current practice—that patient symptoms were significantly less useful than their histories.....What’s more, adding symptoms to the model based on histories didn’t register much of an improvement......“SLIM accuracy was similar to state-of-the-art classification models applied to this dataset,”....“but with the benefit of full transparency, allowing for hands-on prediction using yes/ no answers to a small number of clinical queries.”....I [Rudin] want to create predictive models that are highly accurate, yet highly interpretable, that we can use for trustworthy decision making.
Some models must, for better or worse, deal not with human abstractions like “GRE score” and “number of prior offenses” but with raw linguistic, audio, or visual data. Some medical diagnostic tools can be fed human inputs, like “mild fever” and “asthmatic,” while others might be shown an X-ray or CAT scan directly and must make some sense of it.....In such cases we have little choice but the kinds of large, multimillion-parameter “black box” neural networks that have such a reputation for inscrutability.....But we are not without resources here as well, on the science of transparency’s other, wilder frontier.
It might be understandable, then, for us to want to expect something similar from our machines: to know not only what they think they see but where, in particular, they are looking.....This idea in machine learning goes by the name of “saliency”: the idea is that if a system is looking at an image and assigning it to some category, then presumably some parts of the image were more important or more influential than others......In 2013, Portland State University PhD student Will Landecker was working with a neural network trained to distinguish images in which an animal was present from those with no animals present.
As it turns out, he hadn’t trained an animal detector at all. He’d trained a bokeh detector.
In 2015 and 2016, dermatologists Justin Ko and Roberto Novoa led a collaboration between researchers from Stanford’s medical and engineering schools.....They retrained their network to tell the difference, not between Chihuahuas and Labradors, but between acral lentiginous melanoma and amelanotic melanoma, and thousands of other conditions.
They tested their system against a group of twenty-five dermatologists. The system outperformed the humans.....A cautionary tale from their own experience. The vision system they were using was much more likely to classify any image with a ruler in it as cancerous. Why? It just so happened that medical images of malignancies are much more likely to contain a ruler for scale than images of healthy skin. “Thus the algorithm inadvertently ‘learned’ that rulers are malignant.”....The network could be used to make not just a single prediction—say, whether the patient would live or die—but potentially dozens: how long they’d stay in the hospital, how large their bill would be, and so on......If you had a multitask net predicting all sorts of things from the data—not just death but length of hospital stay or dollar cost of treatment—these anomalies would be much more visible. The asthmatics, for instance, might have better-than-average morbidity but astronomical medical bills. It would be much clearer that these were no ordinary “low-risk” patients to be sent home with instructions to take two pills and call back in the morning.
A team involving Stanford’s school of Medicine was adapting Google’s Inception v3 network to classify images of the retina....And wondered what if they treated this trove of ancillary data—age, sex, blood pressure, etc.—not as additional inputs to the model, but as additional outputs? It might offer a way to make the model more robust. They were in for an enormous shock. The network could almost perfectly tell a patient’s age and sex from nothing but an image of their retina....Age, as it happened, was determined by the model looking mostly at the blood vessels; sex, in contrast, by looking at the macula and the optic disc....By showing where in the image the model is focusing to make its prediction, it really does provide a level of trust and also, you know, a level of validity to the results.”....The combination of multitask learning and saliency techniques showed the field that there were sex differences in the retina that had been overlooked. Not only that; it showed where to find them.
People knew that the bottommost layer of a convolutional network represented basic things: vertical edges, horizontal edges, diagonal edges, a strong single color, or a simple gradient. And it was known that the final output of these networks was a category label: cat, dog, car, and so forth. But it wasn’t really known how to interpret the layers in between.
For the first time they were seeing the second layer. It was a menagerie of shapes. “Parallel lines, curves, circles, t-junctions, gradient patterns, colorful blobs: a huge variety of structure is present already at the second layer.” The third layer was even more complex, beginning to represent portions of objects: things that looked like parts of faces, eyeballs, textures, repeated patterns. It was already detecting things like the white fluff of a cloud, the multicolor stripes of a bookshelf, or the green comb of grass....By the fifth layer, the ultimate categories into which objects were being assigned seemed to exert a strong influence.
In 2015, Google engineers Alexander Mordvintsev, Christopher Olah, and Mike Tyka experimented with a method of starting from an image of random static, and then tweaking its pixels to maximize the probability that the network assigns it a particular label—say, “banana” or “fork.”....It results in fascinating, memorable, often psychedelic, and occasionally grotesque images.....If you optimize for some combination of the obscenity filter and normal ImageNet category labels—for instance, volcanoes—you get, in this case, obscene geography: what look like giant granite phalluses, ejaculating clouds of volcanic ash. Such images are, for better or worse, not easily forgotten......If starting from random static and fine-tuning hundreds of images to maximize the “face” category produces a set of faces that are, say, exclusively white and male, then that’s a pretty good indication that the network won’t recognize other types of faces as readily.
Olah found that traditional scientific journals just weren’t suited for the kinds of rich, interactive, full-color and high resolution visualizations he was making. So he launched a new one.
Recognizing the ineluctably human aspect of interpretability means that things don’t always translate neatly into the familiar language of computer science.....“Some folks think that you have to put down a mathematical definition of what explanation must be.....“Something that is not quantifiable makes computer scientists uncomfortable—inherently very uncomfortable.”.....“Iterating with the users is critical”......This iteration is critical because often what designers think is useful to actual human users simply isn’t. If you’re designing explanations or interpretable models to be used by real people, then the process should be every bit as iterative as designing, say, cockpit controls.
One of Kim’s beliefs is that “humans think and communicate using concepts,” not numbers. We communicate—and, for the most part, think—verbally, leveraging high-level concepts; We don’t talk about the raw minutiae of sensory experience.
Google CEO Sundar Pichai during his keynote address at Google’s 2019 I/ O conference said “It’s not enough to know if a model works,”..... “We need to know how it works.”...By 2017, there were entire symposia at the field’s largest conferences devoted to interpretability and explanation. By 2019, the CEO of Google was proudly describing her work on the company’s biggest stage.
II. Agency
Thorndike [around 1898] sees here the makings of a bigger, more general law of nature. As he puts it, the results of our actions are either “satisfying” or “annoying.” When the result of an action is “satisfying,” we tend to do it more. When on the other hand the outcome is “annoying,” we’ll do it less.....Thorndike calls this idea,....“the law of effect.”...From this seemingly modest and intuitive idea will be built much of twentieth-century psychology.
A road map to artificial intelligence, then, was already taking shape. The “unorganized machines” would borrow directly from what was known about the nervous system, and the “course of education” would borrow directly from what the behaviourists were discovering about how animals (and children) learned.
Turing had begun to sketch out ways that such a network might be trained through trial and error. Indeed, this was precisely the process of “stamping in” that Thorndike had described fifty years before.....The unveiling of Samuel’s research [end of 1950’s] became the stuff of computer science legend. Fellow AI pioneer John McCarthy recounts that when Samuel was getting ready to demonstrate his checkers program on national television, “Thomas J. Watson Sr., the founder and President of IBM, remarked that the demonstration would raise the price of IBM stock 15 points. It did.”.....In 1972, Harry Klopf— argued that “the neuron is a hedonist”: one that works to maximize some approximate, local notion of “pleasure” and minimize some notion of “pain.”.....For the cyberneticists, purpose was tantamount to a goal that could be arrived at as a place of rest.......“All purposeful behaviour,” the cyberneticists wrote, “may be considered to require negative feed-back.”....Klopf was having none of it. For him, organisms were maximizers, not minimizers. Life was about growth, reproduction, endless and boundless and insatiable forward progress in any number of senses. For Klopf, the goal was not homeostasis at all, but the opposite......“Living adaptive systems seek, as their primary goal, a maximal condition (heterostasis), rather than . . . a steady-state condition (homeostasis).”....He wrote. “Neurons, nervous systems, and nations are heterostats.”
Barto & Sutton,” “Sutton & Barto”—would become synonymous with the field of reinforcement learning itself.....Barto and Sutton took Harry Klopf’s idea of organisms as maximizers and gave it a concrete, mathematical form.....As long as the rewards are what is known as scalar: they are commensurate, fungible, of a common currency.....It has led to an idea known as the “reward hypothesis”: “That all of what we mean by goals and purposes can be well thought of as the maximization of the cumulative sum of a received scalar reward.”
We often have to make decisions whose outcomes seem like apples and oranges. Do we work late, improving our standing with our boss but testing the patience of our spouse?
Ruth Chang, for instance, has spent decades arguing that nothing so characterizes the human condition as the incommensurability of the various motives and goals we have.
Sutton himself concedes that the reward hypothesis is “probably ultimately wrong, but so simple we have to disprove it before considering anything more complicated.”
The first challenge is that our decisions are connected. Here reinforcement learning is subtly—but importantly—different.....In reinforcement learning—every decision we make sets the context in which our next decision will be made—A reinforcement-learning system, trying its best to maximize some quantity in some environment, eventually comes to learn what score it achieved, but it may never know, win or lose, what the “correct” or “best” actions should have been.....As Andrew Barto puts it, reinforcement learning is less like learning with a teacher than learning with a critic......The critic may be every bit as wise, but is far less helpful.......Third, not only is feedback terse and not especially constructive, it’s delayed. We may make an unrecoverable blunder on the fifth move of a game, for instance, in which the coup de grâce comes a hundred moves later......In time, studies began to establish that the areas of the brain in which this electrical stimulation was most compelling were those areas involving neurons that produced a neurotransmitter called 3,4-dihydroxyphenethylamine—better known by its abbreviated nickname: dopamine.....They were almost uniquely broadly connected, with the most highly connected cells each having nearly fifteen feet of axonal wiring within the brain.....Could it be that dopamine was literally the molecular currency of reward in the brain?
Policy-based approaches led to a system—be it animal, human, or machine—with highly trained “muscle memory.” The right behaviour just flowed effortlessly. Value-based approaches, by contrast, led to a system with a highly trained “spider-sense.” It could tell right away if a situation was threatening or promising. Either, alone, if fully developed, was enough......As Sutton reasoned, developing good expectations—a good value function—meant reconciling your moment-to-moment expectations with the ultimate verdict that came from reality:...But if you actually had to wait until the end of a game to learn from it, then the credit-assignment problem would indeed be virtually impossible. The logic, he says, is threefold.
1. First, it may be impractical or impossible to remember everything
2. Second, we want to be able to learn even without a final verdict.
3. Third, we ideally want to be learning not just after the fact but as we go along.
Each of these factors, as Sutton explains, makes a theory of expectations more tricky. Guesses fluctuate over time, and in general they get more accurate the closer you are to whatever it is you’re trying to predict.....Cambridge PhD student Chris Watkins had devised a TD algorithm called “Q-learning,” which would turn these predictions into actions.
A young researcher named Gerald Tesauro—in 1992, plugged TD learning into his model, and it took off like a rocket....It was learning guesses from guesses, steadily coming to learn what an advantageous position looked like.
Dayan and a fellow postdoc named Read Montague, working with Salk Institute neuroscientist Terry Sejnowski, had a hunch that not only did the framework of reinforcement learning explain how real human and animal brains might operate—but that it might literally be what brains were doing. “We went after the role of a set of systems in your brain that report on value and reinforcement,” says Montague.....A sudden spike above the brain’s dopamine background chatter meant that suddenly the world seemed more promising than it had a moment ago. A sudden hush, on the other hand, meant that suddenly things seemed less promising than imagined. The normal background static meant that things, however good or bad they were, were as good or bad as expected..... it was its brain learning a guess from a guess?......Temporal-difference learning didn’t just resemble the function of dopamine. It was the function of dopamine.
Schultz, Dayan, and Montague published an explosive paper together in Science in 1997, announcing their discovery to the world. They had found, as they put it, “A Neural Substrate of Prediction and Reward.”....The effect on neuroscience has been transformative. As Princeton’s Yael Niv puts it, “The potential advantages of understanding learning and action selection at the level of dopamine-dependent function of the basal ganglia can not be exaggerated:....Elevated levels of dopamine signal something to the effect of things are going to be better than I thought they were going to be, then that feeling is, itself, pleasurable. And you can see how humans and animals alike would go out of their way to get that feeling, including by way of direct chemical and electrical stimulation of dopamine neurons.....Thinking that things will be better than you thought they would be only works for so long. Eventually you realize they’re not better than you thought they would be, and the dopamine chatter hushes—This, of course, is the classic experience of dopamine-related drugs—cocaine being a prototypical example. The drug works in large part by inhibiting the brain’s reuptake of dopamine, leading to a temporary “flood” of it. The TD story suggests that the brain interprets this as a pervasive sense that things are going to be great—but the dopamine is writing checks that the environmental rewards can’t cash. Eventually the predicted greatness doesn’t come, and the equal and opposite negative prediction error is sure to follow.
As the writer David Lenson puts it, “Cocaine promises the greatest pleasure ever known in just a minute more, if the right image is presented to the eyes, if another dose is administered, if a sexual interaction is orchestrated in just the right way. But that future never comes.....The University of Michigan’s Kent Berridge, for instance, has spent the better part of his career teasing apart the neuroscience of wanting from liking.....People have a stubborn and persistent return to their emotional baseline, regardless of changes in their long-term quality of life.....Lottery winners and paraplegics, famously, are emotionally back to more or less where they started not long after their respective dramatic life changes.....If happiness comes not from things having gone well, not from things being about to go well, but from things going better than expected, then yes, for better or worse, as long as our expectations keep tuning themselves to reality, then a long-term state of being pleasantly surprised should be simply unsustainable.....“Lowering expectations increases the probability of positive outcomes. . . . However, lower expectations reduce well-being before an outcome arrives, limiting the beneficial scope of this manipulation.”....Dopaminergic “happiness” comes in large part from being pleasantly surprised,
Then complete mastery of any domain seems necessarily correlated with boredom—a point that has ethical implications....Reinforcement learning also offers us a powerful, and perhaps even universal, definition of what intelligence is.
This theory, pointedly, does not tell us what we value, or what we ought to value....If, behind door number one, is not the Caribbean vacation we expected but, rather, a trip to view the aurora borealis, our dopamine will quickly and reliably indicate whether we are pleasantly or unpleasantly surprised by this. But how is the value of those alternatives actually being assessed? Dopamine is mum on this point.
Given the behaviour we want from our machines, how do we structure the environment’s rewards to bring that behaviour about? How do we get what we want when it is we who sit in the back of the audience, in the critic’s chair—we who administer the food pellets, or their digital equivalent?
This is the alignment problem, in the context of a reinforcement learner.
[I've skipped from this point to the conclusions because of lack of space. But the conclusions are fairly comprehensive].
Conclusion
“I think vagueness is very much more important in the theory of knowledge than you would judge it to be from the writings of most people.....When you pass from the vague to the precise . . . you always run a certain risk of error”.—BERTRAND RUSSELL.
Don Curtis [who instslled an industrial grade air conditioner in his house and, inadvertently, froze everything] is a perfect example of the problem that comes when increasing our power takes this shielding away. I can’t help thinking of AI as a twenty-ton air conditioner, coming to every home.
Research on bias, fairness, transparency, and the myriad dimensions of safety now forms a substantial portion of all of the work presented at major AI and machine-learning conferences.
Most medical trials are still overwhelmingly done on men. [Hence validity for women is questionable.....The makeup of clinical trials is a double-edged one: even seemingly sensible prohibitions to protect vulnerable groups—not allowing medical trials on pregnant women, for instance, or the elderly—create bias and blind spots.
We need to consider critically, too, not only where we get our training data but where we get the labels that will function in the system as a stand-in for ground truth......ImageNet, for instance, used the judgments of random humans on the internet as the truth.....If most people thought, say, a wolf cub was a puppy, then as far as the image recognition system is concerned, it is a puppy.
ImageNet images each belong to exactly one of a thousand categories. To use this data and the models trained on it, we must accept the fiction that these thousand categories are mutually exclusive and exhaustive.....If we are looking at a picture of, say, a mule, and our labels allow us only to say “donkey” or “horse,” than it must be either a donkey or a horse. It also cannot be ambiguous.
Image recognition systems are often trained with an objective function called “cross-entropy loss”—the numerical details aside, it assigns a penalty for any mischaracterization, no matter which.
In reality, certain types of errors—are probably thousands if not millions of times worse than others.....The geometry of word vectors—the idea that they are represented as distances in mathematical space—makes every analogy symmetrical, in a way that doesn’t always reflect human intuitions about analogy......The human concept of “analogy” turns out to be no less fuzzy and indefinite than any other.
In Chapter 2, we looked at the increasingly widespread use of risk-assessment instruments in the criminal justice system.....We pretend, for the sake of training the model, that we know what a defendant would have done if released. How could we possibly know?...Furthermore, the assumption that incarceration itself has no effect is both likely false.....
To the extent that we want our models to do anything other than repeat and reinforce the past, we need to approach them more deliberately and more mindfully.
Some researchers feel that instead of hashing out these different formalisms, then trying to reconcile them “manually,” we ought rather to simply train a system with examples of things that humans believe are “fair” and “unfair,” and have machine learning construct the formal, operational definition itself........Research shows that humans place greater trust in transparent models even when those models are wrong and ought not to be trusted.
Caution is warranted that we do not simply create systems optimized for the appearance of explanation, or for the sense they give us that we understand them.
Different approaches to reinforcement learning also come with different sets of assumptions.
Many assume that the environment is essentially stable. Many assume both that the agent can’t permanently alter the environment, and that the environment can’t permanently alter the agent. In the real world, many actions change your goals.
Most machine-learning systems presume that they themselves do not affect the world; thus they do not need to model or understand themselves.....This assumption will only get more and more unfounded the more powerful and capable and widespread such agents become.
As AI agents in the world grow ever more sophisticated, they are going to need good models of us to make sense of how the world works and of what they ought and ought not to do.
In our discussion of systems that infer human values and motivations from their behaviour, there are a number of assumptions to unpack. One is that the human or expert is demonstrating “optimal” behaviour. This is, of course, almost never the case.....Even when there is a certain allowance made for error or suboptimality or “irrationality” in human performance, these models nonetheless typically assume that the human is an expert, not a pupil:....the gait of the adult, not the child learning to walk.......The name of this technique—inverse reinforcement learning—is a misnomer. We are making an inference about someone’s goals and values not based on their process of reinforcement learning, but rather from their final behavioural outcome (in technical terms, their “learned policy”).
Perhaps most consequentially, typical inverse reinforcement learning systems imagine there is but one person whose preferences are being modeled. How, exactly, are we to scale this to systems that are, in some sense, the servant of two (or more) masters?
As Stanford computer scientist Stefano Ermon puts it, aligning AI with human values “is something that I think the majority of people would agree on, but the issue, of course, is to define what exactly these values are, because people have different cultures, come from different parts of the world, and have different socioeconomic backgrounds, so they will have very different opinions on what those values are. That’s really the challenge.”....University of Louisville computer scientist Roman Yampolskiy concurs, stressing, “We as humanity do not agree on common values, and even parts we do agree on change with time.”....[I might add that the whole language around values is inconsistent and basically a mess...whole books have been written on value language as I discovered when I did a Masters thesis around axiology....the philosophy or study of values].
Every machine-learning architecture is implicitly resting on a kind of transfer learning at several levels. It assumes that the situations it encounters in reality will resemble, on average, what it encountered in training.....One of the simplest violations of this assumption, however, is the world’s stubborn and persistent tendency to change.
In one example, the training data were from 2016. The English being written and spoken in 2017 was slightly, but measurably, different.....The English of 2018 was more different still.
As Bruno Latour writes, “We have taken science for realist painting, imagining that it made an exact copy of the world. The sciences do something else entirely—Through successive stages they link us to an aligned, transformed, constructed world.”
We must take great care not to ignore the things that are not easily quantified or do not easily admit themselves into our models. The danger, paraphrasing Hannah Arendt, is not so much that our models are false but that they might become true......As we’ve seen, the outbreak of concern for both ethical and safety issues in machine learning has created a groundswell of activity.
We have also seen how the project of alignment, though it contains its own dangers, is also tantalizingly and powerfully hopeful.
Biased and unfair models, if deployed haphazardly, may deepen existing social problems, but their existence raises these often subtle and diffuse issues to the surface, and forces a reckoning of society with itself. Unfair pretrial-detention models, for one thing, shine a spotlight on upstream inequities.
So what's my overall take on the book? I must say that I'm very impressed. He's covered a vast and rapidly growing field in considerable depth. Seems to have talked to everybody and teased out the issues.....splicing the whole together as a kind of history of AI. I must confess that despite researchers seemingly getting closer and closer to alignment between machine learning and what we would desire from it......that I'm not confident that we are there yet. Nor am I confident that we will know when AI gives us an answer, whether that is the correct answer and/or whether it might have undesirable side effects. Still, quite fascinating. An easy five stars from me. show less

There was lump in my throat when Deep Mind’s AlphaGo crushed Lee Sedol at Go, the oldest (3000 year old) arguably most complex strategic board game , cause with that AI not just defeat the greatest player ever but effectively wiped any future association of GO and Humans . No Human will even beat AI at GO again period, that fortress is breached! We have essentially been relegated to a mere factoid in the timeline of this planet.

While capitalism will ensure the inevitability that humans will be pushed “out of the loop” in every aspect – The question is not if but when . Brian Christian’s Alignment Problem educates the reader with the real pitfalls of depending on algorithms and inherent drawbacks of machine learning . Brian show more more than Nick Bostrom’s – Super Intelligence in my opinion dwells much deeper on the alignment problem at hand ; Bostrom set the stage for AI safety and was labelled as an alarmist ; well not anymore .

From dopamine exploiting social media algorithms to parole sentences to mortgage application approvals ; these highly pervasive machine learning algos now control various aspects of humans , while Congress grapples with legislation & red-tape .

The book gives an over arching view on how the ML algos came about around the following “pillars” curiosity, imitation, reinforcement, model bias , bad data samples etc. and why it is crucial to align AI goals with Human values .

And as often is the case the problems are more of the philosophical nature than anything , this also highlights the importance of psychology , social anthropology , neurophysiology and psychoanalysis playing a quintessential part in future development of this nascent field .The latter part of the book deals with possibly the tougher questions which AI posses ; happy to see the Effective Altruism movement founder Will MacAskell get a page in there too . show less

An impressive, conversation-based analysis of how AI systems developed through processes of machine learning (ML) might be constrained to be both safe and ethical. I had little idea of how rich and massive the research on this has been. In nine chapters with carefully chosen one-word headings (Representation, Fairness, Transparency, Reinforcement, Shaping, Curiosity, Imitation, Inference, and Uncertainty), the author describes a sequence of diverse and increasingly sophisticated ML concepts, culminating in what is called Cooperative Inverse Reinforcement Learning (CIRL). Whether AI will ever stop being part of what I regard as the wrongness of modern technology, I don't know, but at least there are people in the field who have their show more hearts in the right place. show less

There is a great book trapped inside this good book, waiting for a skillful editor to carve it out. The author did vast research in multiple domains and it seems like he could neither build a cohesive narration that could connect all of it nor leave anything out.

This book is probably the best intro to machine learning space for a non-engineer I've read. It presents its history, challenges, what can be done, and what can't be done (yet). It's both accessible and substantive, presenting complex ideas in a digestible form without dumbing them down. If you want to spark the ML interest in anyone who hasn't been paying attention to this field, give them this book. It provides a wide background connecting ML to neuroscience, cognitive show more science, psychology, ethics, and behavioral economics that will blow their mind.

It's also very detailed, screaming at the reader "I did the research, I went where no one else dared to go!". It will not only present you with an intriguing ML concept but also: trace its roots to XIX century farming problem or biology breakthrough, present all the scientist contributing to this research, explain how they met and got along, cite author's interviews with some of them, and present their life after they published their masterpiece, including completely unrelated information about their substance abuse and dark circumstances of their premature death. It's written quite well, so there might be an audience who enjoys this, but sadly I'm not a part of it.

If this book was structured to touch directly the subject of the alignment problem it would be at least 3 times shorter. It doesn't mean that 2/3 are bad - most of it is informative, some of it is entertaining, a lot seems like ML things that the author found interesting and just added to the book without any specific connection to its premise. I really liked the first few chapters where machine learning algorithms are presented as the first viable benchmark to the human thinking process and mental models that we build. Spoiler alert: it very clearly shows our flaws, biases, and lies that we tell ourselves (that are further embedded in ML models that we create and technology that uses them).

Overall, I enjoyed most of this book. I just feel a bit cheated by its title and premise, which advertise a different kind of book. This is the Machine Learning omnibus, presenting the most interesting scientific concepts of this field and the scientists behind them. If this is what you expect and need, you won't be disappointed! show less

I think I acquired this book on the recommendation of Tim Spalding, founder of LibraryThing. Because of my profession, I should probably have a better understanding of the mathematics of AI. There's essentially no mathematics in the book. The endorsement on the front cover says that the book is "nuanced and captivating". Nuanced? I guess so. Captivating? I didn't really feel captivated. I'm guessing I have a pretty typical interest in AI and a pretty typical anxiety about its societal effects. Having read the book, I don't think my interest has increased or my anxiety lessened.

I think the book will appeal to those interested in psychology; it feels like there was a lot of that. While there was definitely talk of ethics, I don't leave show more the book feeling very enlightened about practical ethics as it pertains to AI. And it feels like maybe another book on AI might deal with other branches of philosophy in ways that this book doesn't.

So this book may be for you. It wasn't for me. show less

https://www.goodreads.com/review/show/6272678124

I am thoroughly impressed with Christian’s documentation of AI’s development and emergence from nascent geekery to world-altering capital-T Thing. This book released in 2020, and a mere 3.5 years later basically every tech product you’re likely to see has had “AI” thrown at the front or back of its name. There is so much fear, uncertainty, and doubt around this technology that half of the conversations I’m in that involve it seem to want to resolve into people fleeing for the woods.

Christian does a good job of documenting the historical, psychological, ethical, and epistemological origins of AI. I was particularly drawn to the psychological analogies, many of which surprised show more me. I rented this from the library in physical form and so to save my notes for future reference had to painstakingly write page numbers on index cards and go back to scan/dictate the text to my Notes app, but I’m posting those here for my convenience.

—

Notes:

The Alignment Problem

P30 - In one of the first articles explicitly addressing the notion of bias in computing systems, the University of Washington's Batya Friedman and Cornell's Helen Nissenbaum had warned that "computer systems, for instance, are comparatively inexpensive to disseminate, and thus, once developed, a biased system has the potential for widespread impact. If the system becomes a standard in the field, the bias becomes pervasive.", ^40 (Representation)

P49 - As Princeton's Arvind Narayanan puts it: "Contrary to the 'tech moves too fast for society to keep up' cliché, commercial deployments of tech often move glacially-just look at the banking and airline mainframes still running. ML [machine-learning] models being trained today might still be in production in 50 years, and that's terrifying." ^93 (Representation)

Feedback loops

“Machine learning is not, by default, fair or just in any meaningful way.” - Moritz Hardt (^3, Fairness)

“No machinery is more efficient than the human element that operates it.” (??)

“One of the most important things in any prediction is to make sure that you’re actually predicting what you think you’re predicting. This is harder than it sounds.”

P123 - Thorndike sees here the makings of a bigger, more general law of nature. As he puts it, the results of our actions are either "satisfying" or "annoying." When the result of an action is "satisfying," we tend to do it more. When on the other hand the outcome is "annoying," we'll do it less. The more clear the connection between action and outcome, the stronger the resulting change. Thorndike calls this idea, perhaps the most famous and durable of his career, "the law of effect."
As he puts it:

The Law of Effect is: When a modifiable connection between a situation and a response is made and is accompanied or followed by a satisfying state of affairs, that connection's strength is increased: When made and accompanied or followed by an annoying state of affairs its strength is decreased. The strengthening effect of satisfyingness (or the weakening effect of annoy-ingness upon a bond varies with the closeness of the connection between it and the bond. ^7 (Reinforcement)

P127 - Continuing to develop machines that could learn, in other words—by human instruction or their own experience-would alleviate the need for programming. Moreover it would enable computers to do things we didn't know how to program them to do.

P141 - “this is apparently the first application of this algorithm to a complex non-trivial task,” TESAURO wrote. Re: use of algorithms to play go, I got you got, learning from guesses steadily coming to learn one adventure position look like. … “it is spelling it, with zero knowledge built in, the network is able to learn from scratch to play the entire game at a fairly strong, intermediate level of performance, which is clearly better than conventional commercial programs, and which in fact, surpasses comparable networks trained on a massive human expert data set. This indicates that TD learning may work better in practice than one would expect based on current theory.“

P151 - Meanwhile, we take up another question. Reinforcement learning in its classical form takes for granted the structure of the rewards in the world and asks the question of how to arrive at the behavior-the "policy" —that maximally reaps them. But in many ways this obscures the more interesting—and more dire—matter that faces us at the brink of Al. We find ourselves rather more interested in the exact opposite of this question: Given the behavior we want from our machines, how do we structure the environment's rewards to bring that behavior about?
How do we get what we want when it is we who sit in the back of the audience, in the critic's chair—we who administer the food pellets, or their digital equivalent?
This is the alignment problem, in the context of a reinforcement learner. Though the question has taken on a new urgency in the last five to ten years, as we shall see it is every bit as deeply rooted in the past as reinforcement learning itself.

P160 - But Miyamoto had a problem. There are also good mushrooms, which you have to learn, not to dodge, but to seek. "This gave us a real head-ache," he explains. "We needed somehow to make sure the player understood that this was something really good." So now what? The good mushroom approaches you in an area where you have too little headroom to easily jump over it-you brace for impact, but instead of killing you, it makes you double in size. The mechanics of the game have been established, and now you are let loose. You think you are simply playing.
But you are carefully, precisely, inconspicuously being trained. You learn the rule, then you learn the exception. You learn the basic mechanics, then you are given free rein.

P161 - in both cases, the use of a curriculum – an easier version of the problem, followed by a harder version – succeeding in cases we’re trying to learn the more difficult problem by itself could not.

P169 - "As a general rule," says Russell, "it is better to design performance measures according to what one actually wants in the environment, rather than according to how one thinks the agent should behave.”^50 Put differently, the key insight is that we should strive to reward states of the world, not actions of our agent. These states typically represent "progress" toward the ultimate goal, whether that progress is represented in physical distance or in something more conceptual like completed subgoals (chapters of a book, say, or portions of a mechanical assembly). (^50 Shaping).

P185 - Learned helplessness; “As the celebrated aphorist Ashleigh Brilliant put it, “If you’re careful enough, nothing bad or good will ever happen to you.” ^11 (Curiosity)

P202 - All rewards are internal. ^61 (Curiosity).

P222 - Conway lloyd Morgan - “Five minutes’ demonstration is worth more than five hours’ talking where the object is to impart skill. It is of comparatively little use to describe or explain how a skilled feat is to be accomplished; it is far more helpful to show how it is done.” ^32 (Imitation)

P228 - At its root, the problem stems from the fact that the learner sees an expert execution of the problem, and an expert almost never gets into trouble. No matter how good the learner is, though, they will make mistakes – whether blatant or supple. But because the learner never saw the expert get into trouble, they have also never seen the expert get out. In fact, when the beginner makes beginner mistakes, they may end up in a situation that is completely different from anything they saw during their observation of the expert. “That means,“ says Sergey Levine, “that, you know, all bets are off.” (Cascading errors).

P247 - Eliezer Yudkowsky, cofounder of the Machine Intelligence Research Institute, wrote an influential 2004 manuscript in which he argues for imbuing machines, not simply to imitate and hold or norms as we imperfectly embody them, but rather, we should instill in machines what he calls our “coherent extrapolated volition.“ “In poetic terms, “he writes, “our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wish we were.“

P251 - Warneken, along with his collaborator Michael Tomasello of Duke, was the first to systematically show, in 2006, that human infants as young as eighteen months old will reliably identify a fellow human facing a problem, will identify the human's goal and the obstacle in the way, and will spontaneously help if they can-even if their help is not requested, even if the adult doesn't so much as make eye contact with them, and even when they expect (and receive) no reward for doing so.^2 (Inference)

P261 - We are now, it is fair to say, well beyond the point where our machines can do only that which we can program into them in the explicit language of math and code.

P268 - Russell dubbed this new framework cooperative inverse reinforcement learning ("CIRL," for short).^40 In the CIRL formulation, the human and the computer work together to jointly maximize a single reward function - and initially only the human knows what it is.
“We we’re trying to think, what’s the simplest change we can make to the current math and the current theoretical systems that fixes the theory that leads to these sort of existential-risk problems?“ says Hadfield-Menell. “What is a math problem where the optimal thing is what we actually want?“^41 (Inference)

P282 - He has the students play games where they must decide which side of various bets to take, figuring out how to turn their beliefs and hunches into probabilities, and deriving the laws of probability theory from scratch. They are games of epistemology: What do you know? And what do you believe? And how confident are you, exactly? "That gives you a very good tool for machine learning," says Gal, "to build algorithms—to build computational tools —that can basically use these sorts of principles of rationality to talk about uncertainty." (…) Gal: “I wouldn’t rely on a model that couldn’t tell me whether it’s actually certain about its predictions.” (re: uncertainty in models and models communicating uncertainty; ensembling; dropouts…).^14
There's a certain irony here, in that deep learning--despite being deeply rooted in statistics—has, as a rule, not made uncertainty a first-class citizen.
Note from TB: thinking about uncertainty in prioritization. Weighing measures in a prioritization algorithm.

P292 - Another researcher who has been focused on these problems in recent years is DeepMind's Victoria Krakovna. Krakovna notes that one of the big problems with penalties for impact is that in some cases, achieving a specific goal necessarily requires high-impact actions, but this could lead to what's called "offsetting": taking further high-impact actions to counterbalance the earlier ones. This isn't always bad: if the system makes a mess of some kind, we probably want it to clean up after itself. But sometimes these "offsetting" actions are problematic. We don't want a system that cures someone's fatal illness but then-to nullify the high impact of the cure-kills them. ^43 (Uncertainty)
Note from TB: thinking about uncertainty in prioritization again, and how to measure / quantify “impact on PEH,” in algorithm. What is the impact of each stage of the prioritization process, from inflow to referral, etc.

P294 - Turner’s idea is that the reason we care about the Shanghai Stock Exchange, or the integrity of our cherished vase, or, for that matter, the ability to move boxes around the virtual warehouse, is it those things for whatever reason matter to us, and they matter to us because they are ultimately in some way or other tied to our goals. We want to save for retirement, put flowers in the vase, complete the sokoban level. What if we model this idea of goals explicitly? His proposal goes by the name “attainable utility preservation“: giving the system a set of auxiliary goals in the game environment, and making sure that it can still effectively pursue these auxiliary goals after it’s done whatever points-scoring actions the game incentivizes. Fascinatingly, the mandate to preserve a tangible utility seems to foster good behavior in the AI safety gridworlds even when the auxiliary goals are generated at random. ^49 (Uncertainty)

P295 - One of the most chilling and prescient quotations in the field of AI safety comes in a famous 1960 article on the "Moral and Technical Consequences of Automation" by MIT's Norbert Wiener: "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it... then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it."^51 It is the first succinct expression of the alignment problem.
No less crucial, however, is this statement's flip side: If we were not sure that the objectives and constraints we gave the machine entirely and perfectly specified what we did and didn't want the machine to do, then we had better be sure we can intervene. In the Al safety literature, this concept goes by the name of “corrigibility,” and—soberingly—it’s a whole lot more complicated than it seems.^52 (Uncertainty)

P299 - But, they found, there's a major catch. If the system's model of what you care about is fundamentally "misspecified"-there are things you care about of which it's not even aware and that don't even enter into the system's model of your rewards-then it's going to be confused about your motivation. For instance, if the system doesn't understand the subtleties of human appetite, it may not understand why you requested a steak dinner at six o'clock but then declined the opportunity to have a second steak dinner at seven o'clock. If locked into an oversimplified or misspecified model where steak (in this case) must be entirely good or entirely bad, then one of these two choices, it concludes, must have been a mistake on your part. It will interpret your behavior as "irrational," and that, as we've seen, is the road to incorrigibility, to disobedience."^63 (Uncertainty)

——

Notes

Representation
* 40 - Friedman and Nissenbaum, “Bias in Computer Systems.”
* 93 - Narayanan on Twitter: https://twitter.com/random_walker/sta...

Fairness
* 3 - Hardt, “How Big Data Is Unfair.”

Reinforcement
* 7 - Thorndike, The Psychology of Learning.

Shaping
* 50 - Russell and Norvid, Artificial Intelligence.

Curiosity
* 11 - See Henry Alford, “The Wisdom of Ashleigh Brilliant,” http://www.ashleighbrilliant.com/Bril..., excerpted from Alford, How to Live (New York: Twelve, 2009).
* 61 - Singh, Lewis, and Barto. For more discussion, see Oudeyer and Kaplan, “What Is Intrinsic Motivation?”
* Sing, Lewis, and Barto — “Where Do Rewards Come From?” In “Proceedings of the Annual Conference of the Cognitive Science Society,” 2601-06. 2009.

Imitation
* 32 - Morgan, “An Introduction to Comparative Psychology.”

Inference
* 2 - See also Meltzoff, “Understanding the intentions of Others” which showed that eighteen-month olds can successfully imitate the intended acts that adults tried and failed to do, indicating that they ‘situate people within a psychological framework that differentiates between the surface behavior of people and a deeper level involving goals and intentions.’
* The citation for the Warneken paper: Warneken, Felix, and Michael Tomasello. “Altruistic Helping in Human Infants and Young Chimpanzees.” Science 311, no. 5765 (2006): 1301-03.
* 40 - Hadfield-Menell et al., “Cooperative Inverse Reinforcement Learning.” (“CIRL” is pronounced with a soft c, homophonous with the last name of strong AI skeptic John Searle (no relation). I have agitated within the community that a hard c “curl” pronunciation makes more sense, given that “cooperative” uses a hard c, but it appears the die is cast.).
* Note from TB: I agree w/ the hard c note.
* 41 - Dylan Hadfield-Menell, personal interview, March 15, 2018.

Uncertainty
* 14 - Yarin Gal, “Modern Deep Learning Through Bayesian Eyes” (lecture), Microsoft Research, December 11, 2015, https://www.microsoft.com/en-us/resea....
* 43 - As Eliezer Yudkowsky put it, “If you’re going to cure cancer, make sure the patient still dies!” See https://intelligence.org/2016/12/28/a.... See also Armstrong and Levinstein, “Low Impact Artificial Intelligence,” which uses the example of an asteroid headed for earth. A system constrained to only take “low-impact” actions might fail to divert it—or, perhaps even worse, a system capable of offsetting might divert the asteroid, saving the planet, and then blow the planet up anyway.
* 49 - Mind Safety Kesearer o
* designing-agent-incentives-to-avoid-side-effects-elac80ea6107.
* 49. Turner, Hadfield-Menell, and Tadepalli, "Conservative Agency via Attainable Utility Preservation." See also Turner's "Reframing Impact" sequence at http://www.alignmentforum.org/s/7Cdoz... and additional discussion in his "Towards a New Impact Measure," https://www.alignmentforum.org/ posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure; he writes, "I have a theory that AUP seemingly works for advanced agents not because the content of the attainable set's utilities actually matters, but rather because there exists a common utility achievement currency of power." See Turner, "Optimal Farsighted Agents Tend to Seek Power." For more on the notion of power in an Al safety context, including an information-theoretic account of "empowerment," see Amodei et al., "Concrete Problems in Al Safety," which, in turn, references Salge, Glackin, and Polani, "Empowerment: An Introduction," and Mohamed and Rezende, "Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning."
* 51 - Wiener, “Some Moral and Technical Consequences of Automation.”
* 52 - According to Paul Christiano, “corrigibility” as a tenet of AI safety began with Machine intelligence Research Institute’s Eliezer Yudkowsky, and the name itself came from Robert Miles. See Christiano’s “Corrigibility,” https://ai-alignment.com/corrigibilit....
* 63 - For more on corrigibility and model misspecification using this paradigm, see also, e.g., Carey, “Incorrigibility in the CIRL Framework.” show less

The book goes through various projects where machine learning and other artificial intelligence tools have been used to do things, from learning to play games to learning to drive cars. There's a focus on unintended consequences and the rewards associated with activities. First published in 2021, this is probably starting to feel a little behind the times.