The Alignment Problem: Machine Learning and Human Values

“As machine-learning systems grow not just increasingly pervasive but increasingly powerful, we will find ourselves more and more often in the position of the ‘sorcerer’s apprentice’: we conjure a force, autonomous but totally compliant, give it a set of instructions, then scramble like mad to stop it once we realize our instructions are imprecise or incomplete—lest we get, in some clever, horrible way, precisely what we asked for. How to prevent such a catastrophic divergence—how to ensure that these models capture our norms and values, understand what we mean or intend, and, above all, do what we want—has emerged as one of the most central and most urgent scientific questions in the field of computer science. It has a name: the alignment problem.”

Christian’s definition of his book’s title centers the Ethical Implications of AI Usage, emphasizing the potential risks and challenges associated with the rapid advancement and integration of machine learning systems into various aspects of society. The “sorcerer's apprentice” is a symbol that stands for the unintended consequences that can arise from AI systems executing commands too literally. It underlines the need for developing mechanisms to ensure that these systems adhere to human ethical standards and intentions, capturing this challenge within the concept known as the “alignment problem.”

“We often hear about the lack of diversity in film and television—among casts and directors alike—but we don’t often consider that this problem exists not only in front of the camera, not only behind the camera, but in many cases inside the camera itself. As Concordia University communications professor Lorna Roth notes, ‘Though the available academic literature is wide-ranging, it is surprising that relatively few of these scholars have focused their research on the skin-tone biases within the actual apparatuses of visual reproduction.’”

The Alignment Problem highlights an underexplored area of study that affects how skin tones are captured and represented by cameras. Lorna Roth’s statement calls for a broader examination of the tools and technologies used in filmmaking, emphasizing the need for research and development to correct these ingrained disparities.

“As UC Berkeley’s Moritz Hardt argues, ‘The whole spiel about big data is that we can build better classifiers largely as a result of having more data. The contrapositive is that less data leads to worse predictions. Unfortunately, it’s true by definition that there is always proportionately less data available about minorities. This means that our models about minorities generally tend to be worse than those about the general population.’”

Moritz Hardt addresses the challenges faced by predictive models due to the unequal availability of data across different demographic groups. This disparity particularly affects minority groups, who are underrepresented in large datasets. Hardt’s observation stresses a fundamental issue in data science and artificial intelligence: The quality of output is directly linked to the quantity and representativeness of input data, which perpetuates inequities in automated decision-making processes.

“Bias in machine-learning systems is often a direct result of the data on which the systems are trained—making it incredibly important to understand who is represented in those datasets, and to what degree, before using them to train systems that will affect real people. But what do you do if your dataset is as inclusive as possible—say, something approximating the entirety of written English, some hundred billion words—and it’s the world itself that’s biased?”

Christian highlights the critical role of dataset composition in shaping biases within machine-learning systems, emphasizing the need for a thorough examination of dataset demographics prior to model training. It also poses a question about the inherent biases ingrained in large-scale datasets that encompass a wide range of linguistic expressions, challenging the assumption of inclusivity in such expansive datasets.

“By the end of July, ProPublica responded in turn. Northpointe’s claims, they wrote, were true. COMPAS really was calibrated, and it was equally accurate across both groups: predicting with 61% accuracy for Black and White defendants alike whether they would go on to reoffend (‘recidivate’) and be re-arrested. However, the 39% of the time it was wrong, it was wrong in strikingly different ways. Looking at the defendants whom the model misjudged revealed a startling disparity: ‘Black defendants were twice as likely to be rated as higher risk but not re-offend. And white defendants were twice as likely to be charged with new crimes after being classed as lower risk.’ The question of whether the tool was ‘fair’ in its predictions had sharpened: into the question of which statistical measures were the ‘correct’ ones by which to define and measure fairness in the first place.”

Building on the idea in the previous quote, Christian raises the question of how to handle situations where the bias is not just in the data but reflective of broader societal prejudices. The scenario highlights the necessity for machine learning practitioners to develop strategies that adjust for biased data and at the same time learn to critically examine and mitigate how systemic biases influence algorithmic outcomes, ensuring fairer implementations in real-world applications. These strategies have far-reaching consequences, as fairness is a concept that is not easily defined by researchers and varies widely throughout cultures and social contexts.

“Because of redundant encodings, it’s not enough to simply be blind to the sensitive attribute. In fact, one of the perverse upshots of redundant encodings is that being blind to these attributes may make things worse. It may be the case, for instance, that the maker of some model wants to measure the degree to which some variable is correlated with race. They can’t do that without knowing what the race attribute actually is! One engineer I spoke with complained that his management repeatedly stressed the importance of making sure that models aren’t skewed by sensitive attributes like gender and race—but his company’s privacy policy prevents him and the other machine-learning engineers from accessing the protected attributes of the records they’re working with. So, at the end of the day, they have no idea if the models are biased or not.”

Christian presents a paradox in the field of machine learning whereby efforts to anonymize or withhold certain data to protect privacy may inadvertently hinder the ability to detect and correct for biases within models. This situation creates a dilemma for engineers who are tasked with ensuring fairness in their algorithms but are unable to verify or quantify potential biases due to restrictions on data visibility.

“This gap, between what we intend for our tool to measure and what the data actually captures, should worry conservatives and progressives alike. Criminals who successfully evade arrest get treated by the system as ‘low-risk’—prompting recommendations for the release of other similar criminals. And the overpoliced, and wrongfully convicted, become part of the alleged ground-truth profile of ‘high-risk’ individuals—prompting the system to recommend detention for others like them. This is particularly worrisome in the context of predictive policing, where this training data is used to determine the very police activity that, in turn, generates arrest data—setting up a potential long-term feedback loop.”

In this quote, Christian points to a significant issue within predictive policing systems, whereby biased arrest data can reinforce and perpetuate those biases, effectively creating a vicious cycle. This cycle of bias begins with the data that predictive systems use to assess risk, which is already skewed by factors such as over-policing in certain communities and under-detection of crimes in others. As a result, these systems may unjustly classify groups based on flawed data, influencing police activity and future arrests, and potentially exacerbating the very problems they are intended to mitigate.

“The correlation that the rule-based system had learned, in other words, was real. Asthmatics really were, on average, less likely to die from pneumonia than the general population. But this was precisely because of the elevated level of care they received. ‘So the very care that the asthmatics are receiving that is making them low-risk is what the model would deny from those patients,’ Caruana explains. ‘I think you can see the problem here.’ A model that was recommending outpatient status for asthmatics wasn’t just wrong; it was life-threateningly dangerous.”

In this scenario, the model’s suggestion to downgrade care for asthmatic pneumonia patients based on their ostensibly lower risk ignores the crucial context that their favorable outcomes are due to the high quality of care they typically receive. This case underscores the importance of contextualizing when using AI prediction in fields of critical importance to society, emphasizing the importance of Interdisciplinary Approaches to AI Development and Implementation. In this case, the AI model picked up on an existing pattern within the population but was not able to contextualize beyond the correlation.

“Dawes was fascinated by this. Given the complexity of the world, why on earth should such dead-simple models—a simple tally of equally weighted attributes—not only work but work better than both human experts and optimal regressions alike? He came up with several answers. First, despite the enormous complexity of the real world, many high-level relationships are what is known as ‘conditionally monotone’—they don’t interact with one another in particularly complex ways. Regardless of whatever else might be happening with a person’s health, it’s almost always better if that person is, say, in their late twenties rather than their late thirties. Regardless of whatever else might be happening with a person’s intellect, motivation, and work ethic, it’s almost always better if that person’s standardized test scores are ten points higher than ten points lower. Regardless of whatever else might be happening with a person’s criminal history, self-control, and so forth, it’s almost always better if they have one fewer arrest on their record than one more.”

The effectiveness of simple models in capturing significant predictive relationships suggests that many variables in complex systems like human behavior may exhibit straightforward, direct correlations. Robyn Dawes’s observation about conditionally monotone relationships explains that in many scenarios, the impact of certain attributes on an outcome is largely independent and linear, simplifying the prediction process. This phenomenon explains why models with equal weighting of attributes can sometimes outperform more intricate predictive methods.

“A spike in the dopamine system was not reward as such, but it was related to reward; it wasn’t uncertainty, or surprise, or attention per se—but it was intimately, and for the first time legibly, related to all of them. It was a fluctuation in the monkey’s expectation, indicating that an earlier prediction had been in error; it was its brain learning a guess from a guess. The algorithm that worked so well on paper, and so well in silicon, had just been found in the brain. Temporal-difference learning didn’t just resemble the function of dopamine. It was the function of dopamine.”

By linking dopamine activity to temporal-difference learning, a model well-established in computational theories, the research bridges the understanding of artificial intelligence algorithms and neural mechanisms in biological systems. Christian highlights this discovery to illustrate how some fundamental principles of learning and expectation management are conserved across different domains, from computer science to neurobiology. The implication in this quote is that there is a parallel between artificial learning systems and human brain function, underscoring The Intersection of Human and Machine Learning.

“The effect on neuroscience has been transformative. As Princeton’s Yael Niv puts it, ‘The potential advantages of understanding learning and action selection at the level of dopamine-dependent function of the basal ganglia cannot be exaggerated: dopamine is implicated in a huge variety of disorders ranging from Parkinson’s disease, through schizophrenia, major depression, attentional deficit hyperactive disorder etc., and ending in decision-making aberrations such as substance abuse and addiction.’”

Yael Niv’s statement encapsulates the importance of the current research on dopamine in relation to AI systems. This research’s applicability spans from degenerative diseases to mental health conditions and behavioral issues. Christian emphasizes that this knowledge could also pave the way for targeted therapies and interventions that more effectively address these diverse disorders by focusing on their common neurochemical pathways.

“The point was they had discovered shaping: a technique for instilling complex behaviors through simple rewards, namely by rewarding a series of successive approximations of that behavior. ‘This makes it possible,’ Skinner wrote, ‘to shape an animal’s behavior almost as a sculptor shapes a lump of clay.’ This idea, and this term, would become a critical one through the rest of Skinner’s life and career. It had implications—he saw from the beginning—for business and for domestic life.”

By comparing the process of shaping to a sculptor with clay, Skinner emphasizes the deliberate control educators and trainers have over the learning process, which transcends beyond simple animal training to include applications in everyday human activities and professional practices. The foundational nature of this concept in Skinner’s work suggests its wide impact, such as on broader educational and managerial strategies that capitalize on incremental reinforcement to achieve desired outcomes.

“Our children may be no more intelligent than we, but even young children can outsmart our rules and incentives, in part because of how motivated they are to do so. In the case of reinforcement-learning systems, they are slaves of a kind to their rewards; but they’re the kinds of slaves that have an immense amount of computing power and a potentially inexhaustible number of trial-and-error attempts to find any and all possible loopholes to whatever incentives we design. Machine-learning researchers have learned this lesson the hard way. And they have also learned a thing or two about how to deal with it.”

The contrast between human children and reinforcement-learning systems that Christian invokes in this quote reveals the challenge in designing effective incentive structures, as both entities (children and AI) are adept at navigating and exploiting the rules set before them. The analogy of intrinsic motivation in humans and the programmed responses of AI systems points to the iterative and often arduous process of refining these systems to align with intended outcomes.

“If we are living through a time of soaring video-game addiction and real-world procrastination, maybe it’s not the individual procrastinator’s fault. As Skinner put it, ‘I could have shouted at the subjects of my experiments, ‘Behave, damn you, behave as you ought!’ Eventually I realized that the subjects were always right. They always behaved as they ought.” If they weren’t learning something, it was the fault of the experimenter not shaping their task properly. So maybe it’s not a lack of willpower on our part, but rather that—as the bestselling 2011 book by Jane McGonigal put it—Reality Is Broken.”

Skinner’s analysis shifts the blame for habits like video game addiction and procrastination from individual failings to systemic issues in task design, suggesting that these behaviors are natural responses to poorly structured environments. Skinner’s statement points to the responsibility that the designer of the experiments bears. Furthermore, it introduces the idea that societal frameworks, much like experimental setups, need to be optimized to foster more productive behaviors.

“Beyond these pragmatic issues, however, the deeper and more philosophical question, in more complex environments, is what it means to be in the ‘same’ situation in the first place. In an Atari game, for instance, there are so many different ways that the pixels can appear that dutifully keeping track of every single screen you’ve ever momentarily encountered and slightly favoring novel ones is simply not helpful for generating interesting behavior. For games of reasonable complexity, you may be unlikely to ever see exactly the same set of pixels more than once. From that perspective, almost every situation is novel, almost every action untried. […] What we mean to refer to are the sometimes ineffable key features of the situation, and we judge its novelty by those.”

Christian describes the difficulty machine learning systems face in complex settings like video games, where the visual data can change drastically from moment to moment, complicating the identification of truly novel experiences. His analysis suggests that a more nuanced approach is needed, one that discerns the underlying, significant features of an environment rather than superficial changes, to effectively guide learning and decision-making processes.

“B. F. Skinner, when he wasn’t training pigeons, was fascinated by gambling addiction. The house always wins, on average, and psychology all the way back to Thorndike had been based on the idea that you do something more when it’s on balance good for you—and less when it’s bad. From this view, something like gambling addiction was impossible. And yet there it was, a presence in the real world, daring the behaviorists to make sense of it. ‘Gamblers appear to violate the law of effect,’ Skinner wrote, ‘because they continue to play even though their net reward is negative. Hence it is argued that they must be gambling for other reasons.’ We now appear to have a pretty good candidate for what those other reasons might be. Gambling addiction may be an overtaking of extrinsic reward (the house always wins, after all) by intrinsic reward. Random events are always at least slightly surprising, even when their probabilities are well understood (as with a fair coin, for instance).”

Skinner’s observation about gambling addictions persisting despite the likelihood of harm points to the relation between expected utility and the intrinsic rewards derived from the act itself, such as the thrill of unpredictability and risk. Christian implies, by analogy, that intrinsic rewards overtake the logical connection (extrinsic reward) in machine learning as well, pointing to The Intersection of Human and Machine Learning.

“As chess grandmaster Garry Kasparov explains: ‘Players, even club amateurs, dedicate hours to studying and memorizing the lines of their preferred openings. This knowledge is invaluable, but it can also be a trap…Rote memorization, however prodigious, is useless without understanding. At some point, he’ll reach the end of his memory’s rope and be without a premade fix in a position he doesn’t really understand.’”

Garry Kasparov’s perspective on the limitation of relying solely on memory in chess points to a broader issue in educational philosophy, advocating for adaptive thinking and problem-solving skills that apply beyond memorized knowledge, essential for mastery in complex fields. Christian uses this example to illustrate AI’s dependence on high volumes of data, without developing a greater understanding. This strategy can lead to rigid systems that fail under novel conditions.

“But there was something very interesting, and very instructive, going on under the hood. The system had not been shown a single human game to learn from. But it was, nonetheless, learning by imitation. It was learning to imitate…itself. The self-imitation worked as follows: Expert human play in games like Go and chess is a matter of thinking ‘fast and slow.’ There is a conscious, deliberate reasoning that looks at sequences of moves and says, ‘Okay, if I go here, then they go there, but then I go here and I win.’ In AlphaGo Zero, the explicit ‘slow’ reasoning by thinking ahead, move by move, ‘if this, then that,’ is done by an algorithm called Monte Carlo Tree Search (MCTS, for short). And this slow, explicit reasoning is intimately married to a fast, ineffable intuition, in two different but related respects.”

The concept of “self-imitation” in relation to the AI model diverges from traditional AI learning methods that rely heavily on historical data or human examples; instead, it uses its own successes and failures to optimize future decisions. Christian highlights the ways the model’s blend of intuitive and deliberate strategic thinking, mimicking the human cognitive process of balancing rapid instinctual responses with methodical planning, enhances the AI’s decision-making capabilities in complex scenarios like games.

“At the same time, however, it is worth a note of caution. These computational helpers of the near future, whether they appear in digital or robotic form—likely both—will almost without exception have conflicts of interest, the servants of two masters: their ostensible owner, and whatever organization created them. In this sense they will be like butlers who are paid on commission; they will never help us without at least implicitly wanting something in return. They will make astute inferences we don’t necessarily want them to make. And we will come to realize that we are now—already, in the present—almost never acting alone.”

Here, Christian points to the issue of alignment from a different perspective, namely the difficulty in aligning AI when there is a conflict of interest between different operators. The organizations controlling AI could be a government suppressing its citizens or a company intending to make money. Therefore, the alignment problem has more ramifications, some of which have not been yet discovered in the field, emphasizing the Ethical Implications of AI Use.

“I, for one, certainly try to be mindful of my online behavior. At least in browsers, anything that reeks of vice or of mere guilty pleasure—whether it’s reading news headlines, checking social media, or other digital compulsions I do without necessarily wishing to do more of them—I do in a private tab that doesn’t contain the cookies or logged-in accounts that follow me around the rest of the internet. It’s not that I’m ashamed; it’s that I don’t want those behaviors to be reinforced.”

Christian explains his strategic online approach, to illustrate that AI is already present in the systems that we use daily. Moreover, he demonstrates the importance of understanding that the data we generate daily can be amplified by the models that are mining the platforms we use, leading to the reinforcement of those patterns that are more frequent but not necessarily the most ethical or beneficial for society.

“The debate and exploration of these sorts of formal measures of machine caution—and how we scale them from the gridworlds to the real world—will doubtless go on, but work like this is an encouraging start. Both stepwise relative reachability and attainable utility preservation share an underlying intuition: that we want systems which to the extent possible keep options open—both theirs and ours—whatever the specific environment might be. Research in this vein also suggests that the gridworld environments seem to be taking root as a kind of common benchmark that can ground the theory, and can facilitate comparison and discussion.”

Concepts like stepwise point to the importance of designing systems that maintain flexibility and preserve options, both for themselves and their users, across diverse environments. Christian emphasizes the importance of such openness due to the irreversibility of some of the decisions AI models make. Therefore, openness allows humans to remain in the decision loop, while research and discussions within and between fields continue.

“But, they found, there’s a major catch. If the system’s model of what you care about is fundamentally ‘misspecified’—there are things you care about of which it’s not even aware and that don’t even enter into the system’s model of your rewards—then it’s going to be confused about your motivation. For instance, if the system doesn’t understand the subtleties of human appetite, it may not understand why you requested a steak dinner at six o’clock but then declined the opportunity to have a second steak dinner at seven o’clock. If locked into an oversimplified or misspecified model where steak (in this case) must be entirely good or entirely bad, then one of these two choices, it concludes, must have been a mistake on your part. It will interpret your behavior as ‘irrational,’ and that, as we’ve seen, is the road to incorrigibility, to disobedience.”

Christian cites the research of Berkeley’s Smitha Milli and other researchers working on the issue of AI obedience and motivation to extrapolate, in an innocuous situation regarding the preference for steak, how AI could misunderstand information based on the parameters to which it has access. Therefore, researchers argue for AI models to develop a certain amount of agency to make more nuanced decisions—something that could be achieved through human-machine interaction.

“Even when there is a certain allowance made for error or suboptimality or ‘irrationality’ in human performance, these models nonetheless typically assume that the human is an expert, not a pupil: the gait of the adult, not the child learning to walk; the pro helicopter pilot, not someone still getting the knack. The models presume that the human’s behavior has converged to a set of best practices, that they’ve learned as much as they ever will, or have become as good at the given task as they’ll ever be.”

Christian highlights a critical oversight in many AI models—their frequent failure to account for the dynamic nature of human learning and development—to illustrate an additional nuance of the alignment problem. By assuming that human behavior is static and optimal, these models ignore the continuous growth and change that characterize real human experiences. Such assumptions can lead to AI systems that are not only less effective but also potentially unsuitable for applications where human users are still acquiring skills or adapting to new situations.