Persistent racial inequalities in America

Historian Thomas Sugrue is a national expert on the state of persistent racial inequalities in our nation today.  His The Origins of the Urban Crisis: Race and Inequality in Postwar Detroit set the standard for the field of recent urban history when it appeared in 1996. His most recent book, Not Even Past: Barack Obama and the Burden of Race, provides a vivid statement of his current views.  The title refers to William Faulkner's line in Requiem for a Nun, "The past is not dead. In fact it's not even past."  It is a particularly apt metaphor for the status of race in America today.

The book addresses the often-heard view that President Obama's election demonstrates that we live in a post-racial society.  The argument is that if the United States is able to elect an African-American president, then surely issues of racial prejudice are in the past.  However, Sugrue provided current data on disparities of income, employment, education, residence, health, and wealth that demonstrate that structures in our country continue to reproduce grave racial disparities.  By any measure, we are not post-racial.

Here are a few factual findings from the book:

The most persistent manifestation of racial inequality in the modern United States has been racial segregation in housing and education.  From 1920 through 1990, patterns of black white segregation hardened in most of the United States, despite shifts in white attitudes about black neighbors, and despite the passage of local and state antidiscrimination laws and the enactment of Title VIII of the Civil Rights Act (1968) which prohibited housing discrimination nationwide. (101)

Persistent residential segregation compounded educational disparities.   Beginning in the late 1970s, when courts began a thirty-year process of abandoning the mandate of Brown v. Board of Education, school districts around the country resegregated by race, especially by black and white. (103)

African Americans are far more likely than whites to be economically insecure. The statistics are grim.  In 2006, the median household income of blacks was only 62 percent of that of whites. Blacks were much more likely than whites to be unemployed (black unemployment rates have remained one and a half to two times those of whites since the 1950s). (104-5)

Social scientists have documented employers discriminating against job applicants with comparable credentials when one has a "black" name or has a place of residence in a known "black" neighborhood. (105)

The starkest racial disparities in the United States are in wealth (a category that includes such assets as savings accounts, stocks, bonds, and especially real estate).  In 2003, the U.S. Census Bureau calculated that white households had a median net worth of $74,900, whereas black households had a median net worth of only $7,500. (105)

Another important indicator of quality of life is health. ... The racial and ethnic gaps in health and life expectancy are stark.  The life expectancy of whites in 2004 was 78.3; for blacks, it was 73.1. ... In 2003, infant mortality rates were nearly 2.5 times as high for blacks as for whites.  (107)

A couple of empirical sources are particularly central to Sugrue's argument.  One is the 2007 book by William Julius Wilson and Richard Taub, There Goes the Neighborhood: Racial, Ethnic, and Class Tensions in Four Chicago Neighborhoods and Their Meaning for America, which provides quite a bit of empirical data about neighborhood racial attitudes in Chicago.  Another crucial piece of empirical evidence that plays a large role in Sugrue's findings is a major ongoing social-science research project coordinated by the Russell Sage Foundation, the Multi-City Study of Urban Inequality (link).  There are separate volumes on Detroit, Boston, Atlanta, and Los Angeles, and there is a synthesis volume that tries to draw conclusions from all the studies (Urban Inequality: Evidence from Four Cities (Multi City Study of Urban Inequality.)).  The introductory chapter by Alice O'Connor is available on the website; link. Here are a few passages from the introduction:

Nowhere are these intertwined problems more vividly captured than in the complex economic, gender-based, and racial and ethnic divisions of contemporary urban America. In major cities nationwide, overall economic growth is accompanied by higher than average rates of unemployment and poverty, concentrated especially in low-income, working-class minority neighborhoods that have only recently begun to show signs of recovery following decades of steady decline (U.S. Department of Housing and Urban Development 1999). Still, the low-skilled urban workforce, greatly expanded by the “end of welfare,” has little access to local jobs that provide living wages, employment security, and adequate benefits. (1)

Explaining these economic, spatial, and racial divisions is the central purpose of the Multi-City Study of Urban Inequality, a unique inquiry launched in the early 1990s by an interdisciplinary team of social scientists with sponsorship from the Russell Sage and Ford foundations. Based on surveys of households and employers in the metropolitan areas of Atlanta, Boston, Detroit, and Los Angeles, the study affords a comprehensive and systematic look at the roots of inequality in labor markets, residential segregation, and racial attitudes. (2)

The significance of race in urban inequality is to be found not in any single place but in various aspects, and at multiple levels, of social experience: in discriminatory behavior, policies, and institutional practices; in the structural segmentation of labor markets and residential space; and in the attitudes, stereotypes, and ideological belief systems through which people make sense of broader social conditions and determine their own policy preferences. Race has a deep and enduring historical significance as well, still visible in residential color lines constructed by years of racial exclusion, violence, and overtly discriminatory policies; in the persistent racial gaps in education, skills, and capital that stem from opportunity denied; and in the mistrust between minorities and local law-enforcement agencies that has once again erupted around the issue of racial profiling. And race has significance as the basis of a color-coded sense of social hierarchy that affects individual attitudes and behavior, and that is embedded in social structure as well as in shared cultural norms. (5)

The essays in this volume do not tell a simple story. They point to intergroup attitudes, residential and industrial location, discriminatory practices, and the declining prospects for low-skilled workers as important, interlocking sources of inequality across metropolitan America. But they also point to several conclusions about race, both as a shaping force in the distribution of opportunity and as a variable in social scientific analysis. While not offered as a statement of consensus, they do advance the ongoing debate about race and inequality in important ways. (27)

This is careful social scientific research, and it contributes to a much better understanding of the mechanisms that produce and reproduce urban poverty and persistent racial gaps. These mechanisms are both structural and ideational; the processes that lead to discrimination in employment, for example, include both structural factors like urban transportation, and ideational factors such as racial stereotyping by recruiters.

So race continues to be a determining fact of life for many millions of Americans.  But here is the other part of Sugrue's story: our public discourses, and our political process, are virtually silent about these facts.  We simply do not honestly confront the realities of race, whether in electoral competition, in the media, or in other forms of public debate. And we need to.

Persistent racial inequalities in America

Historian Thomas Sugrue is a national expert on the state of persistent racial inequalities in our nation today.  His The Origins of the Urban Crisis: Race and Inequality in Postwar Detroit set the standard for the field of recent urban history when it appeared in 1996. His most recent book, Not Even Past: Barack Obama and the Burden of Race, provides a vivid statement of his current views.  The title refers to William Faulkner's line in Requiem for a Nun, "The past is not dead. In fact it's not even past."  It is a particularly apt metaphor for the status of race in America today.

The book addresses the often-heard view that President Obama's election demonstrates that we live in a post-racial society.  The argument is that if the United States is able to elect an African-American president, then surely issues of racial prejudice are in the past.  However, Sugrue provided current data on disparities of income, employment, education, residence, health, and wealth that demonstrate that structures in our country continue to reproduce grave racial disparities.  By any measure, we are not post-racial.

Here are a few factual findings from the book:

The most persistent manifestation of racial inequality in the modern United States has been racial segregation in housing and education.  From 1920 through 1990, patterns of black white segregation hardened in most of the United States, despite shifts in white attitudes about black neighbors, and despite the passage of local and state antidiscrimination laws and the enactment of Title VIII of the Civil Rights Act (1968) which prohibited housing discrimination nationwide. (101)

Persistent residential segregation compounded educational disparities.   Beginning in the late 1970s, when courts began a thirty-year process of abandoning the mandate of Brown v. Board of Education, school districts around the country resegregated by race, especially by black and white. (103)

African Americans are far more likely than whites to be economically insecure. The statistics are grim.  In 2006, the median household income of blacks was only 62 percent of that of whites. Blacks were much more likely than whites to be unemployed (black unemployment rates have remained one and a half to two times those of whites since the 1950s). (104-5)

Social scientists have documented employers discriminating against job applicants with comparable credentials when one has a "black" name or has a place of residence in a known "black" neighborhood. (105)

The starkest racial disparities in the United States are in wealth (a category that includes such assets as savings accounts, stocks, bonds, and especially real estate).  In 2003, the U.S. Census Bureau calculated that white households had a median net worth of $74,900, whereas black households had a median net worth of only $7,500. (105)

Another important indicator of quality of life is health. ... The racial and ethnic gaps in health and life expectancy are stark.  The life expectancy of whites in 2004 was 78.3; for blacks, it was 73.1. ... In 2003, infant mortality rates were nearly 2.5 times as high for blacks as for whites.  (107)

A couple of empirical sources are particularly central to Sugrue's argument.  One is the 2007 book by William Julius Wilson and Richard Taub, There Goes the Neighborhood: Racial, Ethnic, and Class Tensions in Four Chicago Neighborhoods and Their Meaning for America, which provides quite a bit of empirical data about neighborhood racial attitudes in Chicago.  Another crucial piece of empirical evidence that plays a large role in Sugrue's findings is a major ongoing social-science research project coordinated by the Russell Sage Foundation, the Multi-City Study of Urban Inequality (link).  There are separate volumes on Detroit, Boston, Atlanta, and Los Angeles, and there is a synthesis volume that tries to draw conclusions from all the studies (Urban Inequality: Evidence from Four Cities (Multi City Study of Urban Inequality.)).  The introductory chapter by Alice O'Connor is available on the website; link. Here are a few passages from the introduction:

Nowhere are these intertwined problems more vividly captured than in the complex economic, gender-based, and racial and ethnic divisions of contemporary urban America. In major cities nationwide, overall economic growth is accompanied by higher than average rates of unemployment and poverty, concentrated especially in low-income, working-class minority neighborhoods that have only recently begun to show signs of recovery following decades of steady decline (U.S. Department of Housing and Urban Development 1999). Still, the low-skilled urban workforce, greatly expanded by the “end of welfare,” has little access to local jobs that provide living wages, employment security, and adequate benefits. (1)

Explaining these economic, spatial, and racial divisions is the central purpose of the Multi-City Study of Urban Inequality, a unique inquiry launched in the early 1990s by an interdisciplinary team of social scientists with sponsorship from the Russell Sage and Ford foundations. Based on surveys of households and employers in the metropolitan areas of Atlanta, Boston, Detroit, and Los Angeles, the study affords a comprehensive and systematic look at the roots of inequality in labor markets, residential segregation, and racial attitudes. (2)

The significance of race in urban inequality is to be found not in any single place but in various aspects, and at multiple levels, of social experience: in discriminatory behavior, policies, and institutional practices; in the structural segmentation of labor markets and residential space; and in the attitudes, stereotypes, and ideological belief systems through which people make sense of broader social conditions and determine their own policy preferences. Race has a deep and enduring historical significance as well, still visible in residential color lines constructed by years of racial exclusion, violence, and overtly discriminatory policies; in the persistent racial gaps in education, skills, and capital that stem from opportunity denied; and in the mistrust between minorities and local law-enforcement agencies that has once again erupted around the issue of racial profiling. And race has significance as the basis of a color-coded sense of social hierarchy that affects individual attitudes and behavior, and that is embedded in social structure as well as in shared cultural norms. (5)

The essays in this volume do not tell a simple story. They point to intergroup attitudes, residential and industrial location, discriminatory practices, and the declining prospects for low-skilled workers as important, interlocking sources of inequality across metropolitan America. But they also point to several conclusions about race, both as a shaping force in the distribution of opportunity and as a variable in social scientific analysis. While not offered as a statement of consensus, they do advance the ongoing debate about race and inequality in important ways. (27)

This is careful social scientific research, and it contributes to a much better understanding of the mechanisms that produce and reproduce urban poverty and persistent racial gaps. These mechanisms are both structural and ideational; the processes that lead to discrimination in employment, for example, include both structural factors like urban transportation, and ideational factors such as racial stereotyping by recruiters.

So race continues to be a determining fact of life for many millions of Americans.  But here is the other part of Sugrue's story: our public discourses, and our political process, are virtually silent about these facts.  We simply do not honestly confront the realities of race, whether in electoral competition, in the media, or in other forms of public debate. And we need to.

Persistent racial inequalities in America

Historian Thomas Sugrue is a national expert on the state of persistent racial inequalities in our nation today.  His The Origins of the Urban Crisis: Race and Inequality in Postwar Detroit set the standard for the field of recent urban history when it appeared in 1996. His most recent book, Not Even Past: Barack Obama and the Burden of Race, provides a vivid statement of his current views.  The title refers to William Faulkner's line in Requiem for a Nun, "The past is not dead. In fact it's not even past."  It is a particularly apt metaphor for the status of race in America today.

The book addresses the often-heard view that President Obama's election demonstrates that we live in a post-racial society.  The argument is that if the United States is able to elect an African-American president, then surely issues of racial prejudice are in the past.  However, Sugrue provided current data on disparities of income, employment, education, residence, health, and wealth that demonstrate that structures in our country continue to reproduce grave racial disparities.  By any measure, we are not post-racial.

Here are a few factual findings from the book:

The most persistent manifestation of racial inequality in the modern United States has been racial segregation in housing and education.  From 1920 through 1990, patterns of black white segregation hardened in most of the United States, despite shifts in white attitudes about black neighbors, and despite the passage of local and state antidiscrimination laws and the enactment of Title VIII of the Civil Rights Act (1968) which prohibited housing discrimination nationwide. (101)

Persistent residential segregation compounded educational disparities.   Beginning in the late 1970s, when courts began a thirty-year process of abandoning the mandate of Brown v. Board of Education, school districts around the country resegregated by race, especially by black and white. (103)

African Americans are far more likely than whites to be economically insecure. The statistics are grim.  In 2006, the median household income of blacks was only 62 percent of that of whites. Blacks were much more likely than whites to be unemployed (black unemployment rates have remained one and a half to two times those of whites since the 1950s). (104-5)

Social scientists have documented employers discriminating against job applicants with comparable credentials when one has a "black" name or has a place of residence in a known "black" neighborhood. (105)

The starkest racial disparities in the United States are in wealth (a category that includes such assets as savings accounts, stocks, bonds, and especially real estate).  In 2003, the U.S. Census Bureau calculated that white households had a median net worth of $74,900, whereas black households had a median net worth of only $7,500. (105)

Another important indicator of quality of life is health. ... The racial and ethnic gaps in health and life expectancy are stark.  The life expectancy of whites in 2004 was 78.3; for blacks, it was 73.1. ... In 2003, infant mortality rates were nearly 2.5 times as high for blacks as for whites.  (107)

A couple of empirical sources are particularly central to Sugrue's argument.  One is the 2007 book by William Julius Wilson and Richard Taub, There Goes the Neighborhood: Racial, Ethnic, and Class Tensions in Four Chicago Neighborhoods and Their Meaning for America, which provides quite a bit of empirical data about neighborhood racial attitudes in Chicago.  Another crucial piece of empirical evidence that plays a large role in Sugrue's findings is a major ongoing social-science research project coordinated by the Russell Sage Foundation, the Multi-City Study of Urban Inequality (link).  There are separate volumes on Detroit, Boston, Atlanta, and Los Angeles, and there is a synthesis volume that tries to draw conclusions from all the studies (Urban Inequality: Evidence from Four Cities (Multi City Study of Urban Inequality.)).  The introductory chapter by Alice O'Connor is available on the website; link. Here are a few passages from the introduction:

Nowhere are these intertwined problems more vividly captured than in the complex economic, gender-based, and racial and ethnic divisions of contemporary urban America. In major cities nationwide, overall economic growth is accompanied by higher than average rates of unemployment and poverty, concentrated especially in low-income, working-class minority neighborhoods that have only recently begun to show signs of recovery following decades of steady decline (U.S. Department of Housing and Urban Development 1999). Still, the low-skilled urban workforce, greatly expanded by the “end of welfare,” has little access to local jobs that provide living wages, employment security, and adequate benefits. (1)

Explaining these economic, spatial, and racial divisions is the central purpose of the Multi-City Study of Urban Inequality, a unique inquiry launched in the early 1990s by an interdisciplinary team of social scientists with sponsorship from the Russell Sage and Ford foundations. Based on surveys of households and employers in the metropolitan areas of Atlanta, Boston, Detroit, and Los Angeles, the study affords a comprehensive and systematic look at the roots of inequality in labor markets, residential segregation, and racial attitudes. (2)

The significance of race in urban inequality is to be found not in any single place but in various aspects, and at multiple levels, of social experience: in discriminatory behavior, policies, and institutional practices; in the structural segmentation of labor markets and residential space; and in the attitudes, stereotypes, and ideological belief systems through which people make sense of broader social conditions and determine their own policy preferences. Race has a deep and enduring historical significance as well, still visible in residential color lines constructed by years of racial exclusion, violence, and overtly discriminatory policies; in the persistent racial gaps in education, skills, and capital that stem from opportunity denied; and in the mistrust between minorities and local law-enforcement agencies that has once again erupted around the issue of racial profiling. And race has significance as the basis of a color-coded sense of social hierarchy that affects individual attitudes and behavior, and that is embedded in social structure as well as in shared cultural norms. (5)

The essays in this volume do not tell a simple story. They point to intergroup attitudes, residential and industrial location, discriminatory practices, and the declining prospects for low-skilled workers as important, interlocking sources of inequality across metropolitan America. But they also point to several conclusions about race, both as a shaping force in the distribution of opportunity and as a variable in social scientific analysis. While not offered as a statement of consensus, they do advance the ongoing debate about race and inequality in important ways. (27)

This is careful social scientific research, and it contributes to a much better understanding of the mechanisms that produce and reproduce urban poverty and persistent racial gaps. These mechanisms are both structural and ideational; the processes that lead to discrimination in employment, for example, include both structural factors like urban transportation, and ideational factors such as racial stereotyping by recruiters.

So race continues to be a determining fact of life for many millions of Americans.  But here is the other part of Sugrue's story: our public discourses, and our political process, are virtually silent about these facts.  We simply do not honestly confront the realities of race, whether in electoral competition, in the media, or in other forms of public debate. And we need to.

Decision-making in complex systems

source: The Financial Ninja (link)

How should we make intelligent decisions in contexts in which the object of choice involves the actions of other agents whose choices jointly determine the outcome and where the outcome is unpredictable?  Robert Axelrod and Michael Cohen address these issues in Harnessing Complexity: Organizational Implications of a Scientific Frontier.  They define a complex adaptive system in something like these terms: a body of causal processes and agents whose interactions lead to outcomes that are unpredictable. So the interactions among agents often have unpredictable consequences; and the agents themselves adapt their behavior based on past experiences: "They interact in intricate ways that continually reshape their collective future."  Here is how Axelrod and Cohen put their question:

In a world where many players are all adapting to each other and where the emerging future is extremely hard to predict, what actions should you take? (xi)

This book is about designing organizations and strategies in complex settings, where the full consequences of actions may be hard -- even impossible -- to predict. (2)

Complexity and chaos are often used interchangeably; but Axelrod and Cohen distinguish sharply between them in these terms:

Chaos deals with situations such as turbulence that rapidly become highly disordered and unmanageable.  On the other hand, complexity deals with systems composed of many interacting agents.  While complex systems may be hard to predict, they may also have a good deal of structure and permit improvement by thoughtful intervention. (xv)

Here is a simple current example -- an assembly of 1000 Egyptian citizens in January 2011, interested in figuring out what to do in light of their longstanding grievances and the example of Tunisia. Will the group erupt into defiant demonstration or dissolve into private strategies of self-preservation?  The dynamics of the situation are fundamentally undetermined; the outcome depends on things like who speaks first, how later speakers are influenced by earlier speakers, whether the PA system is working adequately, which positions happen to have a critical mass of supporters, the degree to which the government can make credible threats of retaliation, the presence of experienced organizers, and a dozen other factors.  So we cannot predict whether this group will move towards resistance or accommodation, even when we assume that all present have serious grievances against the Egyptian state.  

The fact of path dependence comes into this understanding of complexity, in that the order of actions by the agents can influence the outcome.  So we could run the Egypt scenario forward multiple times and arrive at different outcomes repeatedly.  We might imagine a tool along the lines of a Monte Carlo simulation that models the range of possible outcomes; and in the sorts of systems Axelrod and Cohen are interested in, the range of outcomes is very wide with no "modal" and most probable outcomes at the core.

The difficulty of prediction in the future development of a complex system derives in part from the adaptiveness of the agents who make it up; but it also derives from the fact of non-linearity of causation in complex systems.  Small influences can have large effects; there is often a discontinuity between the magnitude and direction of a cause and its effect.

What makes prediction especially difficult in these settings is that the forces shaping the future do not add up in a simple, systemwide manner.  Instead, their effects include nonlinear interactions among the components of the system.  The conjunction of a few small events can produce a big effect if their impacts multiply rather than add. (14)

Decision theorists distinguish between situations of parametric rationality and strategic rationality.  In the former the decision maker is playing against nature, with a fixed set of probabilities and causal properties; in the latter the decision maker is playing against and with other rational agents, and the outcome for each depends upon the choices made by all. Game theory offers a mathematical framework for analyzing strategic rationality, while expected utility theory is advanced as a framework for analyzing the problem of choice under risk and uncertainty.  The fundamental finding of game theory is that there are equilibria for multi-person games, both zero-sum and non-zero-sum, for any game that can be formulated in the canonical game matrix of agents' strategies and joint outcomes.  Whether those equilibria are discoverable for ordinary strategic reasoners is a separate question, so the behavioral relevance of the availability of an equilibrium set of strategies is limited.  And here is the key point: neither parametric rationality nor equilibrium-based strategic rationality helps much in the problem of decision-making within a complex adaptive system.

The situation that Axelrod and Cohen describe here is an instance of strategic rationality, but it doesn't yield to the framework of mathematical game theory.  This is because we can't attach payoffs to combinations of strategies for the separate agents; this follows from the unpredictability assumption built into the idea of complexity.  And, second, complex adaptive systems are usually in a dynamic process of change, so that the system never attains an equilibrium state.

Axelrod and Cohen are hoping to provide counsel for how decision makers can "harness" complexity -- that is, how they can design policies and strategies that perhaps push a complex situation in a favorable direction, or that insulate an organization from the worst outcomes that the complex system may produce.

Harnessing complexity ... means deliberately changing the structure of a system in order to increase some measure of performance, and to do so by exploiting an understanding that the system itself is complex. (9)

Axelrod and Cohen make use of three high-level concepts to describe the development of complex adaptive systems: variation, interaction, and selection.  Variation is critical here, as it is in evolutionary biology, because it provides a source of potentially successful innovation -- in strategies, in organizations, in rules of action.  The idea of adaptation is central to their analysis -- in this case, adaptation and modification of strategies by agents in light of current and past success.  Interaction occurs when agents and organizations intersect in the application of their strategies -- often producing unforeseen consequences.  (The strategy of open-source software development is one example that they look at, and the interactions that occurred as open-source innovations encountered closed-source innovations.)  An organization or a population is best served, they argue, when there is a regular source of innovations (variations); when these innovations are implemented in the form of variant strategies; and when it is possible to cultivate more successful variations and to damp out less successful (selection).  Here is how they summarize their view:

Agents, of a variety of types, use their strategies, in patterned interaction, with each other and with artifacts.  Performance measures on the resulting events drive the selection of agents and/or strategies through processes of error-prone copying and recombination, thus changing the frequencies of the types within the system.

And they arrive at eight rules of thumb for "harnessing complexity" when it comes to organizations and social policies:

• Arrange organizational routines to generate a good balance between exploration and exploitation.
• Link processes that generate extreme variation to processes that select with few mistakes in the attribution of credit.
• Build networks of reciprocal interaction that foster trust and cooperation. 
• Assess strategies in light of how their consequences can spread.
• Promote effective neighborhoods.
• Do not sow large failures when reaping small efficiencies.
• Use social activity to support the growth and spread of valued criteria.
• Look for shorter-term, finer-grained measures of success that can usefully stand in for longer-run, broader aims. (156-158)

So how should we understand these heuristics as a conclusion to this analysis?  They function as an "operating manual" for leaders and policy makers attempting to bring about good effects within a population of agents demonstrating adaptive complexity.  And perhaps these are plausible meta-strategies for intervening within a complex social system.

What is worrisome, though, is the implicit functionalism that seems to underlie the book: the idea that agents of good will and having the longterm best interests of the population in mind are making the rules.  But what happened to the predators -- the organized crime figures, the drug lords, the conspirators, the predatorial businesses, the anti-democrats?  Won't they too be looking to exploit (harness) the workings of complexity?  Axelrod's earlier work on repeated prisoners' dilemmas explicitly took into account the availability of strategies designed to exploit the cooperators; and his work on cooperation emphatically makes the point that cooperation is often deployed for anti-social and predatory purposes (cartels, extortion rackets, ...)  (The Evolution of Cooperation: Revised Edition).  Shouldn't this counter-social agency be incorporated into this analysis of complex adaptive systems as well?  As Charles Tilly points out, crime and piracy also depend upon "trust networks" and innovative forms of predation (Trust and Rule).

During the 1980s the Reagan administration wanted to create a "Star Wars" anti-missile shield, and some of their policy makers argued that we could solve the technical challenges because the U.S. had succeeded in putting a man on the moon.  But critics of this military space strategy rejoined, "But the moon didn't fight back;" whereas Soviet scientists and engineers were fully capable of adapting their ICBM technologies to evade the defensive characteristics of a missile shield.  There seems to be something of the same blind spot in this analysis of social complexity; predation and the common good are in competition with each other, and neither has a decisive advantage.

Decision-making in complex systems

source: The Financial Ninja (link)

How should we make intelligent decisions in contexts in which the object of choice involves the actions of other agents whose choices jointly determine the outcome and where the outcome is unpredictable?  Robert Axelrod and Michael Cohen address these issues in Harnessing Complexity: Organizational Implications of a Scientific Frontier.  They define a complex adaptive system in something like these terms: a body of causal processes and agents whose interactions lead to outcomes that are unpredictable. So the interactions among agents often have unpredictable consequences; and the agents themselves adapt their behavior based on past experiences: "They interact in intricate ways that continually reshape their collective future."  Here is how Axelrod and Cohen put their question:

In a world where many players are all adapting to each other and where the emerging future is extremely hard to predict, what actions should you take? (xi)

This book is about designing organizations and strategies in complex settings, where the full consequences of actions may be hard -- even impossible -- to predict. (2)

Complexity and chaos are often used interchangeably; but Axelrod and Cohen distinguish sharply between them in these terms:

Chaos deals with situations such as turbulence that rapidly become highly disordered and unmanageable.  On the other hand, complexity deals with systems composed of many interacting agents.  While complex systems may be hard to predict, they may also have a good deal of structure and permit improvement by thoughtful intervention. (xv)

Here is a simple current example -- an assembly of 1000 Egyptian citizens in January 2011, interested in figuring out what to do in light of their longstanding grievances and the example of Tunisia. Will the group erupt into defiant demonstration or dissolve into private strategies of self-preservation?  The dynamics of the situation are fundamentally undetermined; the outcome depends on things like who speaks first, how later speakers are influenced by earlier speakers, whether the PA system is working adequately, which positions happen to have a critical mass of supporters, the degree to which the government can make credible threats of retaliation, the presence of experienced organizers, and a dozen other factors.  So we cannot predict whether this group will move towards resistance or accommodation, even when we assume that all present have serious grievances against the Egyptian state.  

The fact of path dependence comes into this understanding of complexity, in that the order of actions by the agents can influence the outcome.  So we could run the Egypt scenario forward multiple times and arrive at different outcomes repeatedly.  We might imagine a tool along the lines of a Monte Carlo simulation that models the range of possible outcomes; and in the sorts of systems Axelrod and Cohen are interested in, the range of outcomes is very wide with no "modal" and most probable outcomes at the core.

The difficulty of prediction in the future development of a complex system derives in part from the adaptiveness of the agents who make it up; but it also derives from the fact of non-linearity of causation in complex systems.  Small influences can have large effects; there is often a discontinuity between the magnitude and direction of a cause and its effect.

What makes prediction especially difficult in these settings is that the forces shaping the future do not add up in a simple, systemwide manner.  Instead, their effects include nonlinear interactions among the components of the system.  The conjunction of a few small events can produce a big effect if their impacts multiply rather than add. (14)

Decision theorists distinguish between situations of parametric rationality and strategic rationality.  In the former the decision maker is playing against nature, with a fixed set of probabilities and causal properties; in the latter the decision maker is playing against and with other rational agents, and the outcome for each depends upon the choices made by all. Game theory offers a mathematical framework for analyzing strategic rationality, while expected utility theory is advanced as a framework for analyzing the problem of choice under risk and uncertainty.  The fundamental finding of game theory is that there are equilibria for multi-person games, both zero-sum and non-zero-sum, for any game that can be formulated in the canonical game matrix of agents' strategies and joint outcomes.  Whether those equilibria are discoverable for ordinary strategic reasoners is a separate question, so the behavioral relevance of the availability of an equilibrium set of strategies is limited.  And here is the key point: neither parametric rationality nor equilibrium-based strategic rationality helps much in the problem of decision-making within a complex adaptive system.

The situation that Axelrod and Cohen describe here is an instance of strategic rationality, but it doesn't yield to the framework of mathematical game theory.  This is because we can't attach payoffs to combinations of strategies for the separate agents; this follows from the unpredictability assumption built into the idea of complexity.  And, second, complex adaptive systems are usually in a dynamic process of change, so that the system never attains an equilibrium state.

Axelrod and Cohen are hoping to provide counsel for how decision makers can "harness" complexity -- that is, how they can design policies and strategies that perhaps push a complex situation in a favorable direction, or that insulate an organization from the worst outcomes that the complex system may produce.

Harnessing complexity ... means deliberately changing the structure of a system in order to increase some measure of performance, and to do so by exploiting an understanding that the system itself is complex. (9)

Axelrod and Cohen make use of three high-level concepts to describe the development of complex adaptive systems: variation, interaction, and selection.  Variation is critical here, as it is in evolutionary biology, because it provides a source of potentially successful innovation -- in strategies, in organizations, in rules of action.  The idea of adaptation is central to their analysis -- in this case, adaptation and modification of strategies by agents in light of current and past success.  Interaction occurs when agents and organizations intersect in the application of their strategies -- often producing unforeseen consequences.  (The strategy of open-source software development is one example that they look at, and the interactions that occurred as open-source innovations encountered closed-source innovations.)  An organization or a population is best served, they argue, when there is a regular source of innovations (variations); when these innovations are implemented in the form of variant strategies; and when it is possible to cultivate more successful variations and to damp out less successful (selection).  Here is how they summarize their view:

Agents, of a variety of types, use their strategies, in patterned interaction, with each other and with artifacts.  Performance measures on the resulting events drive the selection of agents and/or strategies through processes of error-prone copying and recombination, thus changing the frequencies of the types within the system.

And they arrive at eight rules of thumb for "harnessing complexity" when it comes to organizations and social policies:

• Arrange organizational routines to generate a good balance between exploration and exploitation.
• Link processes that generate extreme variation to processes that select with few mistakes in the attribution of credit.
• Build networks of reciprocal interaction that foster trust and cooperation. 
• Assess strategies in light of how their consequences can spread.
• Promote effective neighborhoods.
• Do not sow large failures when reaping small efficiencies.
• Use social activity to support the growth and spread of valued criteria.
• Look for shorter-term, finer-grained measures of success that can usefully stand in for longer-run, broader aims. (156-158)

So how should we understand these heuristics as a conclusion to this analysis?  They function as an "operating manual" for leaders and policy makers attempting to bring about good effects within a population of agents demonstrating adaptive complexity.  And perhaps these are plausible meta-strategies for intervening within a complex social system.

What is worrisome, though, is the implicit functionalism that seems to underlie the book: the idea that agents of good will and having the longterm best interests of the population in mind are making the rules.  But what happened to the predators -- the organized crime figures, the drug lords, the conspirators, the predatorial businesses, the anti-democrats?  Won't they too be looking to exploit (harness) the workings of complexity?  Axelrod's earlier work on repeated prisoners' dilemmas explicitly took into account the availability of strategies designed to exploit the cooperators; and his work on cooperation emphatically makes the point that cooperation is often deployed for anti-social and predatory purposes (cartels, extortion rackets, ...)  (The Evolution of Cooperation: Revised Edition).  Shouldn't this counter-social agency be incorporated into this analysis of complex adaptive systems as well?  As Charles Tilly points out, crime and piracy also depend upon "trust networks" and innovative forms of predation (Trust and Rule).

During the 1980s the Reagan administration wanted to create a "Star Wars" anti-missile shield, and some of their policy makers argued that we could solve the technical challenges because the U.S. had succeeded in putting a man on the moon.  But critics of this military space strategy rejoined, "But the moon didn't fight back;" whereas Soviet scientists and engineers were fully capable of adapting their ICBM technologies to evade the defensive characteristics of a missile shield.  There seems to be something of the same blind spot in this analysis of social complexity; predation and the common good are in competition with each other, and neither has a decisive advantage.

Decision-making in complex systems

source: The Financial Ninja (link)

How should we make intelligent decisions in contexts in which the object of choice involves the actions of other agents whose choices jointly determine the outcome and where the outcome is unpredictable?  Robert Axelrod and Michael Cohen address these issues in Harnessing Complexity: Organizational Implications of a Scientific Frontier.  They define a complex adaptive system in something like these terms: a body of causal processes and agents whose interactions lead to outcomes that are unpredictable. So the interactions among agents often have unpredictable consequences; and the agents themselves adapt their behavior based on past experiences: "They interact in intricate ways that continually reshape their collective future."  Here is how Axelrod and Cohen put their question:

In a world where many players are all adapting to each other and where the emerging future is extremely hard to predict, what actions should you take? (xi)

This book is about designing organizations and strategies in complex settings, where the full consequences of actions may be hard -- even impossible -- to predict. (2)

Complexity and chaos are often used interchangeably; but Axelrod and Cohen distinguish sharply between them in these terms:

Chaos deals with situations such as turbulence that rapidly become highly disordered and unmanageable.  On the other hand, complexity deals with systems composed of many interacting agents.  While complex systems may be hard to predict, they may also have a good deal of structure and permit improvement by thoughtful intervention. (xv)

Here is a simple current example -- an assembly of 1000 Egyptian citizens in January 2011, interested in figuring out what to do in light of their longstanding grievances and the example of Tunisia. Will the group erupt into defiant demonstration or dissolve into private strategies of self-preservation?  The dynamics of the situation are fundamentally undetermined; the outcome depends on things like who speaks first, how later speakers are influenced by earlier speakers, whether the PA system is working adequately, which positions happen to have a critical mass of supporters, the degree to which the government can make credible threats of retaliation, the presence of experienced organizers, and a dozen other factors.  So we cannot predict whether this group will move towards resistance or accommodation, even when we assume that all present have serious grievances against the Egyptian state.  

The fact of path dependence comes into this understanding of complexity, in that the order of actions by the agents can influence the outcome.  So we could run the Egypt scenario forward multiple times and arrive at different outcomes repeatedly.  We might imagine a tool along the lines of a Monte Carlo simulation that models the range of possible outcomes; and in the sorts of systems Axelrod and Cohen are interested in, the range of outcomes is very wide with no "modal" and most probable outcomes at the core.

The difficulty of prediction in the future development of a complex system derives in part from the adaptiveness of the agents who make it up; but it also derives from the fact of non-linearity of causation in complex systems.  Small influences can have large effects; there is often a discontinuity between the magnitude and direction of a cause and its effect.

What makes prediction especially difficult in these settings is that the forces shaping the future do not add up in a simple, systemwide manner.  Instead, their effects include nonlinear interactions among the components of the system.  The conjunction of a few small events can produce a big effect if their impacts multiply rather than add. (14)

Decision theorists distinguish between situations of parametric rationality and strategic rationality.  In the former the decision maker is playing against nature, with a fixed set of probabilities and causal properties; in the latter the decision maker is playing against and with other rational agents, and the outcome for each depends upon the choices made by all. Game theory offers a mathematical framework for analyzing strategic rationality, while expected utility theory is advanced as a framework for analyzing the problem of choice under risk and uncertainty.  The fundamental finding of game theory is that there are equilibria for multi-person games, both zero-sum and non-zero-sum, for any game that can be formulated in the canonical game matrix of agents' strategies and joint outcomes.  Whether those equilibria are discoverable for ordinary strategic reasoners is a separate question, so the behavioral relevance of the availability of an equilibrium set of strategies is limited.  And here is the key point: neither parametric rationality nor equilibrium-based strategic rationality helps much in the problem of decision-making within a complex adaptive system.

The situation that Axelrod and Cohen describe here is an instance of strategic rationality, but it doesn't yield to the framework of mathematical game theory.  This is because we can't attach payoffs to combinations of strategies for the separate agents; this follows from the unpredictability assumption built into the idea of complexity.  And, second, complex adaptive systems are usually in a dynamic process of change, so that the system never attains an equilibrium state.

Axelrod and Cohen are hoping to provide counsel for how decision makers can "harness" complexity -- that is, how they can design policies and strategies that perhaps push a complex situation in a favorable direction, or that insulate an organization from the worst outcomes that the complex system may produce.

Harnessing complexity ... means deliberately changing the structure of a system in order to increase some measure of performance, and to do so by exploiting an understanding that the system itself is complex. (9)

Axelrod and Cohen make use of three high-level concepts to describe the development of complex adaptive systems: variation, interaction, and selection.  Variation is critical here, as it is in evolutionary biology, because it provides a source of potentially successful innovation -- in strategies, in organizations, in rules of action.  The idea of adaptation is central to their analysis -- in this case, adaptation and modification of strategies by agents in light of current and past success.  Interaction occurs when agents and organizations intersect in the application of their strategies -- often producing unforeseen consequences.  (The strategy of open-source software development is one example that they look at, and the interactions that occurred as open-source innovations encountered closed-source innovations.)  An organization or a population is best served, they argue, when there is a regular source of innovations (variations); when these innovations are implemented in the form of variant strategies; and when it is possible to cultivate more successful variations and to damp out less successful (selection).  Here is how they summarize their view:

Agents, of a variety of types, use their strategies, in patterned interaction, with each other and with artifacts.  Performance measures on the resulting events drive the selection of agents and/or strategies through processes of error-prone copying and recombination, thus changing the frequencies of the types within the system.

And they arrive at eight rules of thumb for "harnessing complexity" when it comes to organizations and social policies:

• Arrange organizational routines to generate a good balance between exploration and exploitation.
• Link processes that generate extreme variation to processes that select with few mistakes in the attribution of credit.
• Build networks of reciprocal interaction that foster trust and cooperation. 
• Assess strategies in light of how their consequences can spread.
• Promote effective neighborhoods.
• Do not sow large failures when reaping small efficiencies.
• Use social activity to support the growth and spread of valued criteria.
• Look for shorter-term, finer-grained measures of success that can usefully stand in for longer-run, broader aims. (156-158)

So how should we understand these heuristics as a conclusion to this analysis?  They function as an "operating manual" for leaders and policy makers attempting to bring about good effects within a population of agents demonstrating adaptive complexity.  And perhaps these are plausible meta-strategies for intervening within a complex social system.

What is worrisome, though, is the implicit functionalism that seems to underlie the book: the idea that agents of good will and having the longterm best interests of the population in mind are making the rules.  But what happened to the predators -- the organized crime figures, the drug lords, the conspirators, the predatorial businesses, the anti-democrats?  Won't they too be looking to exploit (harness) the workings of complexity?  Axelrod's earlier work on repeated prisoners' dilemmas explicitly took into account the availability of strategies designed to exploit the cooperators; and his work on cooperation emphatically makes the point that cooperation is often deployed for anti-social and predatory purposes (cartels, extortion rackets, ...)  (The Evolution of Cooperation: Revised Edition).  Shouldn't this counter-social agency be incorporated into this analysis of complex adaptive systems as well?  As Charles Tilly points out, crime and piracy also depend upon "trust networks" and innovative forms of predation (Trust and Rule).

During the 1980s the Reagan administration wanted to create a "Star Wars" anti-missile shield, and some of their policy makers argued that we could solve the technical challenges because the U.S. had succeeded in putting a man on the moon.  But critics of this military space strategy rejoined, "But the moon didn't fight back;" whereas Soviet scientists and engineers were fully capable of adapting their ICBM technologies to evade the defensive characteristics of a missile shield.  There seems to be something of the same blind spot in this analysis of social complexity; predation and the common good are in competition with each other, and neither has a decisive advantage.

Herbert Simon's satisficing life

Herbert Simon was a remarkably fertile thinker in the social and "artificial" sciences (The Sciences of the Artificial - 3rd Edition (1969, first edition)).  His most celebrated idea was the notion of "satisficing" rather than "optimizing" or "maximizing" in decision-making; he put forward a theory of ordinary decision-making that conformed more closely to the ways that actual people reason rather than the heroic abstractions of expected utility theory.

Essentially the concept of satisficing takes the cost of collecting additional information into account as a decision maker searches for a solution to a problem -- where to eat for dinner, which university to attend, which product to emphasize in a company's short-term strategy.  And the theory commends the idea that we are best served overall by accepting the "good-enough" solution rather than searching indefinitely for the best solution.  Rather than attempting to inventory all possible choices available at a given point in time and assigning them utilities and probabilities, the satisficing theory recommends setting parameters for a problem of choice, and then selecting the first solution that comes along that satisfies these parameters.  It means searching for a solution that is "good enough" rather than optimal.

And why not go for the optimal solution?  Because the cost of collecting the additional information associated with a broader choice set may well exceed the total benefit of the current decision.  This is obvious in the case of the decision of which restaurant to go to; slightly less obvious in the case of the decision of which university to attend; and perhaps flatly unpersuasive in the case of decisions where the outcome can influence life and death.

I've described the theory of satisficing in a little detail here for an unexpected reason: Simon took some interest in the art of autobiography, and it turns out that he interprets his own life as a series of satisficing decisions.  His autobiography Models of My Life appeared in 1996, and it's an interesting narrative of the intellectual and personal choices that led Simon from Milwaukee to Pittsburgh and beyond.

The idea is particularly apt for Simon's view about how a life unfolds.  He rejects the idea that one's life has an overriding theme.  He discusses the fact that the title of the book is a plural noun -- "Models of My Life".

There is a further reason for using the plural [models].  It is a denial -- a denial that a life, at least my life, has a central theme, a unifying thread running through it. True, there are themes (again the plural), some of the threads brighter or thicker or stronger than others.  Perhaps clearest is the theme of the scientist and teacher, carrying on his persistent heuristic search, seeking the Holy Grail of truth about human decision making.  In my case, even that thread is woven of finer strands: the political scientist, the organization theorist, the economist, the management scientist, the computer scientist, the psychologist, the philosopher of science. (xviii)

Rather than one underlying theme that underlies a person's biography and career, there are multiple choices, directions, and emphases -- that add up to a woven lifetime of contribution when the choices work out well.

Simon accepts the implication that this vision of a life presents: that there is no single "self" underlying all these changes and choices:

Which of the wanderers through these different mazes will step forward at the call for the real Herbert Simon?  All of them; for the "real" self is an illusion.  We live each hour in context, different contexts for different hours.... We act out our lives within the mazes in which Nature and society place us. (xviii-xix)

The analogy between daily decision-making and living a life is a direct one: instead of setting upon a course with very specific goals and objectives, and then taking the steps necessary to bring about the achievement of that system of goals, Simon is recommending a more local form of life decision-making. Build capacities, recognize opportunities, take risks, and build a life as a result of a series of local choices.  It is a form of bounded rationality for living rather than an expression of a fully developed life plan.  So we might say that Simon's "philosophy of living" is entirely consistent with his theory of bounded rationality.

There are a few real surprises in the book -- for example, a conversation between Simon and Jorge Luis Borges in Argentina in 1970.  Simon was fascinated by Borges' use of the idea of a labyrinth in his novels, and wanted to find out from Borges how he was led to this family of metaphors.  Simon himself was drawn to the idea of a series of choices as a maze -- incorporating the insight that there are always unexplored outcomes behind the avenues not taken.  So a labyrinth is a good metaphor for choice within uncertainty and risk.

I have encountered many branches in the maze of my life's path, where I have followed now the left fork, now the right.  The metaphor of the maze is irresistible to someone who has devoted his scientific career to understanding human choice. (xvii)

Here is a snippet of the conversation between Simon and Borges as quoted in the book:

SIMON: I want to know how it was that the labyrinth entered into your field of vision, into your concepts, so that you incorporated in your stories.

BORGES: I remember having seen an engraving of the labyrinth in a French book -- when I was a boy. It was a circular building without doors but with many windows. I used to gaze at this engraving and think that if I brought a loupe close to it, it would reveal the Minotaur.

SIMON: Did you see it?

BORGES: Actually my eyesight was never good enough.  Soon I discovered something of the complexity of life, as if it were a game. In this I am not referring to chess. 

...

SIMON: What is the connection between the labyrinth of the Minotaur and your labyrinth, which calls for continual choice? Does the analogy go beyond the general concept?

BORGES: When I write, I don't think in terms of teaching. I think that my stories, in some way, are given to me, and my task is to narrate them. I neither search for implicit connotations nor start out with abstract ideas; I am not one who plays with symbols. But if there is some transcendental explanation of one of my stories, it is not for me to discover it, that is the task of the critics and the readers.

And a final surprise -- it emerges from the conversation that Borges had read "a very interesting book" early in his life, Bertrand Russell's Introduction to Mathematical Philosophy -- not exactly the most predictable influence on the creator of magical realism.  And Russell's mathematical logic was likewise a formative influence for Simon, at a comparably early age.

There is an interesting short section where Simon discusses one of the directions he did not take in his own personal career maze -- the step of trying to become a college president at Carnegie Mellon or elsewhere (262 ff.).  Simon writes briefly about the reasons why this might have been a realistic aspiration for him -- a history of administrative competence at the department level and a stellar academic record.  But he decided not to pursue the presidency at CMU:

However that may be, I did not seriously consider taking on the context. ... I have never regretted the decision, especially in view of Dick's stellar performance on the job, a performance made possible by a "deviousness" that our colleague Leland Hazard admiringly attributed to him, and that I surely did not possess. (263)

He adds that he didn't have the personality needed to cultivate the community of wealthy businessmen whose support would be essential to Carnegie: "In fact, the close association with the business community that is essential for effective performance as president of a university such as Carnegie Mellon would have been uncomfortable for me" (263).

But here is the way this discussion strikes me (as a person whose career did take him in that direction). Simon gives no evidence here of understanding even the most basic facts about this domain of choice: what the job of president actually is; what the qualities of personality and leadership are that would lead to success; and what the intellectual satisfactions might be in the event that he became a university president. He seems to be working from a very shallow stereotyped view of the job of university president. In other words, Simon had none of the information that would be needed to make an informed career choice about this option. And this suggests that his decision-making on this issue was narrowly bounded indeed -- driven by a few stereotyped assumptions that were probably a poor guide to the reality.

(Here is a lecture by Herbert Simon on organizations, public administration, and markets:)