January 25, 2020

Twin Pillars: Upholding National Security and National Innovation in Emerging Technologies Governance


January 23, 2020



In an era of global technological competition and diffusion of innovation, the United States must uphold the twin pillars of national security and national innovation. The overall success of the U.S. federal government in emerging technologies governance is at best a mixed case and is overall inadequate to the scale and stakes of the challenges and opportunities ahead. This report assesses the current effectiveness of the U.S. federal government approach to emerging technologies with recommended actions to enhance governance in domestic and international contexts.

This report is made possible by the generous support of Leonardo DRS.

Christian S. Haig
Katherine Schmidt

Artificial Intelligence and the Manufacturing of Reality

by Christopher Paul and Marek N. Posard

January 20, 2020

In 2016, a third of surveyed Americans told researchers they believed the government was concealing what they knew about the “North Dakota Crash,” a conspiracy made up for the purposes of the survey by the researchers themselves. This crash never happened, but it highlights the flaws humans carry with them in deciding what is or is not real.

The internet and other technologies have made it easier to weaponize and exploit these flaws, beguiling more people faster and more compellingly than ever before. It is likely artificial intelligence will be used to exploit the weaknesses inherent in human nature at a scale, speed, and level of effectiveness previously unseen. Adversaries like Russia could pursue goals for using these manipulations to subtly reshape how targets view the world around them, effectively manufacturing their reality. If even some of our predictions are accurate, all governance reliant on public opinion, mass perception, or citizen participation is at risk.

One characteristic human foible is how easily we can falsely redefine what we experience. This flaw, called the Thomas Theorem, suggests, “If men define situations as real, they are real in their consequences.”[1] Put another way, humans not only respond to the objective features of their situations but also to their own subjective interpretations of those situations, even when these beliefs are factually wrong. Other shortcomings include our willingness to believe information that is not true and a propensity to be as easily influenced by emotional appeals as reason, as demonstrated by the “North Dakota Crash” falsehood.[2]

Machines can also be taught to exploit these flaws more effectively than humans: Artificial intelligence algorithms can test what content works and what does not over and over again on millions of people at high speed, until their targets react as desired.[3]

Consider the role of Russia in the 2016 British vote to withdraw from the European Union (Brexit). There is evidence that Russian-linked Twitter accounts sent more than a thousand tweets from 3,800 accounts promoting a pro-Brexit vote on the day of voting. Such tweets appeared to have fanned the flames of the pro-Brexit camp on Twitter over time, with the anti-Brexit camp reacting a few days before the election.[4]

Artificial intelligence algorithms can test what content works and what does not over and over again on millions of people at high speed, until their targets react as desired.

Share on Twitter

Emerging technologies including generative adversarial networks, natural language processing, and quantum computing could make such scenarios far more effective. In the future, for example, Russian actors could tailor the messages in these tweets using a combination of the characteristics of the recipients and their behaviors on various online platforms based on user data they legally buy from data brokers, illegal data they purchase from hackers, and data they retrieve themselves.

These opportunities are available today, and some may become increasingly easier to exploit with artificial intelligence. In the future, for example, adversaries like Russia could query these data streams to tailor their messages and test them on social media platforms to identify the most effective messages. Such adversaries could then alter these tested messages accordingly and deploy them to users and those in their social networks via a wide variety of online media (e.g., traditional social media platforms, augmented reality, virtual reality devices, or noninvasive brain-computer interfaces).

Currently, much of this is done manually, but artificial intelligence allows for a change in scale. Emerging technologies will allow this loop of iterative refinement to occur almost instantaneously, in real time, and at a scale affecting far more people than was seen during the Brexit vote.[5]

Manufacturing Reality Isn't New

Humans are and have always been vulnerable to being tricked, provoked, conditioned, deceived, or otherwise manipulated. Since at least the 1960s, the Soviet military and subsequent Russian organizations recognized opportunities for exploiting this vulnerability. That is why the Soviets developed a formal research program—called reflexive control theory—to model how one could manipulate targets' perceptions of reality.[6] The theory focuses on ways to strategically transmit information to targets in ways that subtly change the motives and logics of their decisions. The end goal is to get people to do something by making them believe it is in their best interest, even if it is not. While the Russians weaponized reflexive control theory, Madison Avenue used similar logic to evoke emotion—and sell products to American consumers.

There are countless examples of the Russians using reflexive control. In October 1993, for example, Russian lawmakers took over their own Parliament to advocate for a return to communism. The authorities decided to allow the rebels to occupy a police communications post, giving them access to a secure communications channel then used by police to transmit false conversations between government officials about a plan to storm the occupied Parliament building. After hearing this message, one of the rebel leaders, Parliament Speaker Ruslan Kashbulatov, called on the crowd of supporters to seize a local television station, the first step in a coup. By getting Kashbulatov to make this public request for violence, Russian authorities created justification for storming the Parliament and arresting the dissidents.

Humans are and have always been vulnerable to being tricked, provoked, conditioned, deceived, or otherwise manipulated.

Share on Twitter

Similarly, the East Germans recognized the power of manufactured reality for maintaining internal control. Starting around the 1970s, their Ministry of State Security known as the Stasi expanded the scope of their work from physical abuse of targets—such as torture or executions—to include a certain kind of psychological abuse. The Stasi called this technique Zersetzung, which loosely translates as “decomposition.” It was an organized, scientific effort to collect information about people and then use it in ways that destroyed their sense of self in private and public life. The Stasi broke into the homes of targets to rearrange their furniture, steal items of clothing, or turn off their clocks. They would send compromising photos to loved ones, discredit people in their workplace, remove children from dissident parents or trick them into believing they were mentally ill, something known today as gaslighting. Victims of decomposition struggled to understand why their lives were becoming unrecognizable.

But these Russian and Stasi tactics required careful research and execution to disrupt or manipulate the targets one at a time. The contemporary information environment and modern tools, including artificial intelligence, could slash the transaction costs of such manipulation. The following methods enable a dramatic scaling in the weaponization of information.

Manufacturing Reality Meets Modern Tools

Big Data

By 2025, some predict humans will produce around 463 exabytes each day, enough to fill 212 million DVDs. Their personal data describes individuals with a frightening level of detail. With access to such data from legitimate and illegitimate data brokers, artificial intelligence could combine and match their Amazon purchases, Google searches, tweets, Facebook photos, 401k account balances, credit history, their Netflix viewing habits, online searches, etc.

Precision Marketing and Microtargeting

Big data can help identify which individuals are like others—and which similarities matter—based on status updates, drafts of posted videos, facial-recognition data, phone calls, and text messages. Artificial intelligence will make this increasingly easier in the future. If a manipulator knows enough about one person to send him or her messages to provoke or inspire, sending the same messages to similar individuals should produce similar results at scale.

Shallowfakes, Deepfakes, and Social Bots

Shallowfakes are created by manually doctoring images, video, or audio. Deepfakes use artificial intelligence to superimpose images, videos, and recordings onto source files to subtly alter who is doing what. Artificial intelligence–driven “social bots” can carry on conversations as if they were an actual person. Artificial intelligence will enable a dramatic rise in the number of such inauthentic “people” and make it even harder to tell human conversations from artificial ones.

Generative Adversarial Networks

One of the technologies that helps make deepfakes so realistic is the use of a class of machine-learning systems called generative adversarial networks (GANs). These networks have two neural network models, a generator and discriminator. The generator takes training data and learns how to recreate it, while the discriminator tries to distinguish the training data from the recreated data produced by the generator. The two artificial intelligence actors play the game repeatedly, each getting iteratively better at its job.

At the moment, generative adversarial networks are being used to build deepfakes for fake porn videos and political satire. But their power to manipulate should worry us for several reasons. First, they can use inputs like big data and precision marketing to scale the manufacturing of content like deepfakes. Second, the iterative competition between generator and discriminator neural networks takes place at machine speed and is accurate. Third, the same logic that underpins generative adversarial networks can be applied to other practices.


Artificial intelligence–driven social bots that are now sending you car or skin care advertisements could start chatting with you—actually, experimenting on you—to test what content will elicit the strongest reactions. Your responses would be fed back into a generative adversarial network–like system where you and countless others play the role of discriminator, all helping the artificial intelligence learn how better to manipulate you in future stimuli. You, or others like you, could slowly be pushed into changing your attitudes, preferences, or behaviors toward other groups or on foreign or domestic policy issues. Whoever is first to develop and employ such systems could easily prey on wide swaths of the public for years to come.

Defending against such massive manipulation will be particularly tricky given the current social media landscape, which allows for the easy multiplication of inauthentic individuals, personas, and accounts through the use of bots or other forms of automation. There are several possible ways to develop protective efforts against using artificial intelligence for experimentation on humans. Government regulation is one way. The government could regulate standards of identity authentication by social media companies. This could be done by fining companies that fail to meet these standards, providing tax breaks for companies that meet them, or taxing companies for each user who opens a new social media account. Further, the U.S. Securities and Exchange Commission could develop standards for publicly traded social media companies to report authentic, active users in their filings. In 2016, for example, the Securities and Exchange Commission raised concerns about the ways that Twitter reported Daily and Monthly Active Users on their platforms. The Securities and Exchange Commission could develop standards of reporting authentic users, which would not only protect investors but also force transparency for who is behind what account on these platforms.

AI-driven social bots that now send you car or skin care ads could chat with you—experiment on you—to test what content will elicit the strongest reactions.

Share on Twitter

Alternatively, social media providers might take steps to limit multiple or inauthentic accounts at their own initiative. If these firms believe it is in their interest to do so, there are numerous ways available to them to reduce the propagation of inauthentic accounts. Such measures might include more rigorous identity authentication and verification or perhaps imposition of a nominal cost associated with the opening of each account. Legitimate users would be only minimally inconvenienced by such a cost, but bot herders might find such costs prohibitive or the procedural hurdles of arranging payment for innumerable false accounts a barrier. Further, the presence of a money trail would be one more way to identify inauthentic accounts or even trace back to the perpetrators.

Combined, such steps would align costs and incentives against users opening thousands of artificial intelligence–backed inauthentic accounts. Regardless of new technologies that evolve, the producers of inauthentic content will always need a way to distribute their manipulative messages. It is these distribution channels that will become a first line of defense from the weaponization of our reality.

Unfortunately, current incentive structures leave little impetus for social media firms to take such steps, as the nature of ad revenues reward larger numbers of active accounts, whether or not they are managed by real people or through automation. However, one could imagine users who value interacting with authentic persons rewarding social media companies that take steps to ensure the authenticity of accounts with their business and traffic. Authenticity could become a valued brand attribute for some social media platforms and be incentivized by consumer behavior.

If a third of Americans do in fact believe the government is concealing something about the made-up “North Dakota Crash,” imagine how many more would believe this fictitious event after wading through a deluge of bots, generative adversarial networks, and deepfakes on the internet targeting them with precision-level accuracy?


  • [1] Daniel Chandler and Rod Munday, A Dictionary of Media and Communication, Oxford: Oxford University Press, 2011.
  • [2]Tapan K. Panda, Tapas K. Panda, and Kamalesh Mishra, “Does Emotional Appeal Work in Advertising? The Rationality Behind Using Emotional Appeal to Create Favorable Brand Attitude,“ IUP Journal of Brand Management 10, no. 2 (June 2013): 7-23.
  • [3] Cade Metz, “Finally, a Machine That Can Finish Your Sentence,” New York Times, June 7, 2019.
  • [4] Miha Grčar, Cherepnalkoski, Darko, Mozetič, Igor, and Kralj Novak, Petra “Stance and influence of Twitter users regarding the Brexit referendum,” Comput Soc Netw 4, 6 (2017) doi:10.1186/s40649-017-0042-6
  • [5] David Ingram, “Facebook Says 126 Million Americans May Have Seen Russia-Linked Political Posts,” Reuters, October 30, 2017. This story reports that Russian operatives made about 80,000 Facebook posts over the two years prior to the 2016 U.S. election, and that these posts were seen by roughly 126 million Americans. All signs suggest this engagement took place with minimal automation, that is, these posts were typed by human operatives using human-curated false personas. Imagine what might be achieved with automated processes opening accounts and establishing personas, and with automated content generation and posting. At machine scale, the same level of engagement might be achieved in orders of magnitude less time (days or weeks instead of years) and at orders of magnitude less cost (computer time is always cheaper than human labor).
  • [6] Timothy Thomas, “Russia's Reflexive Control Theory and the Military.” Journal of Slavic Military Studies Vol. 17, Issue 2 (2004): Pp. 237-256.

Christopher Paul is a senior social scientist and Marek N. Posard is an associate sociologist at the nonprofit, nonpartisan RAND Corporation. Paul's research has addressed information operations and the information environment for more than a decade. Posard's research has focused on military sociology, survey methods, and experimental methods.

This commentary originally appeared on Strategy Bridge on January 20, 2020. Commentary gives RAND researchers a platform to convey insights based on their professional expertise and often on their peer-reviewed research and analysis.

Managing the rising influence of nationalism

Supporters of the far-right lawmaker and presidential candidate for the Social Liberal Party (PSL), Jair Bolsonaro, celebrate in Rio de Janeiro, after the former army captain won Brazil's presidential election


Managing the Rising Influence of Nationalism

There is an urgent need for global responses to a host of shared challenges, from climate change and technological disruption to financial imbalances. And yet, perversely, atavistic politics that seek to divide people are returning to the fore in democracies and autocracies alike. What is going on?

The rise and fall and rise of nationalism

All the world’s nations and nation states are organized around myths. In 1983, the Irish historian Benedict Anderson described how political leaders beginning in the 18th century created “imagined communities” in order to build modern, industrialized European states; more recently, Israeli historian Yuval Harari has explained how humanity used “lies” and “stories” to transition from small hunter‑gatherer tribes to large, complex political entities.

The curation of common religious beliefs, the evolution of common norms and behaviour, and the construction of common narratives enabled the emergence of strong polities, united by shared cultures. With the creation of the United Nations in 1945, the international community has recognized highly diverse political entities (former empires, small feudal autocracies or kingdoms, newly‑created and long‑standing democratically constituted nations) as nation states – granting nations the legal status of states under international law and thus offering a further level of legitimacy to national myths.

However, the near simultaneous adoption in 1948 of the Universal Declaration of Human Rights and the founding of the Bretton Woods institutions created a powerful counterforce: because both were forged at a moment of US‑led Western hegemony, the sense in the West was that the world would witness the expansion of nation states governed by increasingly homogenous political systems with a shared commitment to democracy and open markets.

The process of globalization demanded that all states adapt to being part of a shared project and subject themselves to its norms and laws. Globalization thus implied the end of the identity‑led politics that had led to the creation of nation states in the first place, as well as to the national chauvinism that had fuelled the world wars of the 20th century.

The United States and its allies provided the security umbrella under which many nations designed their routes to this modern statehood. As importantly, the fact that the global superpower was a melting pot of former nationalities challenged the idea of concentrating statehood around ethnic identity. America’s civil rights movement in the 1960s showed how a country could progress towards the ideal of a state based on a common set of values rather than ethnicity.

Under US protection, the European Union became the vanguard of this process of post‑nationalism. With the creation of the European Coal and Steel Community followed by the European Economic Community during the 1950s, and then the establishment of the European Union in 1993, member states pooled their sovereign interests into a supranational organization that would help deliver economic and other policy outcomes they could not deliver alone. To the extent that there was an emerging European identity, it was anti‑nationalist.

But the period since 2016 has brutally exposed the exceptional nature of this moment in human history. And it has illustrated the continuing power of myths and of a distinct sense of national identity to mobilize groups of people towards shared goals.

The failure of globalization to deliver its gains equitably across Western societies over the last 30 years has created an upswell of resentment against those politicians and political parties who championed globalization. President Donald Trump was elected to office on the back of his rejection of “globalism” and by championing “Americans first”. Supporters of Brexit pine for a return to Britain’s identity as a “sceptred isle”, separate from the European continent – a “great trading nation” plying the waves to the distant shores of the former British Empire.

America and Britain are not the only countries where the nostalgia for national identity is ascendant. In Russia and Hungary, the threat of global integration has intensified a historical rejection of outsiders and sense of national grievance. Their leaders have turned to national culture and identity as the basis for resistance.

China has carefully curated its period of historical exploitation by Western powers to strengthen popular support for its return as a great power. In India, Prime Minister Narendra Modi and the Bharatiya Janata Party have reawakened Hindu resentment against the country’s minority Muslim community and have used myths about Hindu India’s glorious past to try to galvanize Indians towards its potential great future.

Because of the insecurity it arouses in others, the rise of nationalism begets the rise of more nationalism. South Korea and Japan, which both fear China’s growing power, have allowed historical disputes to fuel their national resentment of the other. Nationalism can also serve more utilitarian purposes. President Jair Bolsonaro claims that he wants to decolonize Brazil from the foreign environmental NGOs that prevent it from exploiting the Amazon.

President Andrés Manuel López Obrador is demanding reparations for Mexico from the Spanish government for the predations of the “conquistadores”. This swirl of historical facts, myths, slights and grievances are now turbocharged by social media in ways that can prevent political leaders from retaking control of the narrative, and policy, later.

Managing and leveraging the new nationalism

States around the world are at different stages of political evolution and their governments have different levels of political legitimacy. But two approaches would help manage the politics of this multinational as well as multiconceptual world so that states can continue to work together towards addressing shared challenges.

The first is to adapt international institutions to take better into account this global reawakening of national identities. The era of permissiveness to Western leadership is ending, while the desire within Western countries to carry the burdens of leadership is waning. Investing over the coming years in the legitimacy of major international institutions such as the United Nations, World Trade Organization, and the International Monetary Fund is essential. Without greater legitimacy, these institutions will find they are increasingly ineffective.

One immediate step should be to begin a process of balancing more equitably the voting weights, exceptions and structures that favour the winners of the 20th century. But international institutions are only as effective as their constituent members. Weak or unstable governments can lead to weak or unstable institutions. It is equally important for national governments and other political actors to focus on their own domestic legitimacy.

Unfortunately, politics in the West may be regrouping around a new duopoly of those who are open to globalization and its attendant rules and are willing to abandon the national myths of the past, and those whose belief in national distinctiveness demands greater protection and scope for autarky. A new polarization of this sort will not lead to a new era of political stability. If the myths are purely nostalgic, they will betray their followers. If, on the other hand, the sense of national identity is eroded, it may prove impossible to retain the democratic legitimacy to engage multilaterally.

Political leaderships among the autocracies face an equally complex problem. National myths are essential to legitimize their non‑democratic systems. If autocratic governments can organize to deliver economic and social progress and security to most of their citizens, then the myths can generally play a supportive role, as China and Singapore have demonstrated.

If they fail to deliver, however, as is now the case in Russia, the temptation for governments to fall back on the notion of “the other” and blame external forces can become not only irresistible but entrenched, given the lack of a democratic pathway to an alternative. The result can be overt or covert conflict.

There is a logical response to both these sets of dynamics. National identities cannot and should not be done away with. Rather than tamping them down in favour of a global, supranational identity, governments need to channel them within models of inclusive domestic governance.

In the first place, even with the need for greater international interdependence, nation states should retain and invest in those instruments of fundamental sovereign power that can protect citizens from external threats, whether military, cyber or criminal. People who have confidence in their governments’ capacity to provide for their immediate security are less likely to retreat into an aggressive nationalism.

Conversely, national governments should devolve the maximum amount of political power over social policies, local development and infrastructure to regional authorities, cities or local communities, with corresponding decentralization of some powers of taxation. At a time of technological disruption and rapid economic change, a strong sense of local identity and solidarity can be a more positive force for adaptation than centrally‑driven policies and narratives.

Governments should then pool regulatory decision‑making over global public goods (protecting biodiversity, controlling greenhouse gas emissions or preventing pandemics) as much as possible at the regional and supranational levels, retaining appropriate levels of national supervision.

The risks of letting national mythologies rise again today without inclusive forms of national and international governance are severe. A large autocracy like China may one day face a structural economic slowdown at the same time as a large democracy like the United States undergoes a fundamental restructuring of its domestic politics, while Europe and others turn inwards. At that moment, the temptation for all sides to use nationalism to mobilize their people and political power will be strong, and the results unpredictably dangerous.

This article was originally published in the World Economic Forum special report, Shaping a Multiconceptual World, in January 2020.

"Emergency in China"

Axios PM
Mike Allen

Data: National Health Commission of the People’s Republic of China, the Centers for Disease Control and Prevention and the World Health Organization; Chart: Danielle Alberti/Axios

The coronavirus isn't yet "a global health emergency," but it's definitely an emergency in China.

The big picture: Coronavirus "has raised the specter of a repeat of the SARS epidemic" in 2002–03 that killed roughly 800 worldwide, the N.Y. Times reports.

  • The death toll is now 18, including one man 600 miles from Wuhan.
  • Major cities have canceled large public gatherings for the Lunar New Year holiday, the most important in the country.
  • Multiple cities are on relative lockdown in hopes of containing the virus.

Why it matters: "Only five global public health emergency declarations have been made in the past. The decisions are fraught, with health authorities wary of causing panic, or of suggesting that governments cannot handle outbreaks on their own," per the Times.

  • The World Health Organization declined to make this declaration today, and it will meet again in 10 days to determine whether to declare an emergency.

Between the lines: Some say the lack of a declaration of a Public Health Emergency of International Concern (PHEIC) may lessen international focus and funding needed to address a potential threat, but others worry that such a declaration could limit the travel and trade important to many people's livelihoods, Axios' Eileen Drage O'Reilly reports.

How should governments regulate AI? IBM Policy Lab offers up solutions

We have already seen concerns about opaque AI systems making safety- and life-critical decisions. Read IBM Policy Lab’s new precision regulation framework for AI that aims to ensure accountability, transparency, fairness and security across AI systems.