Corporate Governance of Artificial Intelligence in the Public Interest

Abstract

Corporations play a major role in artificial intelligence (AI) research, development, and deployment, with profound consequences for society. This paper surveys opportunities to improve how corporations govern their AI activities so as to better advance the public interest. The paper focuses on the roles of and opportunities for a wide range of actors inside the corporation—managers, workers, and investors—and outside the corporation—corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, and governments. Whereas prior work on multistakeholder AI governance has proposed dedicated institutions to bring together diverse actors and stakeholders, this paper explores the opportunities they have even in the absence of dedicated multistakeholder institutions. The paper illustrates these opportunities with many cases, including the participation of Google in the U.S. Department of Defense Project Maven; the publication of potentially harmful AI research by OpenAI, with input from the Partnership on AI; and the sale of facial recognition technology to law enforcement by corporations including Amazon, IBM, and Microsoft. These and other cases demonstrate the wide range of mechanisms to advance AI corporate governance in the public interest, especially when diverse actors work together.

1. Introduction

The corporate governance of artificial intelligence (AI) can benefit from input and activity from a range of stakeholders, including those both within and outside of the corporation. Several recent initiatives call for multistakeholder governance institutions that bring diverse stakeholders together to inform AI governance. Examples include activities of the Global Partnership on AI [1], the European Commission’s High-level Expert Group on AI [2], and research by Cath et al. [3]. To date, less attention has been paid to the important opportunities for different stakeholders to contribute to AI corporate governance in their own right—outside the context of dedicated multistakeholder institutions. Those opportunities are the focus of this paper.

The importance of AI corporate governance is clear. Corporations play a major—perhaps the primary—role in AI research, development, and deployment. Corporate-affiliated researchers published over 50% more AI research papers than academics in the United States in 2018 [4]. Corporate applications of AI touch on many important public issues, including social justice, economic vitality, and international security. Looking ahead, some have proposed that AI could displace large portions of the human labor pool, resulting in chronic unemployment for many people as well as massive profits for AI companies [5]. Corporations are also active in the research and development of artificial general intelligence, a technology that some believe could transform the world in ways that are either radically beneficial or catastrophic [6]. How AI is governed within corporations is therefore of profound societal importance.

To the best of our knowledge, this paper is the first survey of the corporate governance of AI. As reviewed below, prior publications have focused on specific aspects of the topic. This paper offers a broad introduction to the topic and resource for a wide range of scholarships and initiatives to improve AI corporate governance.

Although we aim for this paper to be a broad overview of opportunities in AI corporate governance, it does have some areas of focus. One is on select large corporations at the forefront of AI research and development, in particular Alphabet (the parent company of Google), Amazon, Facebook, and Microsoft. These corporations merit attention because they exercise significant influence on both technological developments and emerging regulatory methods. Additionally, within our discussion of government activities, there is a particular focus on the European Union, which has arguably the most mature regulatory landscape for AI to date. Finally, because this paper is focused on the practical mechanics of AI corporate governance, it mostly focuses on machine learning, the dominant AI paradigm today. These areas of focus are important in their own right; they also serve as examples to illustrate more general points about AI corporate governance that are applicable to other companies, political jurisdictions, and AI paradigms.

Three running examples illustrate how different actors can influence AI corporate governance. The first is Google’s involvement in Project Maven, a U.S. Department of Defense project to classify the content of drone video. In 2018, Google management pulled Google out of Project Maven following media coverage and worker protests. The second example is on the open publication of potentially harmful AI research. In 2019, OpenAI announced its new strategy for publishing such research [7], sparking further debate by, among others, Partnership on AI [8]. The third example is on facial recognition for law enforcement. In 2020, a nexus of activity from nonprofits, the public, governments, and corporate management prompted several companies, including Amazon, IBM, and Microsoft, to stop providing facial recognition technology to law enforcement agencies. Although the paper also discusses other examples, these three run throughout the text and highlight the interconnected influence of different actors on AI corporate governance.

The paper is organized as follows. Section 2 presents the definitions of key terms. Section 3 reviews the extant literature. Section 4 assesses opportunities to improve AI corporate governance for a variety of actors: management, workers, investors, corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, and government. Section 5 concludes.

2. Definitions

Broadly speaking, corporate governance refers to the ways in which corporations are managed, operated, regulated, and financed. Important elements of corporate governance include the legal status of corporations in a political jurisdiction, the relationship between investors and executives, information flows within and outside of the corporation, and specific operational decisions made throughout the corporation [9]. Many people within and outside of a corporation can influence how the corporation is governed. For this reason, we take a broad view on the range of actors relevant for the corporate governance of AI.

Our specific focus in this paper is on how AI corporate governance can be improved so as to better advance the public interest. The public interest can be defined in many ways, such as in terms of costs and benefits, or voter preferences, or fundamental rights and duties. Exactly how the public interest is defined can be important for AI corporate governance, as is seen in a variety of controversies over AI applications. This paper does not take sides on the most appropriate conception of the public interest, with one exception, which is to reject the view that corporations’ sole aim should be to maximize shareholder profits [10], and instead argue that corporations have obligations to a wider range of stakeholders. We recognize that this position is not universally held in the field of corporate governance; however, it does reflect support from many business leaders [11]. Broadly, our aim is to clarify the mechanisms through which corporate governance can be improved to better advance the public interest.

The concept of stakeholders is also central to this paper. Stakeholders have been defined as “any group or individual who can affect or is affected by the achievement of the organization’s objectives” [12] (p. 46). Our focus is specifically on those who can affect how a corporation governs AI. Those who are affected by AI but cannot act to affect it, such as members of future generations, are outside the scope of this paper, except insofar as their interests are part of the overall public interest. It is therefore perhaps more precise to say that we focus on actors, i.e., those who can act to affect AI corporate governance. Likewise, our approach also parallels, but ultimately differs from, the phenomenon of multistakeholderism, which refers to governance activities conducted with participation from multiple types of stakeholders such as governments, corporations, academia, and nonprofits [13]. Multistakeholderism commonly manifests via dedicated institutions that invite participation from multiple types of stakeholders. Cath et al. call for new multistakeholder AI governance institutions [3]. Existing examples include PAI and the OECD Network of Experts on AI, which bring together people from government, industry, academia, and civil society to advance the understanding and practice of AI governance. These are important institutions, and they share this paper’s interest in participation from a wide range of actors. This paper diverges from multistakeholderism by focusing on the full range of opportunities available to different actors and not just the opportunities afforded by dedicated multistakeholder institutions. The paper’s approach is perhaps more similar to the concept of stakeholder capitalism, which calls for corporations to be attentive to stakeholder actions and responsive to stakeholder interests [14].

Artificial intelligence has been defined in many ways. One prominent definition states that AI is an artificial agent that can “achieve goals in a wide range of environments” [15]. However, current AI systems only perform well in certain settings, especially simpler environments for which there are ample data [16]. For this paper, it suffices to employ a social definition: AI is what people generally consider to be AI. This is a bit of a moving target: as the technology has progressed, people’s minimum standards for what they consider AI have risen [17]. This paper focuses on the computer techniques currently considered to be AI, which, in practice, are largely machine learning, as well as more advanced forms of AI that may be developed in the future.

For AI corporate governance, it is also helpful to define AI activities in terms of the AI system lifecycle, i.e., the sequence of activities that take an AI system from its initial conception to its final use. Attention to the lifecycle can help identify and clarify opportunities to improve AI corporate governance. Different actors and activities will have varying influence over different phases of the AI system lifecycle within a corporation. This paper uses the AI system lifecycle to more precisely describe the influence of these actors and activities. In general, efforts to improve AI corporate governance must affect at least one phase of the lifecycle—otherwise, there is no effect on any actual AI systems.

This paper uses an AI system lifecycle framework developed by the OECD Expert Group on AI [18]. Figure 1 illustrates the four phases of the framework. Phase 1 concerns research and design of the AI system. Researchers identify a task for their system, choose a style of model, define performance measures, and select relevant data or other input. This phase includes data collection, cleaning, quality (including bias) checks, and documentation. Phase 2 tests the system to assess performance. This includes testing that covers regression (speed slowdowns), the comparison of previous model behavior to new behavior, and performance across many metrics, e.g., accuracy and calibration measures. Phase 3 puts the system into production. This may include launch testing for real-world use cases, checking compliance with relevant regulations, checking compatibility with legacy software, and assigning responsibilities for managing the AI system. Once the system is deployed, this phase also includes evaluating initial user experience. Phase 4 operates and monitors the AI system in deployment, assessing its outputs and impacts based on the designers’ initial intentions and performance metrics as well as ethical considerations. Problems are identified and addressed by reverting back to other phases or eliminating the AI system.

3. Prior Work

This paper sits at the intersection of literatures on corporate governance and AI governance. Corporate governance is a large field of scholarship with a long history. Good introductions are offered by Monks and Minow [19] and Gordon and Ringe [20]. AI governance is a smaller and relatively new field of study. For more work in this area, see, for example, works from the AI Now Institute [21], Data & Society [22], World Economic Forum [23], Future of Humanity Institute [24], as well as Calo [25].

One body of literature on AI corporate governance studies public policy proposals, primarily for new, dedicated governance bodies. Calo [26] calls for a federal body to address robotics policy. A similar proposal has been discussed in Europe by Floridi et al. [27]. The European Commission has recently proposed to establish a European Artificial Intelligence Board [28]. Scherer [29] outlines a proposal for a dedicated government agency and a voluntary certification scheme that incentivizes companies to submit to agency oversight in return for limited legal liability. Wallach and Marchant [30] propose a governance coordinating committee to support soft law governance that keeps pace with new and emerging AI. Erdélyi and Goldsmith [31] call for an international regulatory agency to address international AI challenges; Cihon et al. [32] argue that it is too soon to establish such an international structure and that further debate is first needed. Clark and Hadfield [33] propose a markets-based approach to AI safety regulation.

Some literature has analyzed existing regulations as they pertain to corporate AI. The E.U. General Data Protection Regulation (GDPR) has been of particular interest in this respect. For example, Wachter et al. [34] argue that the GDPR does not afford a “right to explanation” of automated decision-making, whereas Goodman and Flaxman [35] argue that it does. Another body of literature analyzes the European approach to AI regulation. Smuha [36] analyzes the emerging regulatory environment for making AI trustworthy. Thelisson et al. [37] and Stix [38] survey the broader regulatory landscape in the EU. There is also some literature on a number of national regulations. For example, Wagner et al. [39] analyze different corporate strategies for complying with algorithmic transparency requirements imposed by the German Network Enforcement Act.

There is also an extensive body of literature on specific policy instruments and governance approaches. For example, Senden [40] and Marsden [41] disentangle the concepts of soft law, self-regulation, and co-regulation. Kaminski [42] conceptualizes approaches between command and control regulation and self-regulation as “binary governance”, while Pagallo [43] uses the framing of a “middle-out approach”. Zeitlin [44] discusses the current state of transnational regulation within and beyond the E.U.

The legal liability of corporations for harms caused by AI systems has been another major focus. Broadly speaking, liability regimes aim to compensate victims of harms caused by products and, in turn, encourage producers to avoid the harms in the first place. AI liability regimes are generally not written specifically for the corporate sector, but in practice mainly affect commercial products. The exact form of liability regimes can vary substantially across jurisdictions and circumstances. Separate literatures discuss AI and robotics under liability law in different jurisdictions, including the United States [29,45,46,47,48], the E.U. [49,50,51,52,53], and Germany [54,55,56]. Additionally, more theoretical approach considers whether liability regimes could handle extreme catastrophic risks from AI, such as those of potential long-term artificial general intelligence [57].

A variety of other AI corporate governance topics have also been studied. Buolamwini and Gebru [58] assess the efficacy of targeted audits and publicly shaming AI companies to address biases in facial recognition systems. More generally, Baum [59] explores the social psychology of AI developers as a factor in efforts to steer their work in pro-social directions. Belfield [60] details recent employee activism within the AI community and its impact on AI firms and technological development. Askell et al. [61] analyze competitive pressures on AI firms in terms of their societal impacts. Solaiman et al. [62] examine the societal implications of deciding to publicly disclose AI models, focusing on the case of OpenAI. Cihon [63] reviews the role of international technical standards in governing AI research and development. Baum [64,65] analyzes potential corporate efforts to manipulate public debate about AI risks. O’Keefe et al. [66] propose a novel method of corporate social responsibility that sees AI firms contribute to the public benefit. Avin et al. [67] and Ballard and Calo [68] use forecasting and roleplay methods to study potential future behaviors of actors affecting corporate AI.

A large number of articles published in magazines such as the Harvard Business Review and MIT Technology Review offer practical insights for managers on governing both AI development and adoption within firms. Hume and LaPlante [69] analyze how companies can manage biases and risks along the AI building process. Tiell [70] recommends corporations establish an ethics committee. Chamorro-Premuzic et al. [71] offer a step-by-step approach on how companies can build ethical AI for human resources. Fountaine et al. [72] outline how management should build AI-powered organizations. Abbasi et al. [73] analyze how companies can mitigate the risks of automated machine learning. Hao [74,75] urges AI companies to actually implement their ethical guidelines, while also emphasizing how difficult this will be.

Consulting firms have also published reports on AI corporate governance. Burkhardt et al. [76] of McKinsey describes how Chief Executive Officers (CEOs) can guide employees to build and use AI responsibly. Cheatham et al. [77], also of McKinsey, discuss how managers can mitigate AI risks. Ransbotham et al. [78] of the Boston Consulting Group survey more than 2500 corporate executives on AI topics including how companies are managing AI risks. Several major accounting firms have developed governance frameworks to promote ethical AI [79,80,81]. Deloitte [82] reports on AI risk management in the financial industry. Accenture [83] covers corporate AI ethics committees.

4. Actor-Specific Opportunities to Improve AI Corporate Governance

A variety of actors can improve AI corporate governance so as to better advance the public interest. This section considers nine types of actors. Three are internal to the corporation: managers, workers, and investors. Six are external: corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, and governments.

Although presented separately for clarity, these actors interact, overlap, and co-exist in practice. These various types of actors have important interactions. For example, the media can channel worker influence within firms, facilitate public pressure, and precipitate government action. Actors have the potential to overlap, for example, with governments publishing media reports or taking over management of a company. Ultimately, all actors co-exist within political cultures, which may vary by country and over time [84]: although the following sections describe actions that each actor could take to improve AI corporate governance, we do not offer analysis of the feasibility nor the desirability for such actions within their political cultural contexts.

4.1. Management

Management, as the term is used in this paper, includes all personnel with authority and oversight over other personnel, from top C-suite executives to mid- and lower-level managers. Management is an important—perhaps the most important—type of actor in corporate governance. Management establishes policies, implements processes, creates structures, and influences culture, all of which impact AI development.

One way management can advance AI in the public interest is by establishing corporate policies. One type of policy is strategic objectives. For example, management could establish the objectives of pursuing AI development where it is clearly in the public interest and avoiding contentious settings such as law enforcement. Another type of policy is ethics guidelines that specify how the corporation should develop and use AI and related technologies. Recently, management at many companies have established AI ethics guidelines, including Google, IBM, Microsoft, and OpenAI [85]. An ongoing challenge is to translate ethics guidelines into AI practice [86]. The translation process can include more operational policies on the details of how a company should develop and use specific AI techniques. Likewise, a concern is that published principles could create the appearance of AI corporations acting in the public interest without them actually doing so [87].

Management can also enact processes that translate policies into practice. These processes can take many forms. For example, management can establish new review processes for AI or augment existing review processes, such as those conducted by compliance and risk management teams. Additionally, management can encourage or require the use of documentation methods that generate and distribute information needed to ensure compliance with AI principles [88]. Notable examples include Datasheets for Datasets [89], a standardized reporting document for dataset features, and Model Cards [90], an approach to consistently describing an AI model and its intended use case. These processes could be improved if management were to review their efficacy and publicly share best practice.

Management activity on policies and processes is seen, for example, in the caution of OpenAI on publishing potentially harmful AI work. In 2019, OpenAI released their GPT-2 language model in phases out of concern about its potential harmful applications [7,62]. OpenAI created a review process to evaluate the social impacts of earlier releases before determining if and how to release more advanced versions of GPT-2. OpenAI’s discussion of its phased release [7] references the OpenAI charter [91], a policy document that expresses the principle of factoring safety and security concerns into decisions of what work to publish. (Note: the charter was published in 2018, when OpenAI was a nonprofit.) Although authorship of the charter is attributed to “OpenAI”, it is likely that OpenAI management played a central role in drafting and approving the document, which anchors the organization’s “primary fiduciary obligation” [92]. Additionally, the OpenAI GPT-2 team includes a mix of workers and management, including OpenAI co-founder and Chief Scientist Ilya Sutskever; thus, it can be inferred that management was likely involved in the review process. In summary, the GPT-2 release appears to demonstrate how management may translate policies into processes to support AI development and use in the public interest.

Management can also create structures within the company dedicated to advancing AI in the public interest. Such structures can perform oversight, make recommendations, and provide expertise to people throughout the company. They can consist of company staff and often interact with numerous teams across the organization. Prominent examples include the Microsoft advisory committee AI, Ethics, and Effects in Engineering and Research, the compliance-oriented Microsoft Office of Responsible AI, the Google Responsible Innovation Team, the AI Principles working group within the Google Cloud division, and a Facebook team of policy managers that work with product teams on fairness and explainability problems. Alternatively, the groups can consist of external advisors, such as the Google DeepMind Ethics & Society division’s group of external advisors, the Axon AI and Policing Technologies Ethics Board, and the short-lived Advanced Technology External Advisory Council at Google.

Thus far, these dedicated structures have had mixed success. One success came at Axon. Its ethics board advised against the company selling facial recognition to law enforcement; management followed this advice [93]. A failure occurred at Google, which disbanded its Advanced Technology External Advisory Council soon after its launch amid outcry about its membership [94]. Overall, these structures are relatively new and not yet in wide use, and much is still being learned about them. Nonetheless, one early lesson is that AI governance teams ought to be interdisciplinary. Regardless of where such a team may be within the reporting structure, it may be expected to include lawyers, ethicists, data scientists, engineers, program managers, and other diverse occupational perspectives.

Management can also build AI governance functions into pre-existing structures. Important areas for this may be in compliance and risk management. Current compliance and risk management teams may focus less on AI and more on established risks such as computer security [95]. However, as governments increase their policy attention to AI, the need for corresponding activity within corporations will increase. It is likely that the pre-existing corporate structures could build expertise in AI risks over time, as the field of AI corporate governance matures, as standards are published, and as regulations enter into force. Forward-thinking management can advance this process by setting the groundwork, such as by building AI expertise into pre-existing structures.

Finally, management can help cultivate a corporate culture that supports AI development in the public interest. Corporate culture can play a major role in how a company develops and uses AI [59,96]. Employee onboarding and training could include a focus on responsible AI development [97]. Recruiting efforts could select for, or aim to instill, knowledge of responsible AI development methods. Employee performance reviews and metrics could incentivize these methods’ use, from concretely assessing bias in training data at the design phase to more broadly upholding a culture of responsible development. On the latter, OpenAI has tied compensation levels to adherence to its charter [98]. However, it is unclear what additional steps dominant AI firms are now taking to instill their AI principles into corporate culture.

4.2. Workers

We use the term workers to refer specifically to people who work at the company and do not have managerial authority. This includes, in their subordinate relationship to top management, lower- and mid-level managers. Workers include employees and contractors, both of which are common at AI companies. A wide range of workers affect AI, including researchers, engineers, and product developers. Despite being expected to follow directions from management, workers at AI firms have considerable power to shape corporate governance. Workers are often left with significant latitude to determine corporate activity within their areas of focus, and management is often (although certainly not always) influenced by worker suggestions.

Workers can influence AI corporate governance both directly, through their actions affecting AI systems, and indirectly, by influencing management. While management makes many governance decisions, especially high-level decisions for the corporation and its divisions, many other decisions are left to workers, especially on the specifics of AI design and implementation. Worker opportunities for direct influence may be especially robust at earlier stages of the AI system lifecycle and at corporations and divisions whose management offer workers wide latitude for decision-making. Likewise, worker opportunities for indirect influence may be greatest at corporations and divisions whose management is especially receptive to worker input.

Some of the best opportunities for worker direct influence may be for workers in groups conducting fundamental research, such as Facebook AI Research, Google Brain and DeepMind, Microsoft Research, and OpenAI. Workers in these groups may have significant autonomy from management to pursue their work as they see fit. Indeed, these workers may be influenced less by management and more by academic norms, research fashions, reputational concerns, conference requirements, journal expectations, and their own personal values. Likewise, those seeking to influence corporate AI researchers may find good opportunities via the broader field of AI, such as at leading conferences. For example, as of 2020, the NeurIPS conference uses an ethics review process and requires papers to include a social impact statement [99]. These activities can be important for AI corporate governance due to the significant autonomy of corporate AI researchers.

Workers also have significant opportunities to indirectly affect AI corporate governance by influencing management. However, these opportunities can be risky for workers because of managements’ control over—or at least considerable influence on—workers’ employment and advancement within the corporation. In general, activities that involve a higher degree of worker commitment and risk of reprisal by management will tend to have a greater effect on corporate governance. Low-commitment, low-risk activities can be as simple as raising concerns in project meetings over issues of ethical AI development. These activities tend to be low-profile and not well-documented; colleagues at AI companies inform us that these activities are nonetheless common. More ambitious and risky activities tend to be less common but more visible and more well-documented. These activities can include circulating letters critiquing corporate activity and calling for change, whistleblowing, organizing walkouts, forming unions, and more [60].

Likewise, the extent of indirect worker influence is shaped by management’s receptiveness to worker input and on related management decisions regarding corporate policy and culture. In extreme cases, management can fire workers who push back against management AI corporate governance decisions. For example, Google fired its Ethical AI team co-lead, Timnit Gebru, following a disagreement over the publication of a paper critical of the company’s research on large AI language models [100]. Additionally, several Google employees claim to have been fired as retribution for labor organizing, in possible violation of U.S. labor law [101]. Subtler dynamics include such matters as whether workers have dedicated spaces to organize and articulate their views. For example, Google has internal discussion forums for workers, although management recently hired a team to moderate them [102]. Google management also recently eliminated its regular meetings where employees could address executives [103]. In general, workers will have greater indirect influence on AI corporate governance when they can organize and express views to management without fear of retaliation.

The size of the labor pool also affects both direct and indirect worker influence. Some governance goals may benefit from a large labor pool, such as the goal of solving difficult technical problems in orienting AI toward the public interest. Greater availability of worker talent may make these problems easier to solve. On the other hand, a larger labor pool can make it difficult for workers to self-organize and reach consensus. Likewise, a large labor pool relative to demand for their labor reduces indirect worker influence on AI systems via their influence on management [60].

At present, there is a shortage of talent in the computer science and engineering dimensions of AI, giving workers in these areas considerable indirect influence. These workers are hard to hire and to replace upon firing; therefore, management may be more inclined to accept their demands. This influence could fade if the labor market changes due to increased university enrollment in AI courses and the many government calls for training more people in AI [104] (pp. 111–126); [105]. Labor demand could also shrink if the applications of AI plateau, such as due to a failure to overcome limitations of current deep learning algorithms [16] or due to the rejection of AI applications on moral, legal, or social grounds. For now, though, demand for AI is robust and growing, giving AI scientists and engineers substantial power.

One powerful pattern of indirect worker influence starts with whistleblowing and continues with widely signed open letters. Workers with access to information about controversial AI projects leak this information to media outlets. Subsequent media reports spark dialogue and raise awareness. The media reports may also make it easier for other workers to speak publicly on the matter, because the workers would no longer have to shoulder the burden of being the one to make the story public. The open letters then provide a mechanism to channel mass worker concern into specific corporate governance actions to be taken by management. (See also Section 4.1 and Section 4.8 on the roles of management and the media.)

This pattern can be seen in several recent episodes at Google. In 2018, Google’s participation in Project Maven, a U.S. Department of Defense project to use AI to classify the content of drone videos, was anonymously leaked to Gizmodo [106]. The Gizmodo report does not explicitly identify its sources as Google workers, but this is a likely explanation. Subsequently, over 3000 employees signed an open letter opposing Google’s work on Project Maven [107]. Google management later announced it would leave Project Maven and publish principles to guide its future work on defense and intelligence projects [108]. Additionally, in 2018, The Intercept reported Google’s work on Project Dragonfly, a Chinese search engine with built-in censorship [109]. The Intercept report was also based on an anonymous source that appears to be a Google worker. Subsequently, over 1000 employees signed a letter opposing the project [110]. Google management later ended the project [111].

A somewhat similar pattern is observed in a 2018 episode involving sexual harassment at Google. A New York Times investigation of corporate and court documents and interviews of relevant people found that Google had made large payments to senior executives who left the company after being credibly accused of sexual harassment [112]. Soon after, Google workers organized walkouts in which thousands of workers walked out in support of corporate policy changes on harassment and diversity [113]. The organizers referenced the New York Times report but did not specify the extent to which the walkout was motivated by the report. The organizers later wrote that Google management made some but not all of their requested policy changes [114].

These sorts of worker initiatives are not always successful. In 2018, an unspecified number of Microsoft employees published an open letter calling on Microsoft to abandon its bid for the Joint Enterprise Defense Infrastructure contract, a U.S. Department of Defense cloud computing initiative [115]. Microsoft did not abandon its bid, although Microsoft President Brad Smith did respond by articulating Microsoft policy on military contracts [116]. Additionally, in 2018, hundreds of Amazon employees signed a letter demanding the company stop selling facial recognition services to law enforcement [117]. Management did not stop. Again in 2018, approximately 6000 Amazon employees signed a letter calling on the company to stop using AI for oil extraction. The letter was accompanied by a shareholder resolution making the same argument—an example of investor activity (Section 4.3). Again, management did not stop [118].

4.3. Investors

Corporations take investments in a variety of forms, including by selling shares of stock or issuing bonds. Investors are important because AI is often capital-intensive, requiring extensive funding for research, development, and deployment. Shareholders are the investors with the most capacity to influence corporate governance and are therefore the focus of this section. Indeed, a central theme in corporate governance is the principal–agent problem in which the principals (i.e., shareholders) seek to ensure that their agents (i.e., corporate management) act in the principals’ best interests rather than in those of the agents. In contrast, issuers of bonds are generally less influential, in part because markets for bonds are highly competitive—a corporation can readily turn to other issuers instead of following one issuer’s governance requests.

Investors can influence corporations in several ways. First, investors can voice their concerns to corporate management, including at the annual shareholder meetings required of U.S. companies. Investor concerns can, in turn, factor into management decisions. Second, shareholders can vote in shareholder resolutions, which offer guidance that is generally non-binding but often followed [19] (p. 117). Indeed, even resolutions that fail to pass can still succeed at improving corporate governance; evidence for this has been documented in the context of environmental, social, and governance (ESG) issues [119]. Third, shareholders can replace a corporation’s board of directors, which has ultimate responsibility to manage the corporation, determines strategic direction, and appoints the CEO. For example, shareholders could seek to add more diversity to a board, noting that boards with increased diversity are associated with greater support for corporate social responsibility efforts [120]. Fourth, investors can signal disapproval with corporate governance practices by selling off their investments, and, perhaps, investing in a better governed competitor. Fifth, shareholders can file lawsuits against the corporation for failing to meet certain obligations [121]. These lawsuits are often settled in ways that improve corporate governance [19] (p. 117).

In principle, investors can wield extensive power over a corporation via their control over the board of directors. If management does not follow investor concerns or shareholder resolutions, the shareholders can replace the board with people who will. In practice, however, investor power is often limited. Efforts to replace a board of directors are expensive and rare [19] (p. 117). One study found that activist investors launched 205 campaigns in 2019 and won only 76 board seats [122]. This reality gives management substantial latitude in corporate governance. Nonetheless, management often does respond to investor preferences, especially, but not exclusively, when their preferences affect the company’s stock price.

The power of investors is also influenced by the availability of alternative investment options. A diverse market of AI investment opportunities would provide investors with opportunities to shift their assets to companies that further the public interest. Current market prospects are mixed. On one hand, much of the sector is dominated by a few large companies, especially Google (Alphabet), Amazon, Facebook, and Microsoft. On the other hand, there is also a booming AI start-up scene today; one study identified 4403 AI-related companies that received a total of USD 55.7 billion in funding in the year ending July 2019 [4] (p. 91). Companies such as H20 and Fiddler specifically aim to advance explainable AI systems, creating additional opportunities for investors to promote AI in the public interest.

Investor initiatives should be well-informed by the state of affairs in the company. This requires some corporate transparency. For example, the U.S. Securities and Exchange Commission (SEC) requires companies to disclose investor risk factors [123]. In its disclosure, Google (Alphabet) lists AI-related “ethical, technological, legal, regulatory, and other challenges.” Amazon cites uncertainty about the potential government regulation of AI. Facebook mentions that AI creates a risk of inaccuracies in its community metrics. Microsoft lists AI as a risk of “reputational harm or liability”. However, at each company, AI is only a small portion of the overall disclosure, suggesting that the companies see AI as a minor area of risk. Investors could consider the possibility that the companies are not giving AI risks the attention they deserve.

The efficacy of investor initiatives as an approach to improving AI corporate governance depends on the willingness of investors to take the matter on and the investors’ degree of influence within the company. Investors are diverse and have many interests. Investors with an existing interest in ESG may be especially receptive to promoting AI in the public interest. For example, Hermes, an ESG-oriented investment management business, has written on responsible AI [124] and participated in an investor initiative to create a Societal Risk Oversight Committee of the Board at Alphabet [125]. That initiative ultimately failed [126], in part because it lacked the support of Alphabet’s founders. Alphabet is structured such that their founders retain a majority of shareholder voting power even though they do not own a majority of the shares; Facebook is structured similarly [127]. Although Alphabet and Facebook are extreme cases, in general, investor initiatives will tend to be more successful when they are supported by investors who own a larger portion of shareholder voting power. This applies to public corporations that have issued stock. Investors in private corporations may be especially influential, especially for smaller firms, which often have less access to capital markets. Venture capital firms seeking to promote the public interest may be especially successful in improving AI corporate governance among smaller firms.

The limited influence of shareholder resolutions can also be illustrated by the failed attempt to restrict Amazon’s sale of facial recognition technology to the government. In 2019, shareholders expressed their concern that Rekognition, Amazon’s facial recognition service, poses risk to civil and human rights, as well as shareholder value. They requested the Board of Directors to prohibit sales of such technology to government agencies [128] (pp. 18–19). In 2020, another resolution requested an independent study of Rekognition, including information about the extent to which such technology may endanger civil rights and is sold to authoritarian or repressive governments [129] (pp. 25–26). Both resolutions failed. Even though they would have been non-binding, Amazon tried to block the vote. This unusual attempt was ultimately stopped by the SEC [130]. Although these resolutions did not succeed in achieving reform, they demonstrate that shareholder activism has begun to focus on AI risks in particular.

In short, shareholders wield some influence in the corporate governance of AI. This influence is limited by the sheer volume and variety of risks that weigh on them: AI is not a top of mind. The most used activity available to shareholders thus far has been the resolution. Given that shareholder resolutions are difficult to pass and non-binding when passed, it is unclear if such activities will do much to change corporate governance aside from publicizing particular AI-related governance problems. Over time, as companies continue to emerge that seek competitive differentiation through responsible AI development and as shareholders, particularly institutional investors, continue to value ESG criteria and apply them to AI, the role of investors in responsible AI governance may continue to grow.

4.4. Corporate Partners and Competitors

Other corporations exert influence on a corporation developing or using AI in important ways. These other corporations can be direct competitors, themselves developing or deploying AI systems. Alternatively, they can be corporate partners (or, for brevity, “partners”) that have contractual relationships with said company. Partners can be, among other things, suppliers, customers, or insurers. Partners can use their relationship with the AI company to influence it to advance the public interest.

Competing AI corporations can influence each other through direct market competition and in other ways. As classic economic theory explains, competition can result in greater market share for corporations whose products better advance the public interest. There are exceptions, including where there are negative externalities, i.e., harms of market activity that are not captured by market prices, and monopolies, i.e., where a large market share can be used to exclude competition and set relatively high prices. For example, direct competition to develop more powerful machine learning systems can result in better performance for important applications such as healthcare and transportation, but it can also result in more energy consumption and the externalities of global warming via the use of large amounts of computer hardware [131].

Competitors can also influence each other as peers in the overall field of AI [132]. One AI corporation’s initiatives in the public interest can be adopted or adapted by other AI corporations. For example, in 2020, IBM announced that it would no longer sell facial recognition technology to law enforcement agencies [133]. Amazon [134] and then Microsoft [135] did the same shortly after. These announcements came amidst heightened attention to police misconduct sparked by the killing of George Floyd by the Minneapolis Police Department and subsequent widespread Black Lives Matter protests; therefore, it is possible that each company would have changed its behavior on its own without the others doing the same. However, in an interview, Microsoft’s President explicitly recognized IBM’s and Amazon’s steps [135]. To some extent, the companies may have been jockeying for market share in the face of shifting public opinion, but they may also have been motivated by each other’s example to advance the public interest.

Partners’ ability to influence AI companies can depend significantly on their relative market power. The AI sector is led by some of the largest companies in the world. These companies are often in position to dictate the terms of their partner relationships; they have sometimes used this power to ensure that AI is used in the public interest. For example, Google has limited the use of its facial recognition services to a narrow customer base via its Celebrity Recognition API, and has an extended terms of service to regulate how the technology is used [136]. Similarly, Microsoft vets customers for its facial recognition services; before its blanket 2020 policy was implemented, it reviewed and denied a California law enforcement agency’s request to install the technology on body cameras and vehicle cameras [137].

Partners can impact the reputation of AI companies and, in turn, influence actions to protect that reputation. For example, Article One Advisors, a consulting firm, worked with Microsoft to develop its human rights policies through external engagement, and then publicized this work [138]. The publicity likely boosts Microsoft’s reputation and incentivizes Microsoft to follow through on its public commitments. Corporate partners can also harm AI corporations’ reputations. For example, Google attracted widespread criticism when Randstad, a staffing contractor, allegedly collected facial scans of homeless African Americans in order to improve the performance of Google’s Pixel phone facial recognition features [139]. Reputation is important for large, public-facing corporations such as Microsoft and Google, making this a valuable tool for their corporate partners.

Insurers have distinctive opportunities to influence AI companies. When AI is not in the public interest, that can create substantial risks that may require insurance payouts. Insurers therefore have both the vested interest and the contractual means to compel AI companies to act in the public interest. For comparison, insurers have mandated the adoption of cybersecurity governance and risk frameworks [140,141]; they could do the same for AI. Doing so would improve corporate governance in the insured AI companies. Additionally, it could have further benefits by popularizing innovative practices for AI governance and risk management that are adopted by even uninsured companies. However, some AI risks are not readily handled by insurers, such as emerging risks that are difficult to quantify and price and risks that are too extreme, such as risks from long-term artificial general intelligence.

Finally, it should be noted that corporate partners and competitors consist of management, workers, and investors, whose influence parallels that of their counterparts in AI corporations as discussed in Section 4.1, Section 4.2 and Section 4.3. Workers, managers, and investors who seek to improve AI corporate governance may find additional opportunities in corporate partners and competitors. As an illustrative example, in 2020, FedEx investors pushed FedEx to call for the Washington Redskins American football team to change their name, given its racist connotations. FedEx is a major partner of the team. The initiative was successful: the team name will be changed [142]. This example is not from the AI industry, but it nonetheless speaks to the capacity for actors in corporate partners and competitors to affect positive change.

4.5. Industry Consortia

Industry consortia, as the term is used here, are entities in which multiple corporations come together for collective efforts related to AI governance. We define industry consortia to include entities that include more than just corporations. For example, PAI membership includes corporations, nonprofits, media outlets, and governmental bodies [143]. PAI is perhaps most precisely described as a multistakeholder organization, but it is also an industry consortium. The same holds true for other entities, such as the IEEE Standards Association, whose members include corporations and individuals [144].

Industry consortia can be instrumental in identifying and promoting best practices for AI in the public interest. AI corporations face many of the same challenges and issues. They likewise benefit from best practices being developed for the whole sector and then distributed to each corporation, instead of each corporation “reinventing the wheel”. Industry consortia are well-positioned to serve as the entity that develops best practices for the whole sector. They can query member corporations on what has worked well—or poorly—for them, pooling their collective experience together. They can also conduct in-house research on best practices, with researchers hired by the pooled funds of their member corporations. It may not be worthwhile for every AI corporation to hire their own in-house experts on various facets of AI in the public interest, but it may be worthwhile for the sector as the whole to do so. Industry consortia enable that to happen.

An illustration of these dynamics is seen in PAI’s recent work on best practices on the publishing of potentially harmful AI research. PAI’s work on this was prompted by the work of one of its members. Specifically, OpenAI released its language model GPT-2 in phases out of concern about its potentially harmful uses [62]. Soon after, PAI hosted discussions with OpenAI and other organizations about best practices in publishing potentially harmful research [145], launched a project to develop guidance [8], and advised Salesforce on the release of their language model CTRL [146,147]. PAI’s status as an industry consortium has enabled it to advance publishing practices across organizations.

As best practices are formulated, industry consortia can take the additional step of formalizing them as standards. For example, the Consumer Technology Association (CTA) convened over 50 companies, some but not all of which were CTA members, to develop a standard for the use of AI in healthcare [148]. The IEEE Standards Association is also active on AI standards, as is the International Organization for Standardization [63], although the latter is not an industry consortium. By formalizing best practices into standards, industry consortia can enable corporations across the sector to improve their practices.

Best practices and standards developed by industry consortia can go on to play a legal or quasi-legal role. Governments sometimes enact policies requiring corporations to adhere to certain broad principles of conduct without specifying the details of what particular conduct does or does not meet these principles [149]. The best practices and standards formulated by industry consortia can fill in the details of good conduct. Additionally, regulatory agencies and courts handling liability cases sometimes treat compliance with industry best practices and standards as satisfactory, such that corporations meeting these practices or standards avoid regulatory fines or court judgments of liability to which they would otherwise be subject [150] (p. 17). This can dilute the public benefits of government action, but it also incentives corporations to meet or exceed these standards and practices, potentially bringing net gains for the public interest.

The above are examples of soft law, which can be defined as obligations that, although not legally binding themselves, are created with the expectation that they will be given some indirect legal effect through related binding obligations under either international or domestic law [151]. Soft law has been advocated for AI corporate governance due to its flexibility and ease of adoption [152,153]. In general, it is difficult for governments to create detailed and rigorous laws for complex issues such as those pertaining to AI. The dynamism of emerging technologies such as AI is especially challenging for the development and enactment of “hard” laws. Industry consortia are often better positioned than governments to master the details of the technology and its changes over time, due to the availability of expertise among consortium members. Furthermore, any more binding “hard law” measures enacted by governments are likely to draw on the particulars of pre-existing soft law instruments. These are additional reasons for industry consortia to pursue robust best practices and standards for AI corporate governance in the public interest.

Industry consortium activities do not necessarily advance the public interest. For example, they can pool the resources of member corporations to lobby governments for public policies and conduct information and public relations campaigns that advance industry interests at the public’s expense. In other sectors, such lobbying has often been a major impediment to good public policy [154]. In the coming years, industry consortia could possibly present similar challenges to the public interest.

4.6. Nonprofit Organizations

Nonprofit organizations play several important roles in advancing AI corporate governance in the public interest, including research, advocacy, organizing coalitions, and education. Nonprofits can be advocacy organizations, labor unions, think tanks, political campaigns, professional societies, universities, and more. The distinction between these types of organizations is often blurry, with one organization playing multiple roles.

To date, research has been a primary focus of nonprofit organizations working on AI corporate governance. Research contributions from nonprofit universities and think tanks are too numerous to compile here; many are in Section 3. What follows are some select examples of nonprofit research aimed at influencing corporate governance. Note that all universities mentioned in this section are nonprofit. Upturn, a nonprofit dedicated to advancing equity and justice in technology, worked with researchers at Northeastern University and University of Southern California to produce evidence of previously suspected illegal discrimination in housing advertisements served by Facebook [155]. Ranking Digital Rights (e.g., [156]) reports technology companies’ human rights records and encourages companies to improve their performance. The Electronic Frontier Foundation reports companies’ cooperation with government demands for censorship and also encourages them to improve their performance [157]. The AI Now Institute at New York University publishes reports to provide AI developers with suggestions to reduce bias and increase the public accountability of AI systems [158]. Finally, this paper is another work of nonprofit research on AI corporate governance.

Nonprofit advocacy efforts can often draw on such research. For example, a 2018 advocacy campaign by the nonprofit American Civil Liberties Union (ACLU) opposed Amazon selling facial recognition software to governments [159]. The campaign was supported by research on biases in the software by the ACLU [160] and prior research from a pair of researchers at Microsoft and the Massachusetts Institute of Technology [58]. The ACLU later reported that their campaign was unsuccessful [161]. However, in 2020, following anti-police brutality protests, Amazon stopped selling facial recognition software to law enforcement agencies, as discussed in Section 4.4 and Section 4.7. The ACLU campaign and the research on which it drew may have laid the groundwork for Amazon’s action.

Finally, nonprofits have conducted some work to build the field of AI in directions beneficial to public interest. Black in AI and AI4All are nonprofit organizations that promote diversity within the field of AI. Increased diversity could, in turn, help reduce bias in the design, interpretation, and implementation of AI systems. Additionally, the Future of Life Institute has hosted conferences on beneficial AI and built coalitions in support of open letters calling for AI in the public interest. These field-building initiatives are not specifically focused on corporate governance, but they have included people from AI corporations and are likely to have at least some effect on AI corporate governance.

One challenge facing nonprofit organizations is funding. This is a challenge for nonprofits working on all cause areas, and AI corporate governance is no exception. Firstly, nonprofits may struggle to raise the funds they need to advance their missions. Secondly, some AI nonprofits may turn to AI companies for funding, creating potential conflicts of interest [162,163]. Thirdly, where companies disagree with the nonprofits’ aims, the companies can use their wealth to push back. Although such a dynamic has perhaps not been seen much to date in AI, it has been observed in other sectors, such as the tobacco industry pushing back on the link between cigarettes and cancer and the fossil fuel industry pushing back on the risks of global warming [64]. The extreme wealth of some AI corporations makes the potential for conflict of interest and the imbalance in resources particularly acute. Where these issues arise, nonprofits may fail to advance the public interest.

With that in mind, it can be helpful to distinguish between adversarial and cooperative nonprofit activity. Adversarial activity pushes AI companies in ways that the companies do not want. Cooperative activity proceeds in ways that the companies are broadly supportive of. Cooperative activity may tend to have a more limited scope, limited by the bounds of what companies are willing to support. On the other hand, adversarial activity may struggle to effect change if the companies do not agree with the proposed changes, and in some cases could backfire by galvanizing opposition. Whether adversarial or cooperative approaches are warranted should be assessed on a case-by-case basis.

4.7. The Public

The public, as the term is used here, refers to people acting in ways that are non-exclusive in the sense that broad populations can participate. In the context of AI corporate governance, the primary roles of the public are (1) as users of AI technology, including when the users pay for it and when it is free to them, subsidized by advertising, and (2) citizens who can vote and speak out on matters of public policy. Although not discussed in this section, members of the public can also impact AI corporate governance indirectly by exerting influence on other members of the public.

Users of AI technology can improve AI corporate governance by choosing goods and services that are in the public interest. In other (non-AI) industries, customers are often—although not always—willing to pay more for branded ethical standards [164]. AI users may also be willing to pay more; or, when they are using the technology for free, they could accept a product that has higher ethical standards but is inferior in other respects. This effect can even determine which technologies become dominant, rejecting certain uses and prioritizing others—regardless of how they are marketed [165]. One example in this direction is the gradual shift in social media platform popularity from public platforms such as MySpace, Facebook, and Twitter toward private messaging platforms, such as Snapchat and WhatsApp [166].

Citizens can improve AI corporate governance by supporting good (and opposing bad) AI public policies. They can do this by speaking up, such as in anti-racism and police brutality protests that have influenced AI facial recognition practices, and by voting for politicians who will enact good policies. Public opinion has played an important role in regulatory responses to other advanced technologies such as genetically modified organisms [167,168]. Growing public concern about digital technology, dubbed “techlash”, has prompted calls for antitrust and other policy response. Thus far, AI has not been extensively regulated, although further shifts in public opinion could help to change this.

Public opinion has played an important role with regard to the sale of facial recognition software to law enforcement. As discussed in Section 4.4 and Section 4.6, following 2020 protests against racism and police brutality, several AI companies moved away from providing facial recognition tools for law enforcement. It is worth noting that the protests garnered broad public support [169]. Therefore, the AI corporations’ responses show how public protests and changes in public opinion can advance AI corporate governance in the public interest.

Finally, members of the public can voice their views about AI, prompting changes. For example, the initial launch of Google Glass in 2014 was widely criticized for violations of privacy [170]; it was discontinued and subsequently relaunched for industrial and professional users instead of for the general public [171]. Google Photos, an AI photo classification system, sparked public outcry for labeling a person with dark skin as a gorilla, prompting Google Photos to remove gorilla and other non-human primate terms from its lexicon [172]. In general, public pressure will tend to be more pronounced for highly visible brands [173,174]; the same is likely to also apply for AI companies.

The public faces several challenges to supporting the corporate governance of AI in the public benefit. One is the complexity of AI issues, which makes it hard for people lacking specialized training to know what the issues are or what stances to take. This can be mitigated by efforts for public education, such as Elements of AI [175], an online course which aims at educating 1% of European citizens in the basics of AI. Despite such initiatives, public education remains a difficult challenge due to the complexity of the issues and the competition for public attention. Likewise, public education can take time, in which case it may be most valuable in the medium term. (On medium-term AI issues, see Ref. [176])

Furthermore, some AI issues are important but arcane and not conducive to media coverage or other means of capturing public attention. This holds in particular for low-visibility AI companies, including those that do not market to the public but instead sell their AI to governments or other companies.

In some cases, AI technology users may be relatively disinclined to opt for the more ethical option due to the difficulty of switching from one AI product to another. Impediments can include (1) significant learning curves, as is common for software in general, (2) transition costs, such as the need to re-upload one’s photos and other information to a new site, or the need to inform one’s contacts of their new email address, and (3) network effects, in which a product’s value to one user depends on its use by other users, as in the distinctive communities of people on specific social media platforms. Someone concerned about AI at one company may not have good alternatives, dissuading them from choosing a product more aligned with the public interest. Additionally, AI is often only part of consumer-facing products. Concern about AI may be outweighed by other concerns. For example, a user may appreciate the cultural and educational value of a media sharing site (such as YouTube or Instagram) even if they dislike its recommendation algorithm.

Finally, public action may tend to be primarily oriented toward how AI systems are deployed. Earlier phases of the AI system lifecycle have fewer direct ties to the public and are therefore less likely to garner public attention. For these phases of the AI lifecycle, other types of action may tend to be more important.

4.8. The Media

The media, as the term is used in this paper, refers to both professional and amateur journalists together with their diverse means of distribution, from traditional newspapers to online social media platforms. The media can play an important role in improving AI corporate governance by researching, documenting, analyzing, and drawing attention to good practices, problems, and areas for improvement. The media serves as an important link between actors internal and external to the corporation, and it plays a vital role in distilling and explaining complex technology and business detail in terms that can be understood and used by outside audiences including the public and policymakers. Indeed, media reports have been essential for the insights contained in this paper.

AI corporate governance is often in the news. Several newspapers have dedicated technology sections including The New York Times, Wall Street Journal, and Financial Times. Dedicated technology media sources include The Verge, Wired, and the MIT Technology Review. The Markup is specifically focused on the societal impacts of digital technology companies. All of these outlets devote extensive attention to AI corporate governance.

Media coverage has been instrumental in highlighting problems in AI corporate governance and mobilizing pressure for change. For example, the media has published several reports based on worker whistleblowing, presenting issues at AI companies that otherwise may have stayed internal to the companies (see Section 4.2). In the case of Google’s participation in Project Maven, Gizmodo reported about the project based on leaked information [106]. The media also covered the subsequent protests by Google workers, further amplifying their concerns [107,177,178]. Google management later announced it would leave the project [108]. This example demonstrates how broad media coverage combined with employee activism can have significant influence on corporate decision-making.

Other reporting focuses on adverse societal impacts of AI. One prominent example is a ProPublica investigative report on biased outcomes of the COMPAS algorithm for assessing the risk of a person committing a future crime [179]. The report has been credited with focusing researchers’ attention on fairness in machine learning [180] (p. 29). A more recent example is a New York Times exposé on the little-known facial recognition company Clearview [181], which was scraping personal photos uploaded online to serve as training data. The Times report prompted technology companies to take legal action [182] and nonprofit advocacy organizations to lobby the U.S. government to intervene [183].

Media coverage appears to be most robust for later phases of the AI system lifecycle. Later phases, such as the deployment of AI products, tend to be more publicly visible and of more direct interest to the public and other outside stakeholders. In contrast, earlier phases, such as basic research and development, are less visible and less directly relevant to stakeholders, and therefore may tend to receive less coverage. Coverage of internal corporate activities may be impossible without whistleblowers; these activities can occur across the lifecycle but may be especially frequent during earlier phases.

Other factors can also shape trends in the media coverage of corporate AI. Coverage may tend to be greater where AI intersects with other topics of great public interest, such as racism or unemployment, or for prominent AI companies or celebrity entrepreneurs. Certain events can also draw new attention to existing practices, such as the longstanding privacy flaws of the Zoom videoconferencing platform gaining newfound attention due to the heavy use of Zoom during the COVID-19 pandemic [184]. Risky AI practices may tend to receive the greatest attention in the immediate aftermath of an incident of that type of risk, unless such incidents are too commonplace to be considered newsworthy. This can result in less coverage of emerging and extreme AI risks for which incidents have not previously occurred. (The tendency to overlook extreme risks has been dubbed “the tragedy of the uncommons” [185])

Finally, as with nonprofits, the media faces financial challenges. The business models of many media outlets have been harmed by the rise of the internet and other recent factors. This has resulted in less investigative journalism, meaning fewer resources to report on issues in AI corporate governance. Meanwhile, the AI industry is amidst a financial boom, making it difficult for the media to hire people with expertise in AI. There can even be conflicts of interest, as has been a concern since Amazon founder Jeff Bezos purchased the Washington Post (a concern that Bezos and the Post deny [186]). The media clearly has an important role in advancing AI corporate governance in the public interest, making it vital that its various challenges be overcome.

4.9. Government

Government, as the term is used here, refers to institutions with legal authority over some geographic jurisdiction, whether national (e.g., the United States), subnational (e.g., California), or supranational (e.g., the E.U.). Governments can influence AI corporate governance to promote the public interest by using various policy instruments. In the following, we consider four widely accepted categories of policy instruments: command and control regulation, market-based instruments, soft law, and information and education. We also consider procurement, which is important for the government use of AI. Our focus on these categories of instruments reflects current practices to influence the corporate governance of AI; however, more methods could be used in the future, including the role of state-owned enterprises and direct subsidies.

Command and control regulation uses binding legal rules to specify the required behavior, and enforcement measures to correct or halt non-compliant behavior [187] (p. 107). Many existing regulations are applicable to AI, although they do not address AI specifically. For example, in the E.U., AI systems must comply with existing data protection, consumer protection, and anti-discrimination laws [188] (p. 13). The GDPR contains detailed rules governing the processing of personal data using automated decision-making [189]. In this case, the person whose data are being processed can request “meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing,” although whether this entails a right to explanation is disputed [34,35]. Although these rules are not explicitly about AI (automated decision-making as defined in the GDPR is not synonymous with AI), they are nonetheless applicable to many AI systems [190]. Similarly, labor law is generally not written explicitly for work in AI, but it nonetheless affects how workers and management are allowed to act, as seen for example in allegations of former Google workers that they were fired because of their labor organizing, which may have been in violation of U.S. labor law (Section 4.2) [101].

In recent years, governments around the world have started to work on AI-specific command and control regulations. Proposals have been published, among others, by the E.U. [28,188,191], China [192], and the United States [193,194]. For example, in April 2021, the European Commission published a proposal for an Artificial Intelligence Act [28], following its White Paper on AI [188] and the High-Level Expert Group on AI’s Ethics Guidelines for Trustworthy AI [191]. The new proposal follows a risk-based approach with different requirements for different levels of risk. It prohibits practices which pose unacceptable risks (e.g., social scoring by governments or systems that exploit vulnerabilities of children) and contains specific rules for high-risk systems (e.g., biometric identification systems). These rules include requirements regarding the quality of datasets used; technical documentation and record keeping; transparency and the provision of information to users; human oversight; and robustness, accuracy and cybersecurity. The proposed regulation contains very light and mostly voluntary provisions for AI systems with low or minimal risk. The vast majority of AI systems currently used in the E.U. fall into this category. The requirements for high-risk systems are command and control because they require specific behavior that will be enforced by supervisory authorities. It is worth noting that the proposal still has to be adopted by the European Parliament and the member states.

Market-based instruments affect corporate activity through economic incentives [195] (p. 22); [187] (p. 117). For example, taxes could be used to encourage safe behavior or discourage unsafe behavior, such as via tax discounts for AI systems that have been certified or audited by a third party. Civil liability could incentivize AI companies to mitigate risks from accidents or the misuse of AI systems. Subsidies could support corporate initiatives on AI in the public interest, for example, to support research and development on AI techniques that improve safety. These benefits may also support the public good insofar as they address a market failure or incentivize innovation with future public benefit. The DARPA Grand Challenge for Autonomous Vehicles is one such example, which helped catalyze private investment in the field in the early 2000s [196].

Procurement refers to government purchases of goods and services [197,198], including AI systems. For example, law enforcement agencies have recently sought to purchase facial recognition software. As discussed throughout this paper, this procurement is controversial, with many arguing that it is not in the public interest. This controversy speaks to the fact that governments face important decisions on which AI systems to procure and how to use them to best advance the public interest. Governments can additionally use procurement to influence AI corporate governance by procuring systems that meet high standards of safety and ethics. This incentivizes industry to adopt and maintain such standards. Procurement is thus, in a sense, a demand-side market-based instrument in its potential to use market incentives to advance AI corporate governance in the public interest.

As discussed in Section 4.5, soft law is the non-binding expectation of behavior that has some indirect legal basis. One specific form of soft law in which governments play a central role is co-regulation. It is worth noting that co-regulation is not necessarily a form of soft law, but the concepts are at least interconnected [40,41]. Co-regulation expands on corporate self-regulation to include some government involvement, typically to ensure enforcement [195] (p. 35). For example, the U.S. Federal Trade Commission (FTC) encourages firms to declare privacy policies, and prosecutes firms who deviate from their statements [199]. Conceivably, the FTC could also punish violations of companies’ self-stated AI principles [30]. However, such enforcement has been ineffective. In 2011, Facebook agreed to a settlement with the FTC after being accused of violating its privacy policy [200], but the violations continued, most notably with the 2018 Cambridge Analytica scandal [201]. In 2019, Facebook again settled with the FTC, this time for an unprecedented USD 5 billion and stringent monitoring requirements [202]. However, even the USD 5 billion fine could be seen as simply the cost of business, given that Facebook’s 2019 profit was nearly USD 18.5 billion [203], and especially if the costs can be amortized across multiple years.

Governments can also lead public information and education campaigns [187] (p. 116). A better educated public could incentivize AI companies to improve their governance, as detailed in Section 4.7. Education campaigns could also foster constructive public debate on AI ethics and safety [191] (p. 23). Education takes time, and thus is unlikely to be effective in time-critical situations, but it is otherwise found to often be a cost-effective policy option [195]. Governments can lead AI education campaigns. They can also obtain information about AI corporations, such as by establishing information disclosure requirements as discussed above.

The efficacy of policy instruments can depend on their enforcement. This applies to command and control regulation as well as certain soft law and market-based instruments. Where enforcement is applicable, it is used to ensure compliance. Noncompliance is commonly sanctioned by fines. In extreme cases, corporate activities can be shut down and noncompliant corporate personnel can be imprisoned. A lesser punishment could see companies added to published lists of noncompliant companies, as is the practice in European financial market regulation [204]. However, in practice, governments do not always vigorously enforce compliance. Monitoring and enforcement can be expensive, and governments may not always have the resources or motivation to do so. Weak enforcement can limit the influence of government rules and regulations on AI corporate governance. Additionally, if the people responsible for complying with AI regulations disagree with or resent them—and are sufficiently empowered to act on this disagreement—then it could prompt backlash, such that the people decline to comply and may even do more of the disallowed or discouraged behavior than would occur without the regulation [59].

When selecting a policy instrument to improve the corporate governance of AI, governments need to consider a number of factors. One of these factors is the underlying regulatory approach. Most AI-specific proposals follow a risk-based approach. This approach ensures that regulation does not exceed what is necessary to achieve the underlying policy objective, as is required by the principle of proportionality in E.U. law. Apart from that, governments need to decide between regulation focused on specific technologies, such as AI, or regulation that addresses general issues that can be applicable to multiple technologies, such as privacy. Finally, governments need to decide whether the regulation should be cross-sector or for specific sectors, such as healthcare or transportation.

AI-specific policy instruments face a particular challenge in defining their scope of application [29] (pp. 359–362). There is no generally accepted definition of the term AI, and existing definitions do not meet the specific requirements for legal definitions [205]. A possible solution to this problem would be to avoid the term AI and define other properties of the system, such as certain use cases or technical approaches. This idea may be a worthy focus of future AI policy research and activity.

Policy shapes innovation [206] (p. 249), and this will be no different with AI. Regulation can impose costs on or the outright prohibition of certain types of AI research and applications, thereby limiting innovation in particular areas while making others more attractive. Meanwhile, market mechanisms and procurement may subsidize or otherwise incentivize some types of AI research and development over others. For example, law enforcement procurement of facial recognition may already be stimulating innovation in that branch of AI. Taken as a whole, then, government regulation may be expected to shape AI innovation. Poorly constructed regulation may shape, or particularly limit, innovation in ways that undermine the public interest; this is a bug to be remedied, not a feature. For example, poorly constructed regulation may place a disproportionate burden of AI-related regulatory compliance that falls on smaller companies with fewer resources.

When a government uses policy instruments, the effects are not always limited to that government’s jurisdiction, a phenomenon known as regulatory diffusion. Corporations that work across jurisdictions often follow the regulations of the most stringent jurisdiction across all jurisdictions to gain the efficiencies of a single standardized compliance operation. They may even lobby other jurisdictions to adopt similar rules so as to similarly bind local competitors. Influential jurisdictions in this regard include California and the European Union, sometimes referred to as the “California Effect” [207] and the “Brussels Effect” [208]. Given that leading AI corporations are multinational, policy instruments from a wide range of jurisdictions could shape corporate governance in (their conception of) the public interest globally. Even if companies are not incentivized to comply globally, other jurisdictions may pass similar regulation, imitating the first mover. Regulations do not always diffuse, and corporations may shift operations to jurisdictions with relatively lax regulations. Nonetheless, regulatory diffusion can increase the impacts of policy innovation.

Regulation on law enforcement use of facial recognition technology has demonstrated such regulatory imitation. Municipalities seem to have taken the lead in the United States. Following research on AI and gender and race biases [58,209], some municipalities have started to ban the use of facial recognition technology for law enforcement purposes, including San Francisco [210] and Boston [211]. Even though municipal action has set the agenda for wider action, leading to multiple bills that have been introduced in the U.S. Congress on this topic [212,213], there is not yet a regulation of facial recognition technology at the federal level. Due to the absence of federal regulation, the legal treatment of the technology is currently highly variable across the United States. In keeping with their incentives for regulatory consistency across jurisdictions, Microsoft has repeatedly called for the federal regulation of facial recognition technology [135,214,215]. Under the proposed E.U. AI regulation, all remote biometric identification of persons will be considered high-risk and subject to third party conformity assessment [28]. Certain applications for the purpose of law enforcement will be prohibited in principle with a few narrow exceptions.

Governments may not simply wait for AI policy instruments to passively diffuse; they may support institutionalized international coordination. For example, the OECD AI Principles have been adopted by 45 nations and informed similar principles agreed to by the G-20 countries [216]. The OECD AI Policy Observatory is now developing implementation guidance for the principles, aimed at both government and corporate decision-makers. International coordination is further supported by several UN initiatives [217]. Pre-existing international agreements and initiatives can shape AI corporate governance. Prominent examples include the UN Guiding Principles for Business and Human Rights and the UN Global Compact, which offer guidance for business practices to promote the public interest. Already, Google has adapted some AI systems according to this UN guidance [218]. Additionally, the OECD has published general corporate governance guidance that has been adopted into national regulation [219,220].

5. Conclusions

A wide range of actors can help improve the corporate governance of AI so as to better advance the public interest. It is not, as one might think, a matter for the exclusive attention of a narrow mix of insider corporate elites. To be sure, the opportunities may often be better for some types of actors than others. However, significant opportunities can be found for many actors both within and outside of AI corporations. Importantly, these opportunities are available even in the absence of dedicated multistakeholder institutions that are designed to invite contributions from a more diverse group. Multistakeholder institutions have an important role to play, but they are only one of many means through which diverse stakeholders can improve AI corporate governance.

Often, progress depends on coordination and collaboration across different types of actors, as illustrated in the three primary cases used throughout the paper. First, the example of Google’s Project Maven shows that workers and the media can collaborate to be particularly successful in influencing management. Second, the example of law enforcement use of facial recognition technology demonstrates that novel research, activism by nonprofits, and broad media coverage can build on each other to achieve change in corporate governance. Third, the example of the publication of potentially harmful research shows management, workers, and industry consortia interacting to establish, implement, and share best practices for AI in the public interest. People in each type of actor category would do well to understand not just their own opportunities, but also the broader ecosystem of actors and their interactions. Questions of how best to pursue coordination and collaboration across actors must be resolved on a case-by-case basis, in consideration of the particulars of the issues at play and the relative roles and capabilities of different actors.

Opportunities to improve AI corporate governance are likely to change over time. For example, workers’ influence may diminish over time if the expected increase in the supply of skilled AI workers outpaces demand increases for AI systems. Changes in the economic, cultural, and political significance of AI can alter the opportunities available to many types of actors, such as by shaping the political viability of government regulations. Changes in underlying AI technologies can also be impactful. For example, if there are major breakthroughs toward AI systems that can substitute for most forms of human labor or even approach an artificial general intelligence, then the corporations could end up with substantially greater economic and political clout. This could increase the importance of actions from within the companies, especially from management and investors. On the other hand, such breakthroughs could also increase public and policymaker interest in AI in ways that facilitate more extensive government activity over time. The delay in government responses to emerging technologies, often called the “pacing problem” [221], creates a clear, even if interim, role for other actors in improving AI corporate governance in the public interest as AI research and development continues.

This paper has presented a broad survey of opportunities to improve AI corporate governance across a range of actors. It is, to the best of our knowledge, the first such survey published. As a first pass through a large and complex topic, the paper’s survey has been largely qualitative, and it has also not been comprehensive in its scope. We have sought to map out the overall terrain of AI corporate governance without necessarily identifying or measuring all the hills and valleys. We have focused on select larger corporations, with less attention to smaller ones. We have focused on the United States and Europe, with less attention to other parts of the world. Additionally, to at least some extent, we have covered a convenience sample of cases. The result is a broad but somewhat limited map of the AI corporate governance landscape.

One important area for future research is on evaluating the quality of opportunities to improve AI corporate governance. Specific actors may benefit from guidance on how best to focus their activities. Some actors, such as researchers and philanthropists, have opportunities to bolster the efforts of other types of actors and would benefit from guidance on which other actors are most worth supporting. (These supporting actions fall outside the scope of this paper’s framework and would make for a further area for future research.) To a large extent, the quality of opportunities facing specific actors must be assessed on a case-by-case basis accounting for context and technological particulars, and therefore fall outside the scope of broad surveys such as this paper. An important activity would be to bridge the gap between the more general insights of this survey and the specific insights needed for corporate governance decision-making in the public interest for particular categories of AI policy problems and types of AI systems. Such work should include more specific conceptions of the public interest, because different conceptions can underlie different evaluative standards and generate different practical guidance.

Finally, further research could investigate how AI corporate governance may change over time, especially as companies develop increasingly capable AI systems. This paper has emphasized near-term cases in order to give its study of AI corporate governance a better empirical basis and more immediate practical value. Nonetheless, the potential for extremely large consequences from long-term AI make it a worthy subject of attention. Important questions include how near-term actions could affect long-term AI corporate governance, such as through path dependence in governance regimes, and how future actors can best position themselves to influence long-term corporate AI for the better. One good starting point may be to look more closely at the earliest phases of the AI lifecycle, especially basic research and development, on the grounds that this may be where future advanced forms of AI first appear within corporations.

As the deployment of AI systems and research and development of AI technologies continue, the role of AI corporate governance is expected to only increase over time. With it too increases the importance of experimentation and iteration to develop actors’ strategies to improve the corporate governance of AI companies in the public interest. This paper has surveyed the landscape with the aim of empowering practitioners and catalyzing necessary further research. It is only with this continued work, by the full range of actors considered here, that AI corporations may be expected to support the public interest today, tomorrow, and into the future.

Author Contributions

Conceptualization, P.C., J.S. and S.D.B.; research, P.C., J.S. and S.D.B.; analysis, P.C., J.S. and S.D.B.; writing, P.C., J.S and S.D.B.; funding acquisition, S.D.B. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Gordon R. Irlam Charitable Foundation.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

For helpful input on this research, we thank participants in a seminar hosted by the Global Catastrophic Risk Institute. For input on an initial draft, we are grateful to Ramiro de Avila Peres, Rosie Campbell, Alexis Carlier, Sam Clarke, Jessica Cussins-Newman, Moritz Kleinaltenkamp, Yolanda Lannquist, Joel Lehman, Jeremy Nixon, Dakota Norris, and Cullen O’Keefe. We thank Oliver Couttolenc for research assistance and feedback and McKenna Fitzgerald for assistance in manuscript formatting. Any remaining errors are the authors’ alone. Views expressed here are those of the authors and do not necessarily represent the views of their employers.

Conflicts of Interest

The authors declare no conflict of interest. The paper was researched and drafted prior to Peter Cihon joining his current employer, GitHub, a subsidiary of Microsoft.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figure

Figure 1. AI system lifecycle.