Algorithmic Literacy and the Role for Libraries

Abstract

Artificial intelligence (AI) is powerful, complex, ubiquitous, often opaque, sometimes invisible, and increasingly consequential in our everyday lives. Navigating the effects of AI as well as utilizing it in a responsible way requires a level of awareness, understanding, and skill that is not provided by current digital literacy or information literacy regimes. Algorithmic literacy addresses these gaps. In arguing for a role for libraries in algorithmic literacy, the authors provide a working definition, a pressing need, a pedagogical strategy, and two specific contributions that are unique to libraries.

Headnote

ABSTRACT

Artificial intelligence (AI) is powerful, complex, ubiquitous, often opaque, sometimes invisible, and increasingly consequential in our everyday lives. Navigating the effects of AI as well as utilizing it in a responsible way requires a level of awareness, understanding, and skill that is not provided by current digital literacy or information literacy regimes. Algorithmic literacy addresses these gaps. In arguing for a role for libraries in algorithmic literacy, the authors provide a working definition, a pressing need, a pedagogical strategy, and two specific contributions that are unique to libraries.

INTRODUCTION

Algorithms, in one form or another, are as old as human problem solving and as simple as "a sequence of computational steps that transform the input into the output."1 For centuries they have been effective, and uncontroversial, methodologies. However, the rise of artificial intelligence (the integration of big data, enhanced computation, and advanced algorithms) with its human and greater-than-human performance in many areas has positioned algorithms as transformational and a "major human rights issue in the twenty-first century."2

Algorithmic literacy is important given of the prevalence of algorithmic decision-making in many aspects of everyday life and because "the danger is not so much in delegating cognitive tasks, but in distancing ourselves from-or in not knowing about-the nature and precise mechanisms of that delegation."3 As a result, David Lankes warns of a new type of digital divide with "a class of people who can use algorithms and a class used by algorithms."4 In a 2019 Deloitte survey "only 4 percent reported they were confident explaining what AI is and how it works."5 While a 2019 Edelman survey indicated general awareness of AI, it also revealed a similar lack of knowledge about the details of AI.6 An informed, algorithmically literate public is better able to negotiate and employ the complexities of AI.7

Identifying and acting upon algorithms as a literacy makes them as "fundamental as reading, writing, and arithmetic."8 However, the uncritical use of the term literacy should make one suspicious of extending it to algorithms. Increasingly "literacy" has come to mean merely a body of knowledge or a set of domain-specific skills.9 Various literacies have been described, such as health, death, financial, physical, ocean, religious, visual, dancing, spatial, screen, and porn. This includes a dozen different technology-related literacies.10 The case for algorithmic literacy, and the role for libraries in advancing it, must rest on a clear definition, a recognized problem and need, a pedagogical strategy, and a unique (or at least supportive) contribution libraries can provide.

ALGORITHMS AND LITERACY

While the term "algorithmic literacy" is recent, it has antecedents that cover similar if not equivalent ground. The general terms computer literacy or digital literacy have spawned more specific terms such as cyber literacy, computational thinking, and algorithmic thinking.11 Most of these arise from the field of computer science, where algorithms are central, and focus on the computational nature of algorithms as a "matter of mathematical proof" where "other knowledge about algorithms-such as their applications, effects, and circulation-is strictly out of frame."12 The implications of algorithms in everyday life suggests that a deeper and broader interpretation is required.

Whether a literacy, a mode of thinking, or merely a set of skills, discussions about computation and algorithms have been plagued by "ambiguity and vagueness" and "definitional confusion" resulting in ongoing challenges in establishing core pedagogy in both K-12 and higher education.13 Without a clear, acknowledged, and actionable definition that differentiates it from concepts such as digital literacy, computational thinking, and algorithmic thinking, algorithmic literacy will be relegated to a buzz phrase and the urgency of its recognition and application will be lost. The relationship between algorithms and artificial intelligence might recommend the adoption of "AI thinking" or "AI literacy" as the more appropriate term.14 However, algorithmic literacy is both more foundational than the broader concept of AI and more actionable than just thinking.

Algorithms are not a technology like AI or, more generally, computers. Algorithms provide a structure that frames-and constrains-how we express ourselves. They are a way of seeing and acting in the world and "need to be understood as relational, contingent, [and] contextual."15 While the technical and operational aspects of algorithms are important to understand and use (as they are for the technologies and processes of reading and writing in a new language), they are complemented by a broader awareness:

Literacy is not a set of generic skills or something we do or do not possess, it's a sociocultural practice, it's something that we do, and what we do with literacy depends on the social, cultural, and historical contexts in which we do it. Literacy looks different in different contexts and communities. Literacy is not neutral, it's ideological. There are dominant and marginalized literacies.16

This perspective is the essence of critical algorithm studies, where algorithms are viewed as sociotechnical systems that are "intrinsically cultural . . . constituted not only by rational procedures, but by institutions, people, intersecting contexts, and the rough-and-ready sensemaking that obtains in ordinary cultural life."17 Algorithms as part of increasingly ubiquitous AI, such as machine learning and deep learning systems, reflect and promulgate certain ideologies and have impacts and influences in the full range of human society. Cautions about algorithmic decision-making have identified the far-reaching implications for bias, fairness, privacy, and democratic processes.18 At the same time, numerous national strategies to support AI development have highlighted the substantial economic impact, anticipated to be $15.7 trillion (US) by 2030.19 The idea of algorithmic literacy must encompass multiple perspectives and contexts.

"LITERACIES OF THE DIGITAL"

Computer, internet, information, computation, and algorithmic are all "literacies of the digital."20 While each of these has its own domain and focus, they share common ideas and are generally symbiotic with each other. There is an especially strong and complementary connection between computational literacy and information literacy.21

Computational thinking and algorithmic literacy are closely related even if most definitions of the former fail to fully acknowledge the broader social, economic, and political implications. However, the extensive literature on computational thinking is useful in helping to articulate aspects of algorithmic literacy. Wing's foundational article about computational thinking describes the key characteristics in terms that closely resemble a literacy:

1. conceptualizing, not programming

2. fundamental, not a rote skill

3. a way that humans, not computers, think

4. complements and combines mathematical and engineering thinking

5. ideas, not artifacts

6. for everyone, everywhere.22

Jacob and Warschauer make a strong case for computational thinking as a literacy. Their three-part framework identifies computational thinking as a new literacy embedded in modern sociocultural practices (computational thinking as literacy), discusses how literacy development can be leveraged to foster computational thinking (computational thinking through literacy), and explores ways in which computational thinking can facilitate literacy development (literacy through computational thinking).23

This analysis of computational thinking informs the larger context and broader implications of algorithmic literacy.

DEFINING ALGORITHMIC LITERACY

Scribner and Cole define a literacy as "socially organized practices [that] make use of a symbol system and a technology for producing and disseminating it."24 Therefore, literacy = practices + symbol system + technology. To this definition, Steiner adds a more aspirational and humanistic definition:

By "literacy" I mean the ability to engage with, to respond to, what is most challenging and creative in our societies. To experience and contribute to the energies of informed debate. To distinguish the "news that stays news," as Ezra Pound put it, from the tidal waves of ephemeral rubbish, superstition, irrationalism, and commercial exploitation.25

Literacy is about knowing and meaning making through the processes of internalizing and externalizing information. Literacy enables a reflective, critical, and integrative approach to information that utilizes a broad knowledge base for both understanding and communicating ideas. Finn calls for an algorithmic literacy "that builds from a basic understanding of computational systems, their potential and their limitations, to offer us intellectual tools for interpreting the algorithms shaping and producing knowledge" and thereby provides "a way to contend with both the inherent complexity of computation and the ambiguity that ensues when that complexity intersects with human culture."26 Referring more broadly to "AI literacy," Long and Magerko provide an operational view defining it as "a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace."27

Following an exhaustive analysis of different, and often contradictory, definitions of literacy, information literacy, and digital literacy, Bawden suggests "explaining, rather than defining, terms."28 This provisional description of algorithmic literacy acknowledges that advice.

Algorithmic literacy is the skill, expertise, and awareness to

* Understand and reason about algorithms and their processes

* Recognize and interpret their use in systems (whether embedded or overt)

* Create and apply algorithmic techniques and tools to problems in a variety of domains

* Assess the influence and effect of algorithms in social, cultural, economic, and political contexts

* Position the individual as a co-constituent in algorithmic decision-making.

This description recognizes two overarching concepts: "creativity and critical analysis."29 Creativity involves building, creating, and using algorithms for specific purposes. Critical analysis involves recognizing the application of algorithms in decision-making and the implications of their use in a variety of settings and within certain contexts.

WHY ALGORITHMIC LITERACY?

The need for algorithmic literacy arises from two key and equally important perspectives, both of which essentially focus on power: control and empowerment. Algorithms, especially those using machine learning and deep learning, are complex, opaque, invisible, shielded by intellectual property protection, and most importantly, consequential in the everyday lives of people.30 Control is held by those who build and deploy algorithms, not those who use them.

In part because of these characteristics, people hold significant misconceptions about algorithms, their use, and their effect. In a 2019 global survey of consumers, 72% said they understood what AI was. However, despite AI being used in a wide variety of consumer-facing applications (e.g., email, search, social media), 64% said they had never used AI.31 A study of Facebook users found that 62% were unaware that the News Feed is algorithmically constructed and, even when told this, 12% concluded that it is, as a result, completely random.32

Bias, discrimination, and unfairness in AI have been well documented.33 It is clear that poor data combined with underspecified algorithms and uncritical interpretations of the AI model outcomes can lead to abuses in a variety of ways. There is no quick fix, no automated solution to these problems. Accordingly, those creating algorithms and those using them must be able to question the source of training data, the strengths and weaknesses of learning algorithms, the metrics for success, and how (and for whom) the systems are being optimized. The overarching objectives are accountability and transparency.

Perhaps most critically, the prevalence of algorithms in our lives has changed the way we interact with and use those systems, and the ways we behave in personal and social contexts. We conduct ourselves to be "algorithmically recognizable" allowing us to become "increasingly legible to machines for capture and calculation."34 The danger is that this will "lead users to internalize their [algorithm's] norms and priorities."35

At the same time the power of algorithmic technology is abused and misused, it remains a powerful technology to enhance human capabilities and insight. Algorithms are attributable to dramatic advances in health care and science as well as more mundane (but appreciated) applications such as spam filters. Anti-science sentiments, typified by anti-vaxxers, should not be allowed to undermine the opportunities for algorithms that materially improve the human condition and the natural world.

Those opportunities now extend beyond the well-funded, technology-rich research and corporate AI departments. Increasingly more consumer-friendly tools and applications allow a broader and more diverse population to create algorithmic solutions. The rise of MLaaS (machine learning as a service) brings together powerful cloud-based machine learning environments with accessible toolsets.36

Algorithmic literacy is needed to acknowledge both the technology's power (control) over people and power (empowerment) for people. Recognizing the need for protection and encouragement, many governments have enacted protective legislation and training initiatives. Emblematic of the former is the General Data Protection Regulation (GDPR) of the European Union with its "right to explanation" for algorithmic decisions.37 Exemplary of the latter is Finland's initiative to educate a large portion of their population through "Elements of AI," a free online course.38

Despite these advances there remain power imbalances that require vigilance on the part of 21st century digital citizens. Understanding the power and politics of algorithms recognizes their ontological impact in "new ways of ordering the world."39 Effects this profound suggest a deeper and more comprehensive understanding of algorithms is needed:

Efforts to help people understand algorithms need to continue moving away from a focus on building awareness of algorithms-people increasingly know about "those things called algorithms"-and toward explaining algorithms in such a way that people have a more consistent conceptualization of what algorithms are, what algorithms do, and-what often is overlooked-what algorithms cannot do.40

Algorithmic literacy, like all literacies, is not about mastery but levels of competence appropriate to age, circumstance, and need. Understood simply as recipes or visual decision trees, algorithms are accessible to even those with minimal digital literacy. Public institutions, and specifically libraries, can and must take a lead role in addressing the challenges of this "new world."

THE LIBRARY ROLE IN ALGORITHMIC LITERACY

Libraries have traditionally played a central role in making emerging technologies accessible to their communities whether those be online systems, makerspaces, interactive media, virtual reality, or a host of others. Advancing digital access, digital literacy, and digital inclusion have long been the acknowledged by governments and public agencies as a role of the public library even if not appropriately funded to do so.41 Recently, libraries have begun addressing their role in relation to AI and algorithmic literacy.

The Urban Libraries Council (ULC) conducted an informal poll about AI and public libraries.42 Of the responding libraries (83 of its 150-member library systems), 45% identified AI as important to their leadership with 23% having a staff person dedicated to AI and 27% providing programming to help the public learn about AI. In response to a question of how best libraries could serve their community in this area, 79% said by framing and building awareness of AI, 68% recommended providing continuous education opportunities for the public, and 61% supported the provision of experiential programming. In 2019, the ULC formed a working group to advance the public library role in AI awareness, education, and experiences.

In 2018 the Canadian Federation of Library Associations (CFLA) held a national forum in part focused on artificial intelligence.43 Participant discussions yielded three key priorities with respect to AI: training for library staff, educational materials of for the public, and advocacy initiatives regarding privacy, bias, and transparency. A fourth priority was the inclusion of AI literacy and awareness in MIS and MLIS curricula to facilitate a leadership role for the profession in this area.

Algorithmic literacy programs have two general audiences: members of the community the library serves and the staff of the libraries themselves. For the community, these programs center on awareness and implications, skill development, and application and use.44 Through workshops, hands-on laboratories and makerspaces, consumer checklists, and a variety of informational tools, libraries can provide, or partner in providing, resources in an age- and context-appropriate setting. For library staff, an additional focus is required on advocacy with respect to regulatory issues, system development, and the evolution of the local and national information infrastructures. Library staff can lead, and participate in, advocacy programs that seek to influence government, public agencies, commercial system and service providers, and others about algorithmic literacy.

It is a misconception to think of algorithms, and AI more generally, as arcane topics beyond the ability of library staff to understand and teach. While the technical details of AI are complex, this is not the level of understanding required of staff or needed by the library's community. For example, AI programing at the Frisco Public Library introduced AI Maker Kits and ran Basic AI classes. The Toronto Public Library, through its Digital Innovation Hubs, has offered learning circles in basic AI (using the Finnish Elements of AI course as a foundation) and hosted presentations on various aspects of algorithms in everyday life. By abstracting algorithms to higher level concepts related directly to daily experience (using Facebook is illustrative of many key ideas regarding algorithmic literacy), staff can obtain a sufficient overview from a variety of accessible, introductory texts or videos. Perhaps most importantly, given the new and evolving nature of this technology, library staff should view themselves as co-learners.

No matter the setting or context, an active learning approach is recommended with learners situated as makers as well as consumers.45 A review of the K-12 curricula regarding computational literacy identified active learning strategies based on projects, problem solving, cooperation, and games. The researchers recommend augmenting these with scaffolding strategies, storytelling, and aesthetic approaches.46 While intended for algorithmic literacy initiatives involving children, four design principles from Dasgupta and Mako are relevant for any demographic:

1. make data analysis central and ensure the data is relevant to the learner,

2. manage risk by using sandboxes for experimentation,

3. respect community values about technology that may differ, and

4. support authenticity with real-world examples and scenarios.47

Long and Magerko document a set of 17 core competencies and 15 associated learning design considerations regarding AI literacy.48 Taken together these represent the basis for an algorithmic literacy program for any demographic and any context.

Libraries are encouraged to seek partnerships and collaborations with schools (K-12 and higher education) as well as with non-profit advocacy and training groups.49 Examples among these include The Algorithmic Literacy Project (algorithmliteracy.org) and A.I. For Anyone (https://aiforanyone.org). Many technology companies also offer high quality programs and resources. However, a report from the Public Policy Forum notes that digital literacy campaigns are "too often funded by the very companies that are contributing to the problem."50

A key issue is the lack of assessment instruments. There are none for algorithmic literacy and few for computational thinking. The most prominent of the latter is skills based, focusing on concepts and operational practices and very little on the wider social and cultural implications.51 Library experience with information literacy assessment can inform algorithmic literacy assessment by helping to balance skills and operational concerns with a wider focus on concepts and contextual awareness.

INFORMATION LITERACY AND EXPLAINABLE AI (XAI): UNIQUE LIBRARY CONTRIBUTIONS

While libraries can make contributions to algorithmic literacy through a variety of programs, resources, and advocacy initiatives, two specific areas suggest opportunities for unique contributions: algorithmic literacy as a part of information literacy and algorithmic literacy in support of "explainable AI" (XAI).

Algorithmic Literacy and Information Literacy

Annemaree Lloyd describes the opacity and ubiquity of algorithms as "a wicked problem for librarians and archivists who have a vested interest in equitable access, informed citizenry and the maintenance of public memory" and insists that information literacy "provides resistance to the expansionist claims of algorithms, while at the same time ensuring that people harness the power of this culture to their advantage."52 Information literacy programs championed by libraries have been instrumental in raising awareness and skill building among their user communities. Using information literacy programs as a scaffold, algorithmic literacy can be incorporated into these successful initiatives. However, given the current needs "machine learning and algorithms present frontstage in the information literacy constellation."53

Head et al., in their important 2020 study of algorithms and information literacy, present a view of student perspectives that is both troubling and optimistic.54 The students expressed "a tangle of resignation and indignation" about the effects of algorithms on their lives. For them, algorithms obscure more than they reveal, privacy is compromised, "trust is dead," and skepticism is total. The authors conclude that we face an "epistemological crisis" where algorithms are "stripping individuals of the responsibility to interpret the facticity of the information these systems give us when that interpretation has been performed by the algorithms themselves." However, students also employed "defensive practices" against algorithms, utilized "multiple selves" to preserve their privacy, and were keen to learn how to "fight back" against surveillance and algorithmic decision-making. This is a reminder that "while algorithms certainly do things to people, people also do things to algorithms."55 People have "algorithmic capital" which they can use in "negotiation with algorithmic power."56

With these findings, it seems clear that status quo information literacy programs will not address the unique challenges presented by algorithms. Jason Clark, Scott Young, and Lisa Janicke Hinchliffe took up this challenge with a project funded by an IMLS grant.57 Calling "algorithmic awareness" a "new competency," these researchers identified a gap in the ACRL Framework for Information Literacy that revealed "a lack of an understanding around the rules that govern our software and shape our digital experiences."58 Those rules are the "invisible logic" of algorithms that need to be made transparent for users and library staff. Deliverables from this project include an integrated curriculum, syllabus, and software prototype that respond uniquely to the pedagogical challenges of algorithmic literacy.59

In promoting ML (machine learning) literacy, Ryan Cordell also calls for a specific pedagogical approach that would "emphasize the situated-ness of ML training data and experiments, including the biases or oversights that influence the outcomes of academic, economic, and governmental ML processes."60 Recommendations from this report provide guidelines for developing staff expertise, running pilot projects, and creating toolsets and checklists supportive of responsible machine learning.

Algorithmic Literacy and Explainable AI (XAI)

Perhaps a less obvious way for libraries to contribute to algorithmic literacy is through explainable AI (XAI).61 Difficulties in interrogating algorithms to assess bias, discrimination, and unfairness (as well as other deficiencies such as veracity and generalizability) have led to widespread interest in XAI. The purpose of XAI is to "enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners" and to deploy AI systems that have "the ability to explain their rationale, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future."62 There is complementarity between the objectives of XAI and algorithmic literacy. Both seek transparency, promote understanding, and facilitate accountability. Both recognize the primacy of human agency in human-machine interaction.

XAI is accomplished through a variety of techniques, strategies, and processes. These can involve unambiguous proofs, technical and statistical interventions for verification and validation, and authorizations that rely on standards, audits, and policy directives.63 Explanations are contextual. System designers, professionals, regulators, end users, and the general public need explanations specific to their objectives and tailored to their skills and knowledge.

As algorithmic decision-making is increasingly embedded in the information tools, services, and resources provided by libraries and promoted to users, XAI and algorithmic literacy can operate in close association. Libraries can incorporate aspects of XAI into algorithmic literacy programming and the principles of algorithmic literacy (and more generally information literacy) can inform how XAI is sensitive and responsive to different explanatory needs.

XAI is still an emergent field but it has had, and will continue to have, a profound impact on the development of machine learning systems. The opportunity for library involvement is immediate: librarians need to become well versed in these technologies, and participate in their development, not simply dismiss them or hamper them. We must not only demonstrate flaws where they exist but be ready to offer up solutions. Solutions grounded in our values and in the communities we serve.64

A repeated message from LIS researchers is that library-developed tools to interrogate AI systems are essential components in advancing algorithmic literacy.65 These tools can address the complexity and opacity of machine learning systems and provide levels of explainability and transparency in contextually appropriate ways. One such tool, either as a stand-alone system or embedded in an existing discovery system, might provide a user with access to the nature, and potential bias, of the training data, the general efficacy of the learning algorithm(s) used, and the generalizability of the trained model to different contexts. This XAI scorecard would integrate the objectives of XAI, algorithmic literacy, and information literacy. By leveraging and developing library staff skills and by partnering with AI research and industry groups "libraries can become ideal sites for cultivating responsible and responsive ML."66 Padilla views this engagement as not just a technical initiative but a library-wide effort to promulgate "responsible operations" with AI, noting that library practices "that embed transparency and explainability increase the likelihood of organizational accountability."67

CONCLUSION

Algorithms are "the new power brokers in society" and "we are growing increasingly dependent on computational spectacles to see the world."68 Lash argues that this development has altered the rules by which society operates. Constitutive rules (e.g., rules that define the boundaries of society) and regulative rules (e.g., the rules define how we operate in society) are now joined by "algorithmic, generative rules." These rules are "compressed and hidden and we do not encounter them in the way that we encounter constitutive and regulative rules. Yet this third type of generative rules is more and more pervasive in our social and cultural life of the post-hegemonic order."69

Algorithmic literacy is a means to understand this new set of rules and to encourage the skills and abilities so people can use algorithms and not be used by them. Libraries have typically championed accessible technology and its effective use. The ubiquity of algorithmic decision-making and its profound impact on everyday lives makes the recognition and promotion of algorithmic literacy a critical new challenge and imperative for libraries of all types.

Sidebar

Michael Ridley ([email protected]) is Librarian Emeritus, McLaughlin Library, University of Guelph, Ontario, Canada. Danica Pawlick-Potts ([email protected]) is PhD Candidate, Faculty of Information and Media Studies, Western University, Ontario, Canada. 2021.

ENDNOTES

1 Thomas H. Cormen et al., Introduction to Algorithms, 3rd ed. (Cambridge MA: MIT Press, 2009), 13.

2 Yoav Shohman et al., "AI Index 2017 Report" (Stanford, CA: Human-Centered AI Initiative, Stanford University, 2017), http://cdn.aiindex.org/2017-report.pdf; Safiya Noble, Algorithms of Oppression: How Search Engines Reinforce Racism (New York: New York University Press, 2018), 1.

3 Jos de Mul and Bibi van den Berg, "Remote Control: Human Autonomy in the Age of Computer-Mediated Agency," in Law, Human Agency, and Autonomic Computing, ed. Mireille Hildebrandt and Antoinette Rouvroy (Abingdon: Routledge, 2011), 58.

4 Lee Rainie and Janna Anderson, "Code-Dependent: Pros and Cons of the Algorithmic Age" (Pew Research Center, February 2017), http://www.pewinternet.org/wp-content/uploads/sites/9/2017/02/PI_2017.02.08_Algorithms_FINAL.pdf.

5 "Canada's AI Imperative: From Predictions to Prosperity" (Toronto: Deloitte, 2019), 16, https://www.canada175.ca/en/reports/ai-imperative?&id=ca:2el:3or:awa_2019_FCC_Omnia1:from_dca_FCCOmnia2.

6 "2019 Edelman AI Survey," Edelman, 2019, https://www.edelman.com/sites/g/files/aatuss191/files/2019-03/2019 Edelman AI Survey Whitepaper.pdf.

7 Jenna Burrell, "How the Machine 'Thinks': Understanding Opacity in Machine Learning Algorithms," Big Data & Society 3, no. 1 (2016), https://doi.org/10.1177/2053951715622512; Rainie and Anderson, "Code-Dependent."

8 Jeannette Wing, "Computational Thinking, 10 Years Later," Communications of the ACM 59, no. 7 (2016): 10, https://doi.org/10.1145/2933410.

9 Loanne Snavely and Natasha Cooper, "The Information Literacy Debate," Journal of Academic Librarianship 23, no. 1 (1997): 9-14, https://doi.org/10.1016/S0099-1333(97)90066-5.

10 Alfred Thomas Bauer and Ebrahim Mohseni Ahooei, "Rearticulating Internet Literacy," Cyberspace Studies 2, no. 1 (2018): 29-53, https://doi.org/10.22059/jcss.2018.245833.1012.

11 Evelyn Stiller and Cathie LeBlanc, "From Computer Literacy to Cyber-Literacy," Journal of Computing Sciences in Colleges 21, no. 6 (2006): 4-13; Peter J. Denning and Matti Tedre, Computational Thinking (Cambridge MA: MIT Press, 2019); Z. Katai, "The Challenge of Promoting Algorithmic Thinking of Both Sciences- and Humanities-Oriented Learners," Journal of Computer Assisted Learning 31, no. 4 (2015): 287-99, https://doi.org/10.1111/jcal.12070.

12 Nick Seaver, "What Should an Anthropology of Algorithms Do?" (American Anthropological Association, Chicago, 2013), 1-2, http://nickseaver.net/papers/seaverAAA2013.pdf.

13 Jesús Moreno-León and Marcos Román-González, "On Computational Thinking as a Universal Skill," in IEEE Global Engineering Education Conference (EDUCON, Santa Cruz de Tenerife, Spain: IEEE, 2018), 1684-89; Shuchi Grover and Roy Pea, "Computational Thinking in K-12: A Review of the State of the Field," Educational Researcher 42, no. 1 (2013): 38-43, https://doi.org/10.3102/0013189X12463051; Betual C. Czerkawski and Eugene W. Lyman III, "Exploring Issues about Computational Thinking in Higher Education," TechTrends 59, no. 2 (2015): 57-65.

14 Daniel Zeng, "From Computational Thinking to AI Thinking," IEEE Intelligent Systems (November/December, 2013), 2-4; Duri Long and Brian Magerko, "What Is AI Literacy? Competencies and Design Considerations," in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20 (Honolulu, HI: Association for Computing Machinery, 2020), 1-16, https://doi.org/10.1145/3313831.3376727.

15 Rob Kitchin, "Thinking Critically about and Researching Algorithms," Information, Communication & Society 20, no. 1 (2017): 18, https://doi.org/10.1080/1369118X.2016.1154087.

16 Karen Nicholson, "Information into Action? Reflections on (Critical) Practice" (Workshop on Instruction in Library Use (WILU), University of Ottawa, 2018), 7-8, https://ir.lib.uwo.ca/fimspres/51/.

17 Nick Seaver, "Algorithms as Culture: Some Tactics for the Ethnography of Algorithm Systems," Big Data & Society 4 (2017): 10, https://doi.org/10.1177/2053951717738104.

18 Virginia Eubanks, Automating Inequity: How High-Tech Tools Profile, Police, and Punish the Poor (New York: St. Martin's Press, 2018); Noble, Algorithms of Oppression; Cathy O'Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (New York: Crown, 2016); Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (Cambridge, MA: Harvard University Press, 2015).

19 Time Dutton, "Building an AI World: Report on National and Regional AI Strategies" (Toronto: CIFAR, 2018), https://www.cifar.ca/docs/default-source/ai-society/buildinganaiworld_eng.pdf?sfvrsn=fb18d129_4; PriceWaterhouseCooper, "Sizing the Prize: What's the Real Value of AI for Your Business and How Can You Capitalise?," 2017, https://www.pwc.com/gx/en/issues/analytics/assets/pwc-ai-analysis-sizing-the-prize-report.pdf.

20 Allan Martin and Jan Grudziecki, "DigEuLit: Concepts and Tools for Digital Literacy Development," Innovation in Teaching and Learning in Information and Computer Sciences 5, no. 4 (2006): 249-67, https://doi.org/10.11120/ital.2006.05040249.

21 Rosanne Cordell, "Information Literacy and Digital Literacy: Competing or Complementary?," Communications in Information Literacy 7, no. 2 (2013): 177-83, https://doi.org/10.15760/comminfolit.2013.7.2.150; Andreas Dengel and Ute Heuer, "A Curriculum of Computational Thinking as a Central Idea of Information & Media Literacy," in Proceedings of the 13th Workshop in Primary and Secondary Computing Education (WiPSCE'18) October 4-6, 2018, Potsdam, Germany (New York: ACM, 2018), https://doi.org/10.1145/3265757.3265777; Sarah Gretter and Aman Yadav, "Computational Thinking and Media & Information Literacy: An Integrated Approach to Teaching Twenty-First Century Skills," TechTrends 60 (2016): 510-16, https://doi.org/10.1007/s11528-016-0098-4.

22 Jeannette Wing, "Computational Thinking," Communications of the ACM 49, no. 3 (2006): 35.

23 Sharin Rawhiya Jacob and Mark Warschauer, "Computational Thinking and Literacy," Journal of Computer Science Integration 1, no. 1 (2018): 3, https://doi.org/10.26716/jcsi.2018.01.1.1.

24 Sylvia Scribner and Michael Cole, The Psychology of Literacy, ACLS Humanities E-Book (Series) (Cambridge, MA: Harvard University Press, 1981), 99.

25 George Steiner, "School Terms: Redefining Literacy for the Digital Age," Lapham's Quarterly 1, no. 4 (2008): 198.

26 Ed Finn, "Algorithm of the Enlightenment," Issues in Science and Technology 33, no. 3 (2017): 25; Ed Finn, What Algorithms Want: Imagination in the Age of Computing (Cambridge, MA: MIT Press, 2017), 2.

27 Long and Magerko, "What Is AI Literacy?," 2.

28 David Bawden, "Information and Digital Literacies: A Review of Concepts," Journal of Documentation 57, no. 2 (2001): 233.

29 Gretter and Yadav, "Computational Thinking," 510.

30 Pasquale, The Black Box Society; O'Neil, Weapons of Math Destruction.

31 "What Consumers Really Think about AI: A Global Study," Pega, 2019, https://www.ciosummits.com/what-consumers-really-think-about-ai.pdf.

32 Motahhare Eslami et al., "First I 'Like' It, Then I Hide It: Folk Theories of Social Feeds," in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16 (San Jose, CA: Association for Computing Machinery, 2016), 2371-82, https://doi.org/10.1145/2858036.2858494.

33 Julia Angwin et al., "Machine Bias," ProPublica, May 23, 2016, https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing; Eubanks, Automating Inequity; Noble, Algorithms of Oppression; Pasquale, The Black Box Society; Ruha Benjamin, Race after Technology: Abolitionist Tools for the New Jim Code (Polity Press, 2019); O'Neil, Weapons of Math Destruction.

34 Tarleton Gillespie, "The Relevance of Algorithms," in Media Technologies: Essays on Communication, Materiality, and Society, ed. Tarleton Gillespie, Pablo J. Boczkowski, and Kirsten A. Foot (Cambridge, MA: MIT Press, 2014), 184; Sun-Ha Hong, Technologies of Speculation: The Limits of Knowledge in a Data-Driven Society (New York: New York University Press, 2020), 2.

35 Gillespie, "The Relevance of Algorithms," 187.

36 Altexsoft, "Comparing Machine Learning as a Service: Amazon, Microsoft Azure, Google Cloud AI, IBM Watson," Data Science (blog), September 27, 2019, https://www.altexsoft.com/blog/datascience/comparing-machine-learning-as-a-service-amazon-microsoft-azure-google-cloud-ai-ibm-watson/.

37 European Union, "Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016," 2016, http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679; Bryce Goodman and Seth Flaxman, "European Union Regulations on Algorithmic Decision Making and a 'Right to Explanation,'" AI Magazine 38, no. 3 (2017): 50-57, https://doi.org/10.1609/aimag.v38i3.2741.

38 Finland, "Work in the Age of Artificial Intelligence: Four Perspectives on Economy, Employment, Skills and Ethics" (Helsinki: Ministry of Economic Affairs and Employment, 2018), http://urn.fi/URN:ISBN:978-952-327-313-9.

39 Taina Bucher, If . . . Then: Algorithmic Power and Politics (New York: Oxford University Press, 2018), 20.

40 Alison J. Head, Barbara Fister, and Margy MacMillan, "Information Literacy in the Age of Algorithms: Student Experiences with News and Information, and the Need for Change" (Project Information Literacy, 2020), 41, https://www.projectinfolit.org/uploads/2/7/5/4/27541717/algoreport.pdf.

41 Paul T. Jaeger et al., "The Intersection of Public Policy and Public Access: Digital Divides, Digital Literacy, Digital Inclusion, and Public Libraries," Public Library Quarterly 31, no. 1 (2012): 1, https://doi.org/10.1080/01616846.2012.654728.

42 "ULC Snapshot: Artificial Intelligence," Urban Libraries Council Weekly Newsletter, July 18, 2018.

43 Canadian Federation of Library Associations, "Artificial Intelligence and Intellectual Freedom: Key Policy Concerns for Canadian Libraries" (Ottawa: CFLA, 2018), http://cfla-fcab.ca/wp-content/uploads/2018/07/CFLA-FCAB-2018-National-Forum-Paper-final.pdf.

44 Martin and Grudziecki, "DigEuLit."

45 B. Alexander, S. Adams Becker, and M. Cummins, "Digital Literacy: An NMC Horizon Project Strategic Brief" (Austin, TX: The New Media Consortium, 2016), https://www.nmc.org/publication/digital-literacy-an-nmc-horizon-project-strategic-brief/.

46 Ting-Chia Hsu, Shao-Chen Chang, and Yu-Ting Hung, "How to Learn and How to Teach Computational Thinking: Suggestions Based on a Review of the Literature," Computers & Education 126 (2018): 296-310, https://doi.org/10.1016/j.compedu.2018.07.004.

47 Sayamindu Dasgupta and Benjamin Mako Hill, "Designing for Critical Algorithmic Literacies," ArXiv:2008.01719 [Cs], 2020, http://arxiv.org/abs/2008.01719.

48 Long and Magerko, "What Is AI Literacy?"

49 Alexander, Adams Becker, and Cummins, "Digital Literacy."

50 Edward Greenspon and Taylor Owen, "Democracy Divided: Countering Disinformation and Hate in the Digital Public Sphere" (Ottawa: Public Policy Forum, 2018), 19, https://ppforum.ca/wp-content/uploads/2018/08/DemocracyDivided-PPF-AUG2018-EN.pdf.

51 Marcos Román-González, Juan-Carlos Pérez-González, and Carmen Jiménez-Fernández, "Which Cognitive Abilities Underlie Computational Thinking? Criterion Validity of the Computational Thinking Test," Computers in Human Behavior 72 (2017): 678-91, https://doi.org/dx.doi.org/10.1016/j.chb.2016.08.047.

52 Annemaree Lloyd, "Chasing Frankenstein's Monster: Information Literacy in the Black Box Society," Journal of Documentation 75, no. 6 (2019): 1476, https://doi.org/10.1108/JD-02-2019-0035.

53 Head, Fister, and MacMillan, "Information Literacy in the Age of Algorithms," 42.

54 Head, Fister, and MacMillan, "Information Literacy in the Age of Algorithms."

55 Taina Bucher, "The Algorithmic Imaginary: Exploring the Ordinary Affects of Facebook Algorithms," Information, Communication & Society 20, no. 1 (2017): 42, https://doi.org/10.1080/1369118X.2016.1154086.

56 Tanya Kant, Making It Personal: Algorithmic Personalization, Identify, and Everyday Life (Oxford: Oxford University Press, 2020), 152.

57 Jason Clark, Lisa Janicke Hinchliffe, and Scott Young, "Unpacking the Algorithms That Shape Our UX" (Washington, DC: IMLS, 2017), https://www.imls.gov/sites/default/files/grants/re-72-17-0103-17/proposals/re-72-17-0103-17-full-proposal-documents.pdf.

58 Association of College and University Libraries, "Framework for Information Literacy for Higher Education," 2015, http://www.ala.org/acrl/standards/ilframework; Jason Clark, "Building Competencies around Algorithmic Awareness" (Washington, DC: Code4Lib, 2018), https://www.lib.montana.edu/~jason/talks/algorithmic-awareness-talk-code4lib2018.pdf.

59 Jason Clark, Algorithmic Awareness (2018; repr., GitHub, 2020), https://github.com/jasonclark/algorithmic-awareness.

60 Ryan Cordell, "Machine Learning + Libraries: A Report on the State of the Field" (Washington DC: Library of Congress, 2020), 31, https://labs.loc.gov/static/labs/work/reports/Cordell-LOC-ML-report.pdf.

61 Michael Ridley, "Explainable Artificial Intelligence," Research Library Issues, no. 299 (2019): 28- 46, https://doi.org/10.29242/rli.299.3.

62 Matt Turek, "Explainable Artificial Intelligence (XAI)" (Arlington, VA: DARPA, 2016), https://www.darpa.mil/program/explainable-artificial-intelligence; DARPA, "Explainable Artificial Intelligence (XAI)" (Arlington, VA: DARPA, 2016), http://www.darpa.mil/attachments/DARPA-BAA-16-53.pdf.

63 Ashraf Abdul et al., "Trends and Trajectories for Explainable, Accountable, and Intelligible Systems: An HCI Research Agenda," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18 (New York: ACM, 2018), 582:1-582:18, https://doi.org/10.1145/3173574.3174156; Wojciech Samek and Klaus-Robert Muller, "Towards Explainable Artificial Intelligence," in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, ed. Wojciech Samek et al., 2019., Lecture Notes in Artificial Intelligence 11700 (Cham: Springer International Publishing, 2019), 5-22; Alejandro Barredo Arrieta et al., "Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI," ArXiv:1910.10045 [Cs], 2019, http://arxiv.org/abs/1910.10045.

64 R. David Lankes, "Decoding AI and Libraries," R. David Lankes (blog), July 3, 2019, https://davidlankes.org/decoding-ai-and-libraries/.

65 Catherine Coleman, "Artificial Intelligence and the Library of the Future, Revisited," Digital Library Blog (blog), November 3, 2017, https://library.stanford.edu/blogs/digital-library-blog/2017/11/artificial-intelligence-and-library-future-revisited; Head, Fister, and MacMillan, "Information Literacy in the Age of Algorithms"; Cordell, "Machine Learning + Libraries"; Clark, Hinchliffe, and Young, "Unpacking the Algorithms."

66 Cordell, "Machine Learning + Libraries," 2.

67 Thomas Padilla, Responsible Operations. Data Science, Machine Learning, and AI in Libraries (Dublin, OH: OCLC Research, 2019), 10, https://doi.org/10.25333/xk7z-9g97.

68 Nicholas Diakopoulos, "Algorithmic Accountability Reporting: On the Investigation of Black Boxes" (New York: Tow Center for Digital Journalism, Columbia University, 2014), 2, https://doi.org/10.7916/D8ZK5TW2; Finn, "Algorithm of the Enlightenment," 24.

69 Scott Lash, "Power after Hegemony: Cultural Studies in Mutation?," Theory, Culture & Society 24, no. 3 (2007): 71, https://doi.org/10.1177/0263276407075956.