TEACHING AND LEARNING INFORMATION SYNTHESIS: An intervention and rubric based assessment

Headnote

The purpose of this research was to determine how information synthesis skills can be taught effectively, and to discover how the level of synthesis in student writing can be effectively measured. The intervention was an information synthesis lesson that broke down the synthesis process into sequenced tasks. Researchers created a rubric which they used to assess students' levels of information synthesis demonstrated in their final research essays. A form of counting analysis was also created to see if other methods could help in measuring synthesis.

Findings from the rubric analysis revealed that students appear to benefit from the synthesis lesson. The level of synthesis, however, remains low overall. In addition, the study showed that the different measures of synthesis established were able to identify different levels of information integration. Discovering effective ways to measure and teach synthesis continues to be essential in helping students become information literate.

INTRODUCTION

Perhaps the most essential, and certainly one of the most complex research skills, is the ability to synthesize information. One researcher, J.D. Johnson (2009) writes: "... the ability for people to assimilate information they find into coherent personal strategies is perhaps the critical modern survival skill" (p. 601). Information synthesis is the process of analyzing and evaluating information from various sources, making connections between the information found, and combining the recently acquired information with prior knowledge to create something new. Information synthesis strategies are essential skills. Without them, we cannot derive new knowledge from these large amounts of data (Larsen, Wactlar & Friedlander, 2003; National Science Board, 2005). Effective information synthesis is also vital in developing effective writing and communication skills to share new knowledge. Coherent information synthesis is, therefore, required to productively participate in and contribute to our information-rich society. Yet college students have difficulty analyzing and synthesizing different pieces of information (Howard, Serviss, & Rodrigue, 2010).

The research questions explored in this study ask whether information synthesis skills can be taught effectively by scaffolding this complex cognitive task, and how the level of synthesis in student writing can be effectively measured. The study described here investigates an information synthesis lesson given to students in a university English writing class. The lesson broke down the synthesis process into several stages requiring students to go through information synthesis in discrete steps by means of individual reading and group discussion. At the end of the lesson students were asked to write a synthesized paragraph. The authors also collected final research papers in the class. Both writing products were analyzed for evidence of synthesis to evaluate the impact of the synthesis lesson as well as to establish different metrics for measuring synthesis.

Findings from the study revealed that students appear to benefit from the synthesis lesson. There were more instances of information synthesis in the final papers of students who received the lesson. The level of synthesis however, remained low overall. In addition, studies measuring synthesis identified different levels of information integration. The synthesis rubric used in this study reaffirms that synthesis, and the assessment of it, includes numerous skills and competencies. The implications of these findings suggest that teaching synthesis through scaffolding this process requires more than a single lesson and should perhaps be provided early in the semester. Rubrics and additional metrics that identify synthesis can be used to communicate to students about certain features of synthesized papers and can help instructors and librarians to more accurately assess student work and provide them meaningful feedback for improvement.

PROBLEM STATEMENT

The literature on information seeking behavior shows that students have superficial information seeking and research skills (Asher, Duke, & Green, 2010; Fitzgerald, 2004; Head and Eisenberg, 2009; Kolowich, 2010). In one study only 50% of student participants were able to successfully synthesize information from multiple sources (Mateos & Sole, 2009). Information synthesis is a key skill for participants in our knowledge society and requires complex processing (Fitzgerald, 2004; Goldman, 2004). Yet information literacy instruction and practice tend to favor easily-defined skills that often only emphasize the search component of the research process, leaving out higher order processes like information synthesis (Lloyd, 2007; Montiel-Overall, 2007; Simmons, 2005; Tuominen, Savolainen, & Talja, 2005; Webber & Johnston, 2000). Similarly, in the writing classroom, teachers are largely unfamiliar with how to teach synthesis sometimes implying it is a linear process (McGinley, 1992), leading Mateos and Sole (2009) to call for a "unique, careful teaching approach" (p. 448). In response, this study seeks to address the question of how to teach students to effectively synthesize information from multiple sources, and how to effectively assess and identify synthesis when it occurs in student work.

LITERATURE REVIEW

Information synthesis appears in the literature under several different guises. Commonly used terminology to describe the process of analyzing and evaluating information from various sources is multiple document (or source) comprehension (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Goldman & Scardamalia, 2013), multiple document processing (Goldman & Scardamalia, 2013), transliteracy (Andretta, 2009; Thomas et al., 2008), intertextuality (Goldman, 2004; Stadtler & Bromme, 2013), writing from sources (Howard, Serviss, & Rodrigue, 2010), and information synthesis (Blake & Pratt, 2002; Goldschmidt, 1986). Relevant literature to this study can be found in library and information science (information literacy and information problem solving models), education (cognition and literacy instruction), and composition and rhetoric (writing). Each field approaches the subject from a different angle. Different research and resulting instructional approaches of the various fields are discussed below.

Information Problem Solving Models

Information problem solving models, also known as information literacy models, mostly serve as scaffolds for teaching the research process or as frameworks when studying the same process. Information synthesis appears in all of the most well-known models. The The Big 6 model is an information problem solving model developed by Eisenberg and Berkowitz, and is used widely in K-12 schools. This model includes synthesis as step five in their six-step stages, which also includes task definition, information seeking strategies, location and access, use of information, and evaluation (Eisenberg & Berkowitz, 1990; Lowe & Eisenberg, 2005). This particular step includes organizing from multiple sources and then presenting the information. Here students are directed to read and then write from their notes from previous steps and to reflect on how best to present this information. In Stripling's six-phase Model of Inquiry, synthesis is contained in the construct phase (Stripling, 2010). This is where the bridge is built from previous knowledge to draw new conclusions, where conflicting information is confronted, conclusions are drawn, and evidence-based opinions are formed. In the Information Search Process (ISP), Kuhlthau does not use the label of synthesis, but does include the process itself in the formulation stage (Kuhlthau, 1991; Kuhlthau, Heinström, & Todd, 2008). The formulation stage is the fourth of six stages, where the learner forms a focus from all the information encountered, identifying and selecting ideas from multiple sources to form a focused perspective.

Information synthesis, a higher form of educational thinking

Information synthesis most commonly appears in the education literature as a level in the original Bloom's Taxonomy of Learning Domains (Bloom, 1956). This Taxonomy is a classification for understanding student learning and to promote higher forms of educational thinking. The Taxonomy is often depicted as a pyramid with the higher forms of thinking at the top. For understanding the cognitive domain of learning, the Taxonomy builds upon steps beginning with factual knowledge and moving to comprehension, application, analysis, synthesis, and evaluation. Synthesis, ranked second from the top in the original Taxonomy, is considered one of the most important goals in the field of education. Here, synthesis is defined as the building of structures or patterns from a variety of elements with emphasis on creating some new meaning or a new structure from the elements. Some of the keywords involved in synthesis include: combine, create, design, and summarize. In Krathwohl's (2002) revision of Bloom's Taxonomy, the original categories were renamed and their definitions revised to represent more active thinking. Synthesis was renamed to "create" and changed places with "evaluation" as the top category in the domain. The create category is defined as putting together elements to make a whole, including the elements of generating, planning, and producing.

The role of synthesis in text comprehension

Information synthesis can be seen in the area of text comprehension, specifically in multiple-source comprehension (also known as multiple document processing, or intertextuality). Historically, text comprehension research involved single-document comprehension; this was not extended to multiple texts until the 1990s when Wineburg (1991) studied how novices and experts reasoned about a historical event using multiple documents. Using think-aloud protocols Wineburg identified the strategies people used to come to a conclusion. More researchers followed (see Stadtler & Bromme, 2013 for details), resulting in a better understanding of the various strategies employed when processing multiple documents (e.g. Perfetti, Rouet, & Britt, 1999). According to Goldman and Scardamalia (2013), to be successful at synthesizing information from multiple documents, students need to be taught content knowledge, source expertise, and an understanding of how knowledge is created in the field of study. Once these are in place students can evaluate information, integrate it into existing belief structures, and create new knowledge. Both Jucks and Paus (2013) and Goldman and Scardamalia (2013) note the social aspect of creating meaning, and they emphasize the use of discussion when teaching multiple-document processing in general and the resolution of conflicting information between documents. Based on this research, the current study incorporates group discussion in the intervention, alternated by individual reading and reflection.

Synthesis in the writing classroom

The rhetoric and composition literature addresses how students learn to synthesize multiple texts. Synthesis is complex in nature; therefore, the reading processes, the writing processes, and writing from multiple sources are all relevant to this discussion.

This literature also considers synthesis to be a cognitively demanding task (Mateos, Martin, Villalon & Luna, 2008), which requires multiple activities such as organization, comprehension, problem detection, and problem solving (Bråten, I., Strømsø, 2003). Similarly, Nelson and Hayes (1988) noted that in order to write from multiple sources, students had to "coordinate a number of supporting activities." Flower, et al. (1990) determined that synthesis is a risky endeavor, where the reader's experience and knowledge, the text, and "reality itself may resist synthesis" (p. 50). Not surprisingly, only 50% of those high school and university students who were studied could successfully synthesize (Mateos & Sole, 2009). Torraco (2005) views synthesis as a creative activity "that produces a new model, conceptual framework, or other unique conception informed by the author's intimate knowledge of the topic" (p. 362). The same author describes four forms of synthesis, including a research agenda, a taxonomy or classification construct, alternative models or conceptual frameworks, and meta-theories.

Nancy Spivey (1989), a major contributor to research in this area, elaborates on the major components of discourse synthesis, including the ability to "select, organize and connect content from sources texts as they compose their own new texts" (p. 9). Like Wineburg (1991), both McGinley (1992) and Spivey (1984) note that much variation existed in students' writing processes with proficient and non-proficient college readers making different decisions in the ways they chose to make connections between texts.

This literature also focuses on how to effectively teach the synthesis process. Mateos and Sole (2009) found that very few teachers knew how to help students go beyond connecting main ideas between different sources, while McGinley (1992), looking at the connection between writing and thinking processes, concluded that teachers should avoid implying that writing from multiple sources is wholly linear. Following a collaborative approach similar to the current study, Fluellen (2011) paired students together to read aloud and map concepts. McGregor (2011) used coded, graphical representations of student work to start conversations about how students use sources. Another solution requires instructional technology to teach synthesis at younger stages, such as Turb°Cite or TurboWrite (Tooley, 2005).

Measuring information synthesis

The usefulness of using rubrics to help measure information literacy skills has been well documented by Oakleaf and others. Oakleaf comments on the ability of a rubric to "capture useable data about information-seeking behavior," and on the value of the rubric development process itself (2007, p. 28).

While no comprehensive rubric exclusively evaluating information synthesis was found, numerous rubrics have been created that evaluate information literacy, and they typically include elements of synthesis. Commonly found aspects of synthesis are as follows: establishing associations between texts, recognizing patterns among information (similarities, differences, unique instances), organizing information to express these relationships and patterns by using transitional sentences, and other explicit or implicit markers. Relevant rubrics include the Inquiry and Analysis VALUE Rubric (Association of American Colleges and Universities [AACU], 2010), Northern Arizona University's (NAU) Synthesis Essay Rubric, Rubric Assessment of Information Literacy Skills (RAILS) Using Information to Accomplish a Purpose Rubric, the Evergreen Synthesis Paper Rubric (Ford), and the General Education Assessment Rubrics (Klassen, 2014). We determined that combining and rethinking pieces of these current rubrics would best suit the task of determining whether or not synthesis was present in student papers.

The rubrics listed above and an in-depth analysis of numerous student papers informed the creation of the information synthesis rubric (see Appendix A) used in this study. The VALUE rubric was beneficial as it was developed by teams of faculty at colleges and universities across the United States. The aim of the VALUE project, which resulted in the creation of 16 different rubrics, was to have a national framework to support common dialog and understanding in specific areas for undergraduate level work (AACU, 2010). The Inquiry Analysis rubric heavily informed a category (subscale D) in our rubric which focused on source organization. The Synthesis Essay Rubric was a collaborative effort between NAU's eLearning Center and faculty to design effective instruction and assessment. The Synthesis Essay Rubric was especially important in creating subscale C in the present study: identifying conversations. The Using Information to Accomplish a Purpose Rubric was created for a library instruction teaching workshop focused on assessment and deposited in the RAILS repository, a funded research project investigating the use of analytical rubrics in the assessment of information literacy. Terry Ford, from Evergreen State College, created the Synthesis Paper Rubric. Ford's rubric was helpful in providing vocabulary to describe different skill levels for the present study: emerging, developing, and proficient. Ford's rubric was also used extensively to create subscale E: analyzing sources to create something new. Klassen's General Education Assessment Rubrics (2014) helped to distinguish levels of progression as students become more proficient in developing a range of skills essential to general education. The rubric adapted two of the seven categories in the Synthesis Rubric, including information literacy skills and synthesis and critical thinking skill patterns. The researchers relied on these five rubrics to develop a rubric which more adequately provided a guide for identifying synthesis in student work.

RESEARCH DESIGN AND METHODS

The research team included two faculty librarians with a focus on library instruction, and two teaching faculty from the Instructional Technology & Learning Sciences Department, all with a vested interest in helping students improve their synthesis skills. The researchers developed an information synthesis lesson, which was implemented in four sections of an English 2010 class. English 2010, Intermediate Writing: Research Writing in a Persuasive Mode, is a required second-year composition class. The focus of the class, according to the course description, is the "writing of reasoned academic argument supported with appropriately documented sources. [It] focuses on library and Internet research, evaluating and citing sources, oral presentations based on research, and collaboration." Writing products were collected at the end of the intervention in addition to students' final papers for the course. These products were analyzed using the synthesis rubric that was created and adapted from the previously mentioned rubrics.

Participants

The 87 study participants were enrolled in four Spring 2013 sections of English 2010, all taught by the same instructor. The enrollments for the four sections consisted of 21, 22, 23, and 21 students respectively. While biographical data on individual students was not collected, the total participant pool was 32 female students and 55 male students. There were five freshmen, 29 sophomores, 34 juniors, and 19 seniors. For the paper analysis, nine papers were randomly selected from each section.

The control group participants were enrolled in four Fall 2011 sections of English 2010, all taught by the same instructor. There were 89 total control group students, of whom 44 were female and 45 were male. There were six freshmen, 26 sophomores, 37 juniors, and 20 seniors.

Measures

To measure the extent to which information synthesis took place in the written products (synthesis paragraphs and final papers), researchers in this study developed an information synthesis rubric based on an in-depth analysis of student papers and a series of information literacy rubrics that included aspects of synthesis. The rubric consisted of five categories: A. Source variety, B. Uses information from sources effectively, C. Identifies conversations among information from different sources, D. Organizes sources overall in a meaningful, purposeful way, and E. Analyzes sources to create something new or draw conclusions and make generalizations. These categories were scored as unacceptable (0), needs improvement (1), developing (2), advanced/ mastery (3). The rubric was revised through three iterations in which researchers applied it to a total of ten student papers from the same course in a previous semester. This initial assessment process ensured consistency between raters in measuring students' level of proficiency in synthesizing information, and resulted in the final rubric version in Appendix A.

Procedures

The information synthesis lesson was implemented in four sections of ENGL 2010. The same researchers taught all four sections. The lesson (see Appendix B) lasted approximately 75 minutes and broke the synthesis process down into sequenced tasks. The lesson was based on an information synthesis workshop developed by Johnson at Arizona State University West (2003). Students were placed in groups of three. Using PowerPoint, the librarians provided instructions and discussed the characteristics of synthesis.

After basic definitions were established, the guided practice began. Each step of the guided practice had clear time delineations with corresponding instructions on a PowerPoint slide. Students were instructed to read two articles. Each student read one unique article and one article in common with the other members of his or her group. Students each highlighted the main points or areas about which they had questions. Then students wrote the five main points of their assigned articles on post-it notes, shared their notes with group members, and worked together to group main points by topic. Once students established general categories, they regrouped their post-it clusters and titled each one. At the end of the lesson, students were instructed to individually write a paragraph that synthesized one of their group's clusters.

Two researchers observed the lessons to record students' reactions, paying special attention to student questions and their difficulties with particular steps in the process. After each lesson, the researchers asked for student and instructor feedback through debriefing and a short online survey. They collected students' clusters of concepts derived from in-class readings for later analysis, and at the end of the semester, they collected students' final papers as well.

RESULTS

The researchers scored all student papers using the rubric and tested their inter-rater reliability. They then measured whether the information synthesis intervention improved student synthesis skills, based on student papers and the synthesized paragraphs. Finally, the researchers established whether counting visible markers of synthesis could predict the level of synthesis in student papers scored by the rubric.

The information synthesis rubric

To test the information synthesis rubric for reliability, the researchers scored 72 final student papers (36 from the control group and 36 from the treatment group). Each paper was rated by two raters individually and then discussed to form a consensus. To test the reliability of the rubric, inter-rater reliability was calculated using Cronbach's alpha. Raters had acceptable reliability overall (α = .72); the breakdown was α = .73 on control papers and α = .71 on treatment papers. Cronbach values between 0.70 and 0.95 are considered to be acceptable (Tavakol & Dennick, 2011).

The impact of the information synthesis lesson

The researchers calculated the difference between the control and treatment group papers for each subscale. To determine if there was a significant difference between the calculated scores of the control and treatment papers, an analysis of variance was calculated for each of the five subscales of the rubric. Subscales A, B, D, and E had no significant differences. A main effect F (1, 70)=7.36, p<.01 was reported for Conversations Among Information from Different Sources (subscale C), indicating that the treatment group papers were better than the control group papers for this subgroup. Out of the five subscales, subscale C is essential, particularly in its relationship to category E, which focuses on students' ability to enter the conversation in a meaningful way. In order to enter the conversation, students must first be able to identify and articulate the conversations already taking place, which is the focus of category C. Table 1 reports the analysis of variance for all five subscales.

Synthesis Paragraph Analysis

At the end of the intervention, each group had several information clusters on their butcher paper (see figure 1). Every cluster contained several sticky notes, each note including a key point extracted from articles read by the students. The sticky notes were grouped together in topical clusters as part of an iterative group process.

At the end of the lesson, student participants were asked to pick a cluster and write a paragraph synthesizing it. Students submitted their synthesis paragraphs using a web form. In the writing instructions for the synthesized paragraph (provided in class and also on the form), students were asked to use the article ID (e.g., A, B, C) to cite an article in the paragraph. The same web form also collected additional data: student name, group number, how many different articles were contained on the sticky notes in the cluster they used for writing their paragraph. Students submitted 74 synthesized paragraphs. After removal of paragraphs without in-text citations, paragraphs based on single-article clusters, and paragraphs from students who missed the intervention, a total of 52 paragraphs remained for the analysis.

The 52 paragraphs were analyzed using subscale C of the synthesis rubric, which seemed to most robustly address the key skill of making connections between sources. No or very weak connections were made for 9 (17.3%) paragraphs; only implicit connections between the articles were made for 10 (19.2%) paragraphs; some explicit connections were made for 14 (26.9%); and several explicit connections were made for many of the paragraphs (19 or 36.5%).

When comparing the level of synthesis in the paragraphs with the level of synthesis found in student papers, the paragraphs fared much better in the highest score category of subscale C (Identifies conversations among information from different sources). This remained true after adding paragraphs without text citations back into the analysis (see Table 2). This finding suggests that our intervention does help students with the process of synthesizing at the paragraph level, but that the process does not necessarily transfer to the same extent to a larger scale project, such as an entire paper.

Other methods for measuring synthesis

After scoring each paper using the rubric, the researchers wanted to know if there were other identifiers of synthesis, such as how many times a student used a source across paragraphs. With this in mind, trends were identified using a counting analysis developed by the researchers. First, each source was highlighted and numbered in the works cited/bibliography list with a different color. Second, each source used in the paper was highlighted in the matching color. A source could have been used in the paper, but not listed in the reference list. These sources were counted and added to the works cited list, using the author or title, if available. Third, a comment was added next to each source noting if it was quoted or referenced or both. Fourth, the number of sources used in each paragraph was counted (i.e., source #2 was used in three different paragraphs). And fifth, the researchers counted how many paragraphs from each essay used one source, how many used two sources, and so on (see Table 3).

General counting observations

As can be seen from Table 3, the control and the treatment papers had the same percentage of paragraphs with the same number of sources. Over one-half (56%) of the paragraphs citied no sources, while almost one-third (29%) cited only one source. However, having a large number of single-source paragraphs does not necessarily mean the paper scored low on synthesis overall, as evidenced by control group paper 046-12. This paper received one of the two highest synthesis rubric totals (score 10), but had nine paragraphs with only a single source in them. In contrast, the other high scorer on the synthesis rubric, paper 046-17 (score 10), had three paragraphs with two sources in them and one paragraph with four sources, which could indicate a high level of synthesis.

Source use

A common scenario for students was to include a number of sources in the references page that were not referenced or quoted in the paper. In this case, use refers to any time a source was referenced either through paraphrasing or by direct quoting.

Most students only used a single source per paragraph; there were 119 instances of this for the control group and 161 instances for treatment group papers. The treatment group used 54 more sources total than the control group.

Papers that used one source often-an instance of a source being used in three or more different paragraphs-tended to have lower synthesis scores (mode=5). From the control group, paper 049-20 with a rubric score of nine, and paper 046-12 with a score of 10 were the exceptions. These students tended to use one source frequently, but they had higher synthesis scores. From the students in the treatment group, most papers using one source in three or more paragraphs scored a six or seven from the rubric.

In general, students paraphrased their sources much more often than quoting. In total, students from the control group quoted their sources 60 (26%) times and paraphrased them 167 (74%) times, while students with the treatment group quoted 70 (23%) times and paraphrased 233 (77%) times (see Table 4). In terms of information synthesis, this indicates that students from the treatment group are using multiple sources in their papers. and they use the information from those sources in a more advanced way, paraphrasing instead of directly quoting. These two areas are covered by subscale A and B in the rubric.

Comparing the synthesis rubric and the counting analysis

While the number of sources per paragraph did not necessarily predict a high or low score on the synthesis rubric as a whole, there was a connection between papers that scored a two or higher in subscale C from the rubric; they all had at least one paragraph with two sources.

There is a higher correlation between the counting analysis and the subscale C consensus score for the treatment papers (r=0.74) versus the control papers (r=0.63). Occasionally higher scores had more sources used, however there were also instances where there was no correlation between high rubric scores and high number of sources used. For example, one of the papers only used three sources and scored an 8 on the rubric. From the treatment group, another paper used four sources and scored an 8 on the synthesis rubric. This indicates that number of sources used does not necessarily correlate with a student's ability to synthesize that information.

DISCUSSION

All three assessment techniques-the rubric assessment of the paragraphs from the intervention, the rubric assessment of the final papers in comparison to the control group, and the counting analysis of how sources were used within final papers-indicate students struggle with synthesis and benefit from teaching methods that break down the different skills involved in synthesis. Each of these techniques revealed key student behaviors that can help librarians and educators modify their instruction practices.

These findings confirm the difficulty of the synthesis process for students, detailed in much of the previous literature (Howard, Serviss, & Rodrigue, 2010). The findings also show that teaching interventions in this process can help students improve their ability to synthesize. In this case, the intervention slowed down this process for students, and it helped students identify main ideas and collaboratively group those ideas with others. Interestingly, the skill of being able to identify conversations among different texts appeared to transfer to students' final papers. Students may have also benefited from having explicit definitions of synthesis, such as sharing and discussing the specific elements of the comprehensive synthesis rubric used in this study.

Defining Synthesis & Rubric Creation

As detailed in the literature, definitions of what synthesis is and what it looks like vary widely. One of the major components of this research emerged with the realization that solid markers for measuring synthesis are difficult to find, largely due to its complexity and scope. The creation of this rubric helped to pinpoint what synthesis involves and what it looks like in a paper. The researchers determined that five categories are most significant when identifying the presence of information synthesis: mainly source variety (subscale A), using information sources effectively (subscale B), identifying conversations among sources (subscale C), organizing sources meaningfully (subscale D), and analyzing sources to create something new (subscale E). These categories and their descriptions at each level vary from other existing rubrics, although they are adapted from pieces of existing ones.

During the rubric creation and revision phase, the researchers continuously reached for measurable language whenever possible. For example, source variety (subscale A) clarifies numerically how varied the sources must be to be categorized at each level, focusing particularly on whether the student uses sources that speak to a variety of perspectives rather than using sources that all trend towards the same view. Subscale B, using information sources effectively, attempts to distinguish haphazard use of sources from summarizing main ideas. Subscale C, identifying conversations, underwent the most revision during this process, including defining "explicit" as sources that included textual indicators (i.e., using terms such as "likewise" or "in contrast"), or the sources are placed side by side in the same paragraph. Subscale D attempts to show variation in the effectiveness of the organization of sources. Subscale E differentiates between works that make assumed generalizations and those that come to well-reasoned logical conclusions as a result of how they use and respond to their sources.

These carefully articulated categories, or elements of synthesis, combined with measurable and specific descriptions at each level, helped the researchers to feel confident in how they were assessing students' ability to synthesize. The categories also helped the researchers to better understand where an instructor or librarian might intervene in a student's process and how to more carefully scaffold activities that support students in areas in which they struggle. The results of the inter-rater reliability analysis reinforced confidence in the reliability of the rubric, as well as in the usefulness of the intervention.

Impact of the intervention on the synthesis paragraphs

Judging by the synthesized paragraphs that students wrote directly after the intervention (scored on subscale C of the rubric), it appears that the synthesis lesson helped students with the process of synthesizing information from difference sources into paragraphs. This effect is much less clear when looking at the final student papers. One of the reasons for this could be that the intervention combined individual thought processes with group discussions, while the final papers are individual efforts. Another reason could be that the paragraphs were written immediately after the intervention, while the papers were not completed for another two weeks. Additionally, several students had already started on their final papers before our intervention; they may have been too far along in the writing process to start over and fully apply the synthesis process they learned in class.

Another aspect to consider is suggested by Cavanaugh and Perlmutter (1982) who noted that students tend to use strategies when instructed to do so, but stop using these strategies when they are no longer required. Students in this study went through the synthesis strategy as part of the intervention and just before they wrote their synthesis paragraphs. The instructions for the final paper did not mention that students should use the synthesis strategy. There are two possible solutions for this in future work. One would be to slowly fade the scaffold, which would lengthen the intervention. The other would be to create a synthesis tool that would serve as a cognitive support with the synthesis steps built in across the entire research and writing process. Pea (2004) calls this latter approach a case of distributed intelligence, where the intelligence is incorporated into the software. Changing the intervention to allow fading the scaffold should be possible by using an informed learning approach (Maybee, Bruce, Lupton, Rebmann, 2013; Bruce and Hughes, 2010), where students actively use information to learn instead of being taught an information skill like synthesis in isolation. The researchers are currently experimenting with this approach in an online class where students curate content to create learning modules.

The impact of the intervention on final papers

The majority of the control group papers (from the Fall class that did not receive the intervention) received a score of one on subscale C, while the majority of the treatment group papers submitted after the intervention received a score of two. This indicates that our intervention had an effect. Interestingly, the synthesis scores for submitted paragraphs are more evenly distributed across all four categories in each subscale compared to the final paper scores, which are mostly clustered in the two middle scores. Conversely, papers that used three or four sources in a paragraph did not necessarily correlate to having higher rubric scores.

Effectiveness of Counting Analysis

The counting method did reveal some interesting patterns about how students use sources. There does not appear to be a correlation between the number of paragraphs containing certain numbers of cited sources and the total synthesis rubric score. This might be because the individual subscales are measuring something different that cannot be captured by counting.

Also, when looking at how many times sources were used across the submitted paragraphs, there was a significant difference between the control and the experimental group on subscale C. The rubric scores from the other categories did not appear to relate significantly to the counting analysis results. Subscale C did correlate well with the rubric scoring because the language of the rubric clarified that papers that received a two in subscale C had to have at least two sources in one paragraph in conversation with one another.

While this analysis was fairly limited in correlating numbers of sources to identifying synthesis, more research is needed to explore alternative ways of measuring this skill. In general, this analysis did help identify student behaviors relating to synthesis. There were also quite a few instances of sources listed in a reference page that were not actually used in an identifiable way in the paper. This may reflect a tendency for student to place more emphasis on finding and documenting sources, rather than learning to use them effectively to support their argument and contribute something new to a conversation. These findings indicate a continued need for teaching synthesis, but also for revising scaffolds, such as the intervention used in this study, to target some of these specific areas.

CONCLUSION

Despite the difficulty of learning and teaching synthesis, breaking down the processes involved in synthesis, both in definition and in scaffolded practice, can help instructors teach and assess synthesis more effectively. In this study, the authors created a comprehensive, reliable rubric that clearly defined multiple aspects of synthesis and helped to determine which elements of synthesis students struggle with, as well as identifying when synthesis was employed in student work. Using this rubric, the researchers determined that the attempts to break down the synthesis process in the intervention may have improved students' ability to identify conversations within the literature and incorporate those conversations, in identifiable ways, into their final persuasive research papers.

These findings are relevant for instructors and for librarians as they support the research process. The issue of whether librarians should be involved in teaching synthesis is an important one. As librarian roles continue to shift, and as they collaborate with faculty to find ways to help students with the threshold concepts in the Framework for Information Literacy for Higher Education, librarians may be increasingly more involved in teaching higher level concepts like synthesis. In this study, a librarian taught the intervention, but the lesson plan can also be shared by librarians with instructors looking for ways to help students learn synthesis. It is clear from previous studies on student research skills, such as reports from Project Information Literacy (Head & Eisenberg, 2009), that appropriate usage of information is a major area of difficulty for students. Students and instructors may not recognize that librarians can assist the synthesis process, but as this study shows, they can. Librarians must be clearer about their role in helping with any facet of the research process, including synthesis.

Future avenues for research include adapting and creating more interventions that breakdown the synthesis process, particularly in relation to the five categories identified in the rubric used for the present study. In this case, the intervention targeted identifying conversations, but more research is needed in how to help students with the other rubric elements. Further inquiry into using the informed learning approach may also help identify effective ways to scaffold elements of synthesis. Alternative ways to assess synthesis, such as the counting analysis method attempted here, also warrants more exploration. Methods like these can help librarians and instructors identify when synthesis is occurring and what it looks like in order to teach students effective strategies for synthesizing information.

References

REFERENCES

Afflerbach, P., & Cho, B. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. Israel & G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 69-90). New York, NY: Routledge.

Andretta, S. (2009). Transliteracy: take a walk on the wild side. Paper presented at the World Library and Information Congress: 75th IFLA General Conference and Assembly, Milan, Italy. Retrieved from http://eprints.rclis.org/14868/

Asher, A., Duke, L., & Green, D. (2010). The ERIAL project: Ethnographic research in Illinois academic libraries. Academic Commons, (May). Retrieved from http:// www.academiccommons.org/2014/09/09/ the-erial-project-ethnographic-research-in-illinois-academic-libraries/

Association of American Colleges and Universities (2010). Inquiry and Analysis VALUE Rubric. Retrieved from https:// www.aacu.org/value/rubrics/inquiry-analysis

Association of College & Research Libraries (2014). Framework for information literacy for higher education (draft). Retrieved from http:// acrl.ala.org/ilstandards/wp-content/ uploads/2014/11/Framework-for-IL-for-HE-draft-3.pdf

Blake, C., & Pratt, W. (2002). Collaborative information synthesis. Paper presented at the Meeting of the American Society of Information Science and Technology (ASIS&T), Philadelphia, PA.

Bloom, B. S. (1956). Taxonomy of educational objectives: the classification of educational goals. New York, NY: Longmans, Green.

Bråten, I., & Strømsø, H. I. (2003). A longitudinal think-aloud study of spontaneous strategic processing during the reading of multiple expository texts. Reading and Writing: An Interdisciplinary Journal, 16(3), 195-218. doi: 10.1023/ A:1022895207490

Britt, M. A., & Aglinskas, C. (2002). Improving students' ability to identify and use source information. Cognition and Instruction, 20(4), 485-522.

Bruce, C., & Hughes, H. (2010). Informed learning: A pedagogical construct attending simultaneously to information use and learning. Library & Information Science Research, 32(4), A2-A8. doi:10.1016/ j.lisr.2010.07.013

Cavanaugh, J. C., & Perlmutter, M. (1982). Metamemory: A critical examination. Child Development, 53, 11-28. doi: 10.2307/1129635

Eisenberg, M., & Berkowitz, R. E. (1990). Information problem-solving: The Big Six Skills approach to library & information skills instruction. Norwood, N.J.: Ablex Pub. Corp.

Fitzgerald, M. A. (2004). Making the leap from high school to college: Three new studies about information literacy skills of first-year college students. Knowledge Quest, 32(4). Retrieved from http:// www.libs.uga.edu/cloc/readings/ fitzgerald.pdf

Flower, L., Stein, V., Ackerman, J., Kantz, M. J., McCormick, K., & Peck, W. C. (1990). Reading-to-write: Exploring a cognitive and social process. New York: Oxford University Press.

Fluellen, J. E., Jr. (2011). Connecting minds: Emergence of a future bent, writing/ thinking intensive psychology course. Occasional Paper #10. Retrieved from http://files.eric.ed.gov/fulltext/ ED521161.pdf

Ford, T. Synthesis Paper Rubric. Resources for first year and learning communities. Retrieved from http:// academic.evergreen.edu/f/fordter/ synthrubric.htm

Goldman, S. R. (2004). Cognitive aspects of constructing meaning through and across multiple texts. In N. Shuart-Faris & D. Bloome (Eds.), Uses of intertextuality in classroom and educational research (pp. 317-351). Greenwich: Information Age Pub.

Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356-381. doi: 10.1002/ rrq.027

Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31 (2), 255-269. doi: 10.1080/ 10824669.2013.773217

Goldschmidt, P. G. (1986). Information synthesis: A practical guide. Health Services Research, 21(2 PT 1), 215-237. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/ articles/PMC1068946/

Head, A. J., & Eisenberg, M. B. (2009). Lessons learned: How college students seek information in the digital age. Project Information Literacy (PIL) at the University of Washington's Information School, Seattle, WA. Retrieved from http:// projectinfolit.org/images/pdfs/ pil_fall2009_finalv_yr1_12_2009v2.pdf

Howard, R. M., Serviss, T., & Rodrigue, T. K. (2010). Writing from sources, writing from sentences. Writing & Pedagogy, 2(2), 177-192. doi: 10.1558/wap.v2i2.177

Johnson, C. R. (2003). Synthesizing information. Workshop materials. Arizona State University West. Glendale, AZ. Retrieved January 24, 2012 from http:// www.west.asu.edu/johnso/synthesis/ synthesis.html#examples

Johnson, J. D. (2009). An impressionistic mapping of information behavior with special attention to contexts, rationality, and ignorance. Information Processing & Management, 45(5), 593-604. doi: 10.1016/ j.ipm.2009.04.005

Jucks, R., & Paus, E. (2013). Different words for the same concept: Learning collaboratively from multiple documents. Cognition and Instruction, 31(2), 227-254. doi: 10.1080/07370008.2013.769993

Klassen, P.T. (2014). General Education Assessment Rubrics. Retrieved from http:// documentingexcellence.com/examples/ rubricgened/rubric.htm

Kolowich, S. (2010, October 18). Searching for better research habits. Inside Higher Ed. Retrieved from https:// www.insidehighered.com/news/2010/09/29/ search

Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41(4), 212-218. doi: 10.1207/ s15430421tip4104_2

Kuhlthau, C. C. (1991). Inside the search process: information seeking from the user's perspective. Journal of the American Society for Information Science, 42(5), 362-371.

Kuhlthau, C. C., Heinström, J. & Todd, R.J. (2008). The 'information search process' revisited: is the model still useful? Information Research, 13 (4).

Larsen, R., Wactlar, H., Friedlander, A., & University of Pittsburgh. (2003). Knowledge lost in information: Report of the NSF workshop on research directions for digital libraries, June 15-17, 2003, Chatham, MA. Pittsburg, PA: University of Pittsburg, School of Information Sciences in cooperation with the Department of University Marketing Communications.

Lloyd, A. (2007). Recasting information literacy as sociocultural practice: Implications for library and information science researchers. Information Research, 12, 1-13.

Lowe, C. A., & Eisenberg, M. B. (2005). Big 6TM Skills for information literacy. In K. E. Fisher, S. Erdelez & L. McKechnie (Eds.), Theories of information behavior (pp. 63-68). Medford, NJ: Information Today.

Macedo-Rouet, M., Braasch, J. L. G., Britt, M. A., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204-226. doi: 10.1080/07370008.2013.769995

Mateos, M., Martin, E., Villalon, R., & Luna, M. (2008). Reading and writing to learn in secondary education: Online processing activity and written products in summarizing and synthesizing tasks. Reading and Writing: An Interdisciplinary Journal, 21(7), 675-697. doi: 10.1007/ s11145-007-9086-6

Mateos, M., & Sole, I. (2009). Synthesising information from various texts: A study of Procedures and products at different educational levels. European Journal of Psychology of Education, 24(4), 435-451. doi: 10.1007/BF03178760

Maybee, C., Bruce, C. S., Lupton, M., & Rebmann, K. (2013). Learning to use information: Informed learning in the undergraduate classroom. Library & Information Science Research, 35(3), 200-206. doi: 10.1016/j.lisr.2013.04.002

McGinley, W. (1992). The role of reading and writing while composing from multiple sources. Reading Research Quarterly, 27 (3), 227-248.

McGregor, J. (2011). A Visual Approach: Teaching Synthesis. School Library Monthly, 27(8), 5-7. Retrieved from http:// www.schoollibrarymonthly.com/articles/ Mcgregor2011-v27n8p5.html

Montiel-Overall, P. (2007). Information literacy: Toward a cultural model. Canadian Journal of Information and Library Science, 31(1), 43-68.

National Science Board. (2005). Long-lived digital data collections : enabling research and education in the 21st century. Arlington, VA: National Science Foundation. Retrieved from http:// www.nsf.gov/pubs/2005/nsb0540/

Nelson, J., & Hayes, J. R. (1988). How the writing context shapes college students' strategies for writing from sources. Berkeley, CA: Center for the Study of Writing, Carnegie Mellon University.

Northern Arizona University. Synthesis Essay Rubric. Retrieved from http:// www2.nau.edu/d-elearn/assessment/ documents/Essay_2_rubric.pdf

Pea, R. D. (2004). The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity. Journal of the Learning Sciences, 13(3), 423-451. doi: 10.1207/s15327809jls1303_6

Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. v. Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99- 122). Mahwah, NJ: Lawrence Erlbaum Associates.

Oakleaf, Megan J. (2006). Assessing information literacy skills: A rubric approach. Retrieved from http:// www.meganoakleaf.info/ oakleafdissfrontpapers.pdf

Rouet, J.-F. (2006). The skills of document use: from text comprehension to Web-based learning. Mahwah, NJ: L. Erlbaum Associates.

Rubric Assessment of Information Literacy Skills (RAILS): Using Information to Accomplish a Purpose (2011). Assessment Immersion. Retrieved from http:// railsontrack.info/rubrics_details.aspx? wkid=181&catid=12

Simmons, M. H. (2005). Librarians as disciplinary discourse mediators: Using genre theory to move toward critical information literacy. portal: Libraries and the Academy, 5(3), 297-311. doi: 10.1353/ pla.2005.0041

Spivey, N.N. (1984). Discourse synthesis: Constructing texts in reading and writing. (Outstanding Dissertation Monograph Series). Newark, DE: International Reading Association.

Spivey, N. N., & King, J. R. (1989). Readers as writers composing from sources. Reading Research Quarterly, 7-26.

Stadtler, M., & Bromme, R. (2013). Multiple document comprehension: An approach to public understanding of science. Cognition and Instruction, 31(2), 122-129. doi: 10.1080/07370008.2013.771106

Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers' memory for and use of conflicting information from science texts as function of presentation format and source expertise. Cognition and Instruction, 31(2), 130-150. doi: 10.1080/07370008.2013.769996

Stripling, B. (2010). Teaching students to Think in the digital environment: Digital literacy and digital inquiry. School Library Monthly, 26(8), 16-19. Retrieved from http://www.schoollibrarymonthly.com/ articles/Stripling2010-v26n8p16.html

Strømsø, H. I., Bråten, I., Britt, M. A., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31 (2), 176-203. doi: 10.1080/07370008.2013.769994

Tavakol, M., & Dennick, R. (2011, June). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53-55. doi: 10.5116/ijme.4dfb.8dfd

Thomas, S., Joseph, C., Laccetti, J., Mason, B., Mills, S., Perril, S., & Pullinger, K. (2007). Transliteracy: Crossing divides. First Monday, 12(12). doi: 10.5210/ fm.v12i12.2060

Tooley, M. (2005). Big6 TurboTools and synthesis. Library Media Connection, 24(2), 30-31.

Torraco, R. J. (2005). Writing Integrative Literature Reviews: Guidelines and Examples. Human Resource Development Review, 4(3), 356-367.

Tuominen, K., Savolainen, R., & Talja, S. (2005). Information Literacy as a sociotechnical practice. Library Quarterly, 75(3), 329-345.

Webber, S. A. E., & Johnston, B. (2000). Conceptions of information literacy: new perspectives and implications. Journal of Information Science, 26(6), 381-397.

Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83(1), 73-87. doi: 10.1037/0022-0663.83.1.73

Wolfe, M. B. W., & Goldman, S. R. (2005). Relations between adolescents' text processing and reasoning. Cognition and Instruction, 23(4), 467-502. doi: 10.1207/ s1532690xci2304_2

Appendix

(ProQuest: Appendix omitted.)