1. Introduction
A face-to-face conversational narrative is co-constructed by all conversation partners present, i.e., narrators who influence the storyline utilizing a multimodality of communication resources (e.g., gestures, language and images), though in practice it is often dominated by the spoken word [1,2]. Engaging in conversations to share personal experiences is a process of interrelated emotional, linguistic, social, language and narrative skills (e.g., recalling, describing and retelling experiences) [3,4,5]. Theories of social semiotics and visual anthropology suggest that individuals are sign-makers who create and use the most suitable semiotic resources to share their experiences and thoughts [6,7]. Story-sharing occurs through the production of signs, which takes place through social interactions, such as a multimodal conversation [7].
Today, digital media, such as film, are more often used to augment face-to-face conversations in private, educational and research settings [8,9]. These images, adopted as resources, do not stand alone, but instead are part of interactions to construct narratives, conveying the thoughts, ideas and emotions represented by the images. In visual anthropology, this means of integrating images within conversations is referred to as film elicitation [10,11]. Through the collaborative viewing of a film, narrators can pinpoint and refer to a specific film frame [12,13]. An object, action or person in the film may become a shared referent when mutually recognized by narrators, which can then be used to augment/clarify communication, bolstering joint attention and promoting story growth [14,15]. When participants create and share their own films, the conversation can become even more personalized, a procedure grounded in another established visual anthropology method known as participatory film [16]. By making choices during filming and film-editing, participants can create their own personal media resources captured from their own perspectives to share experiences [17,18]. The combination of participatory film-making with film elicitation may enhance multimodal story-sharing for everyone but especially for individuals with complex communication needs (CCNs), who can experience particular difficulties with spoken or written language [9,19].
People with CCNs have limited or no natural speech, which may challenge them in expressing themselves easily and being understood by conversation partners. They may benefit from access to augmentative and alternative communication (AAC) technologies, such as a speech-generating device (SGD), and other solutions to express themselves, with a range of resources such as (computerized) spoken, written and/or signed language, symbols and images [20,21]. Aided narrators often use only single-word or truncated utterances and seldom engage in spontaneous conversations to share a personal experience, since they are challenged by linguistic structures, time, power imbalance, underestimation and, for those who use high-tech AAC supports, unwieldy technical interface designs [4,22]. As a result, they may struggle to share personal experiences in spontaneous daily conversations [23,24]. To assist aided narrators, natural (typically) speaking conversation partners may dominate, asking limiting yes/no questions, or questions for which they already know the answers, in a way that takes the lead in the storytelling [25]. If the exchange of personal stories becomes uncomfortable, due to time pressures or miscommunication, aided narrators may withdraw from these important social interactions [4]. Such experiences can lead to social isolation, limiting opportunities for aided narrators to develop their language and narrative skills as conversationalists and potentially hindering personal development [26].
New audiovisual technologies and media resources, such as film, may assist an aided narrator in making these social interactions more enjoyable and in parallel develop multimodal literacy skills [27,28]. In recent years, audiovisual media have become more accepted and the associated technologies (e.g., mobile phones, iPads and tablets) more accessible and available, including to film-/sign-makers who require assistive technologies related to motoric disabilities [23,25]. Several AAC studies have shown the efficacy and potential of such media in supporting learning activities [11,29]. However, videos and photographs adopted in AAC practice are typically captured, created and prepared by individuals other than the aided narrators themselves, often in combination with prepared spoken/written texts. While these resources may provide the visual context of an event to render it more understandable, they seem less conducive to supporting a conversation about an aided narrator’s personal experiences. These images and texts, prepared by others, may lead the conversation in a direction that diverges from that intended by the aided narrator [30].
Narrators/sign-makers who rely on AAC often demonstrate agency in finding creative resources, such as finger-/eye-pointing to an object in their environment or in an image from a past event [23,31]. By capturing, editing and using their own self-created films as a resource, named here a personal-video-scene (PVS), narrators with CCNs may exert more agency and control in their conversational narratives [18,19]. Conversational control can be reflected in the following: (1) the capacity to initiate or sustain a topic (topic development) (2) with verbal and nonverbal communicative expressions, creating comments and questions (conversational structure), which (3) must be understood by co-narrators to create common ground (conversational repair), a necessity for (4) maintaining a conversation (conversation maintenance) [32,33]. In the following paragraphs, these four components of conversation control are discussed in more detail, related to self-created digital film, a PVS.
1.1. Topic Development
A conversational narrative is an interactive process where topic development is accomplished by investment from all interlocutors [1,33]. This process of story growth relates to conversational control, since each narrator needs the technology, resources, space and time to initiate and sustain a topic, a subject or theme of conversation. The availability and type of resources may influence the process of topic development [7]. A PVS, displayed on the screen of a phone or TV, can offer contextual and detailed topic information of a past event, such as people in attendance, objects, locations and actions, to initiate or sustain topics. Effectively, the PVS is a compilation of pre-selected topics, captured and prepared by the aided narrators themselves, which may assist all narrators in the construction of questions and comments.
1.2. Conversational Structure
A conversational narrative is a construct of multiple discourse units (DUs), which are defined by Light [5] as any communicative act initiated within the context of a conversational narrative—verbal, visual or otherwise. Each DU may consist of single or multifarious semiotic resources such as (computerized) spoken, written, manually signed, finger-/eye-pointing, symbols or images (moving or static) such as the PVS, which can merge to form a multimodal whole [7]. These DUs may be employed by narrators to construct comments and questions, underpinning the conversational structure [33]. The availability of a multimodality of resources may provide narrators with more control in the construction and expansion of types of comments and questions. A PVS can be employed as (1) a DU in itself or it can (2) be part of a DU, such as a comment and/or question, which in all cases need to be understood by all interlocutors.
1.3. Conversational Repair
The co-construction of a conversational narrative develops through the mechanism of a clarification of each DU (e.g., comment or question), to establish common ground between interlocutors [8,34]. Evidence of common ground is provided by indications of mutual understanding, such as verbal or nonverbal confirmations or a DU which refers to, or incorporates, the shared information [14]. A narrator’s DU may become a hint if it is not understood by a co-narrator, who may then provide suggestions or guesses [35]. At that moment, a repair framework may be activated, which is referred to by Laakso and Klippi [32] as a hint–guess sequence (HGS). A HGS may end in either a communication breakdown (no mutual understanding) or a confirmation from the (aided) narrator that mutual understanding has been reached. A quick yes/no request for confirmation that a DU has been well (linguistically) understood is defined by Clarke and Brennan [14] as backchanneling.
The PVS’s audiovisual footage may offer narrators more control in this reciprocal process of calibration and fine-tuning, boosting mutual understanding to sustain a conversation.
1.4. Conversational Maintenance
Clark and Brennan [14] described obstacles in grounding in relation to the common principle of “least effort”, which is complicated by time pressures and errors, affecting conversational maintenance. The availability and employment of semiotic resources, each with their own qualities and limitations, influences the interactions of a conversation [7]. As a referent, the PVS, viewed by all interlocutors, can potentially become the focus point of joint attention, assisting in the grounding process [31]. Joint attention is not simply the viewing of the film on a screen, but it implies that narrators have achieved enough overlap to construct a convergent shared space [36]. Evidence of joint attention is observed when narrators employ or refer to the PVS with(in) a DU, followed by a confirmation that mutual understanding has been reached [15]. Emotional (non)verbal expressions such as laughter or encouraging utterances such as “aha”, in combination with the PVS’s observed information, may assist in building a shared space to maintain a conversation.
2. The Aim of the Present Study
Daily social interactions and multimodal conversations entailing topics small and large are essential for building connections and fostering personal, social and multimodal literacy skills [25,27]. The multimodality of resources employed during social interactions between people is the dynamic that enables the construction of meaning [7]. Aided communicators often do not have access to natural speech and compensate for this deficit by adopting the richness of a multitude of technologies and resources such as gestures, sounds, movements, (computerized) spoken/written language, gestures and—with our current research—the audiovisual information of the PVS. The focus is not on the resources as stand-alone communicative acts but how—in concert—these multimodalities influence the dynamics of an aided conversational narrative. Drawing on theories and methods from the fields of AAC, social semiotics and visual anthropology, a method called Film as Observable Communication (FaOC) was developed during the research project “My Film, My Story” [19]. The FaOC method is a practical multimodal literacy tool to assist children and adults with CCNs to create and use their own self-created films, PVSs, as a resource in their daily story-sharing, at school and at home. A multimodal transcription format was developed to document and study a film-elicited conversation.
Recently, we described the theoretical background and the practical application of the FaOC method to create, use and transcribe self-created film media as a resource in daily story-sharing, in a tutorial paper (see [19]). The purpose of the current study is to examine in more detail how self-created film media, a PVS, produced though audiovisual technology by the aided narrators themselves can assist in social interactions, such as a multimodal conversational narrative.
This paper uses a case study methodology to describe the following:
The process of creating and using the PVS in a film elicitation conversation;
The multimodal transcription and coding model used to study this process of aided sign-making and story-sharing, in relation to the four components for exercising agency and control: topic development, conversational structure, conversational repair and conversational maintenance [32,33].
3. Method and Materials
The current case study is part of a larger research project “My Film, My Story”, which took place between 2012 and 2024, in eight special education schools and rehabilitation centers in the Netherlands [16,19]. The developed FaOC method was employed during the project to assist children and young adults with CCNs to produce and use a PVS as a resource for daily multimodal story-sharing about their experiences, at home and at school. More information about the FaOC method including the FaOC website and the “My Film, My Story” project can be found in Appendix A.
In this study, we focus on one of the project participants, named Mike, who created a PVS about his school trip to the Steamship Rotterdam and used it for story-sharing at home. To preserve anonymity, a pseudonym is used in place of the participant’s actual name.
3.1. Participants
Mike is a 17-year-old boy with severe speech and physical impairments resulting from cerebral palsy (bilateral dystonia). He lives at home with his sister and mother, who are both skilled at interpreting Mike’s vocalizations, gestures and facial expressions, which he uses in addition to his high-tech AAC device. Mike attends a special school for children with disabilities. Mike has low muscle tone, experiences problems with motor planning, has limited natural speech and is unable to walk. For the past decade, he has used an electric wheelchair for mobility and a high-tech AAC device for communication (Tellus 4 with Mind Express) mounted on his wheelchair, both of which he controls via a joystick. Mike himself, his school support team and his family indicated that his cognitive abilities were consistent with his chronological age, yet he experiences severe difficulties with his spoken language and literacy. According to his teachers and speech language therapist, Mike possesses greater skills than his traditional school test scores suggest, since his CCN makes it challenging to assess him accurately, a problem acknowledged in prior studies (e.g., refs. [4,37]).
Mike, his parents and his teacher described how daily conversations typically commence, with minimal topic development and repair required to reach mutual understanding. After school, he is typically brief in sharing his experiences, and it is most often his mother who asks (yes/no) questions to gain a better understanding of his day. Mike prefers to make brief comments (2- to 3-word sentences) and loves to make jokes, using his high-tech AAC device, but he prefers natural speech/sounds and gestures/miming. Mike expressed concerns as to whether others recognize his intellect, since he experiences significant difficulties in establishing understanding when interacting with unfamiliar communication partners. He reported that building new relationships, and befriending others, is complicated.
3.2. Ethics
Mike and his co-narrators were briefed before and during the commencement of the project and each video-recorded session. Each narrator was free to cease participation or stop a session if they felt uncomfortable at any time. Mike and his co-narrators had all worked with, and been video-recorded by, the first author during previous encounters and were thus accustomed to her presence and being filmed. All narrators were familiar with the processes of the observation through the distribution of information sheets and organized briefings in person and had access to all recordings and transcripts. In this participatory research project, the participants had the role of co-researchers in the design of the FaOC method. Informed consent was obtained from all participants, including GDPR permissions approved by the Ethical Board of Social Science, Radboud University, Nijmegen, the Netherlands, ECSW2015-0903-301 Legel-Steenbergen.
3.3. Setting and Materials
The FaOC method was employed by Mike to create and use his PVS in story-sharing, taking place in multiple settings. Following a preparatory session at Mike’s school, the FaOC method procedure was implemented according to Grove’s story-sharing model [25], which includes three steps: (1) filming the PVS at the steamship location (story capturing); (2) editing the PVS at school (story preparation) and (3) sharing the PVS at Mike’s home (story-sharing). The FaOC procedure steps are described briefly below, but more extensively in the Results Section. There is no protocol for creating and sharing a PVS, allowing freedom of choice for the (aided) film maker and their co-narrators.
Preparation and Procedures
Before commencing the self-created film process, there was a test–train–try phase to familiarize Mike with the audiovisual technology required to create the PVS, thus ensuring he had full technical access and control. This took place at his school. Mike’s electric wheelchair provided a suitable platform to secure an iPad with a tripod. Using an interface connected to his wheelchair joystick, Mike was able to switch the iPad camera on and off. He used the same joystick, with a Bluetooth interface, to access an Apple MacBook Pro equipped with Final Cut Pro X video editing software version 10.4. During a one-day workshop at school on the FaOC method, Mike practiced filming and editing (see Appendix A).
The procedure consisted of the following three steps:
Filming. During a school trip, Mike visited the Steamship (SS) Rotterdam. He captured his film story elements from his perspective, using his iPad;
Editing: Mike prepared his PVS by selecting and editing his film footage at school;
Story-sharing: Mike shared his experience on the ship during a film elicitation conversation with his mother, a family friend and his sister at home, while they watched the PVS on a TV screen.
3.4. Research Design and Data Collection
The data for this case study were collected through ethnographic fieldwork, including ethnographic filming, fieldnotes/diaries and participant observation, to describe the three steps of the FaOC method for film/story production. The focus of this paper is the conversational narrative. However, an insight into the creation of a PVS is included to demonstrate how the aided film-/sign-maker’s choices during filming, selecting and editing are part of the story-sharing. The film-elicited conversational narrative between Mike and his co-narrators was recorded on video to allow for transcription/coding procedures using the FaOC transcription format [19].
The conversation began in the usual manner for Mike and his family, without the PVS, although the film resource was available from the start. Each narrator could choose to use the PVS by simply starting the film on the screen. The incorporation of the PVS created two distinct phases of the conversation, without and with the PVS. Analyzing the conversation in these two consecutive parts was necessary to document the unfolding dynamics under the same conversational circumstances. This paper does not aim to establish causality, but rather to explore the use of personal film footage as a supporting resource during a conversational narrative.
The first author video-recorded the conversation using a single camera (Sony HDR-FX1) positioned at a wide enough angle to frame both the narrators and Mike’s PVS on the screen. The conversation was then transcribed, coded and analyzed using an Apple MacBook Pro, with Microsoft Excel and Matlab R2020b to generate graphics. The first author’s role as an ethnographic researcher and the research methods deployed are detailed in the Author Contributions section provided at the end of this paper, and in Appendix A.
3.4.1. Transcription
The multimodal FaOC transcription format [17,19] was used to transcribe the film elicitation conversation, which is here briefly summarized as follows (see Appendix A). Two types of video footage were transcribed:
The conversation video-recorded by the researcher.
The PVS created by Mike.
Since moving images remain challenging to include in a transcript, still frames (screenshots) were captured and used. As described earlier, the discourse unit (DU) was selected as the basic unit of transcription and analysis, to display all communicative acts initiated within the context of a conversation, which may consist of single or multifarious resources such as spoken, written, manual signs, finger-/eye-pointing or an image, such as the PVS, that develops or sustains the topic of conversation, assists with conversational repair or facilitates maintenance of the conversational flow (conversational control) [5,33].
Two separate transcripts were prepared from the same conversation, without and with PVS support, each with their own timeline, to make further conversation analysis and comparison possible. The FaOC transcript format, shown in Figure 1, consists of an image timeline, incorporating nine columns. The first, Column 1, is a timeline (min. s), which is activated at the moment one of the narrators makes or adds a verbal or nonverbal DU. Column 2 includes the still frames of the video-recording of the conversation, and Column 3 a still frame of the PVS. These PVS frames are included in the transcript, when one of the narrators includes or refers to the PVS in a DU, such as making it part of a comment or question or by finger-/eye-pointing to the screen. Column 4 identifies who is communicating or the PVS itself.
What is expressed by each narrator is documented as a DU in Column 5. A short description of the PVS frame is also documented in this column. Each DU is articulated with the terminology and notation defined by von Tetzchner and Basil [21]. Since none existed for a PVS, the following symbols are introduced: «a brief description when the PVS is running» and <a brief description of the PVS when it is paused>. Column 6 displays all the communicative resources used by name, e.g., gesture (g) and audiovisual resource (avr). Column 7 includes the coding of the still PVS frame displayed in Column 3. Column 8 includes notes on narrative insights’ (sub)topics. Column 9 displays conversational maintenance, observed in any expression (e.g., laughter), which is explained in more detail in the next paragraph.
The conversation was transcribed by three independent transcribers, one of whom had prior familiarity with Mike’s modes of communication. Differing descriptions or interpretations were discussed until a consensus could be reached, to deliver an agreed transcript. For validation purposes, Mike and his mother were given the opportunity to read and observe the conversation recording and transcripts. They acknowledged appreciation for this approach but had no comments to add to the content.
3.4.2. Coding
After consensus about the conversation transcript was reached between the three raters and agreed upon by Mike and his mother, the transcript was coded. Each DU was coded in relation to the four components of conversational control, according to Müller and Soto’s [33] classification.
Topic development: Topic development (Column 10) was coded as an ongoing synchronous topic versus a newly introduced asynchronous topic. To observe this process and be able to code whether a topic is new or sustained may require some finesse. The moment a novel topic, named an asynchronous topic, is introduced may be clear, but when the topic is discussed in more detail or with related stories (relevant to the topic), it can become open to multiple interpretations. The continuation of the same topic was encoded here as a synchronous subtopic. During transcription, both topics and subtopics were descriptively documented with keywords in Column 8.
In brief, topic development codes are as follows:
A new topic, asynchronous topics (a);
The sustainment of a topic, synchronous subtopics (s);
If non-related topics were discussed, they were coded as off-topic (ot).
Conversational structure: Conversational structure was coded as a question (oblige) or comment, in Column 11. In brief, a question is a request, an obligation (oblige), for an answer or a comment. The separation between questions and comments was typically straightforward to identify, for example, by the presence of a question mark. Still, in some cases, it was more challenging to code: for example, to identify a nonverbal DU (as AAC users often adopt), when a question or comment related to earlier contributions or when a reciprocal comment/question was made. Each question or comment was placed into different categories, as displayed in Column 12. Questions were divided into two categories: (1) a yes/no (closed) question, which elicits a yes/no answer, and (2) an open question, which elicits extended or more detailed answers. Comments can be summarily divided into three categories: (1) a yes/no answer; or (2) a solicited (information) comment, which is solicited by a comment or a (open or yes/no) question, and (3) an unsolicited comment, which is not triggered by a preceding question or comment. A yes/no question and related yes/no answer may look simple yet may have diverse functions, such as (1) a typical yes/no answer out of interest or (2) a quick double check/request for confirmation related to an expressed DU (e.g., comment) or backchanneling (see Introduction).
Conversational structure codes are as follows: question/oblige (O); open question (OQ); yes/no question (YNQ); comment (C); yes/no answer (YNA); solicited (si) or unsolicited (ui) comment.
Conversational repair: Conversational repair, in Column 13, displays the coding of a hint–guess sequence (HGS). The HGS is triggered by a hint from the aided narrator, followed by (multiple) guesses from the co-narrators, and it ends with either a positive confirmation or communication breakdown. In some situations, the confirmation may come after multiple conversational turns (e.g., DUs, sentences, sounds, laughter) later during the story-sharing, for example, when the PVS displays the information required to reach mutual understanding.
The codes of repair, displayed with HGS, are as follows: hint (h), guess (g), communication breakdown (bd) and, most importantly, confirmation (cf).
Conversational maintenance: Last is conversational maintenance in Column 9, which is also discussed in the Transcription Section. Resource use (e.g., spoken or signed language, finger-/eye-pointing, gestures, images) and emotional expressions or filler utterances, such as “oh, aa”, (e.g., smiling, laughter, sounds) may be subtle and open to multiple interpretations. Still, these verbal and nonverbal expressions were included in the documentation, since they can influence the flow and maintenance of the conversation. Often, it is more an emotional sense of positivity or negativity, remaining too subjective to describe or code.
No coding acronyms were utilized, since expressions were written out in full.
PVS code: The PVS was included in the transcript as a DU when one of the narrators incorporated or referred to it with a DU (e.g., comment/question), employing verbal or nonverbal resources [19]. The PVS was coded as an asynchronous (a) topic when its use prompted a new topic or as a synchronous (s) subtopic when the same topic was sustained. The PVS was coded as either an “unsolicited (ui)" or “solicited (si)" comment, depending on how the PVS was utilized. For example, when one of the narrators was browsing through the film for a specific scene to give an answer or comment to a previous comment (C) or question (O), it would be coded as a solicited comment. The PVS was coded as a hint (h) by the aided narrator, followed by the guesses (g) of co-narrators, activating an HGS to reach common ground. For more narrative insight, each PVS still frame was coded, in Column 7, using the visual grammar model of Kress and Van Leeuwen’s [38], such as a person (p), action (a), object (o), details (d) or location (l).
3.5. Data Analysis
The first step in the data analysis, after transcription and coding, was counting each observed DU produced by each narrator, related to the following:
Topic development: the number of asynchronous and synchronous units;
Conversational structure: the number and type of questions (obliges) and comments;
Conversational repair: the number of hints, guesses and confirmations/breakdowns after an HGS;
Maintenance and resource use: the number of resources used by the narrators and instances of emotional expressions.
To gain some insight into the narrative content, a descriptive overview of the (sub)topics of the conversation, including those visible in the PVS, was made feasible. The second step of the analysis was to describe the difference, without and with PVS support, related to the four components of conversation control.
Inter-Rater Reliability
Three trained transcribers/coders (Raters 1, 2 and 3), the first author and two Master’s students in pedagogical/social sciences transcribed and coded the conversation. An additional transcriber/coder (Rater 4), an experienced speech–language pathologist with experience working with children who use AAC, coded the HGS sequences. Coding an HGS is complex, since the content and resource use needs to be understood in detail, both technically and linguistically, as interactions can be very subtle. For example, a hint could be given with a small gesture or a wrongly spelled word.
An independent statistician analyzed the coded conversational narrative, as related to the presence or absence of PVS use. Inter-coder reliability was calculated using Krippendorff’s Alpha [39]. In total, two comparisons were made from the coded transcript, without and with PVS use. The lowest value for KALPHA = 0.97 (see Table 1). The inter-coder reliability for Rater 1 and Rater 4 coding repair is lower than that for the remaining three raters (1, 2 and 3), coding topic development, structure and maintenance, but it remains high.
4. Results
The findings of this qualitative descriptive case study are presented in two sections. The first section provides a detailed description of the three procedural steps of the FaOC method: (1) capturing (filming) story elements, (2) preparing (selecting/editing) the PVS and (3) sharing the story using the PVS. The second section presents the results of the analysis of the film-elicited conversational narrative, with a focus on conversational control, both before and after the deployment of the PVS.
4.1. The Three Steps of Film/Story Production
4.1.1. Step 1: Story Collection and Filming
During the filming, Mike used an iPad to capture the audiovisual details and contextual information of his visit to the SS Rotterdam. He selectively filmed by turning the camera on and off through an interface and a control button connected to the iPad, carefully choosing subject matter and how he wanted to film. While operating the camera, he made specific visual framing choices by maneuvering his wheelchair in reverse or raising and lowering the chair, to capture close-ups and widescreen shots.
Mike filmed in detail the decoration and interior of the rooms of the ship, such as the dining room, ballroom and theater. Over the course of his 8-h visit, Mike captured the moments that most interested him in two hours of film. Even though it was Mike’s first film event, he was meticulous in his choices about what, how and when to film. He demonstrated confidence in the process, methodically capturing those images that mattered to him and his personal narratives, displaying significant discipline.
4.1.2. Step 2: Story Preparation and Film Editing
During the story preparation phase, Mike carefully selected and organized, through film-editing, the narrative elements from his two-hour film footage into a 25 min film, identifying the most significant aspects ((sub)topics) he wanted to share from his perspective. The editing process, which was conducted at school, required four hours of focused attention. Mike reported enjoying this phase of the process, which allowed him an opportunity to reflect and prepare for the upcoming conversation with his family. He showed confidence in his ability to select the scenes that best illustrated his experience. However, while being generally satisfied with his final product, he regretted not filming certain details, such as the lunch or the bridge to the ship, which at the time he thought were not essential for his story.
4.1.3. Step 3: Sharing the Story
The story-sharing took place at Mike’s home, with his mother, sister and a family friend present (i.e., co-narrators). They gathered around a television screen, where they could view the PVS while simultaneously observing each other. The co-narrators were aware of where Mike had been for his experience, as per the consent agreement, but the specific content of the PVS remained unknown to them. The conversation began with Mike introducing the visit to the SS Rotterdam. Mike and his co-narrators were able to start, stop or rewind the PVS at will. Mike would signal when he wanted to start and/or pause the film by raising his arm, as he was unable to operate the remote control by himself. The topics and subtopics observed in the PVS developed rapidly at times, which made it challenging for narrators to pause at a specific frame in order to construct a comment and/or question. Narrators occasionally returned to earlier moments in the film, rewinding to review details for clarification. Communicating during the film’s playback was challenging at times, as moments could be missed and/or narrators could experience an overload of information. The conversation lasted about an hour, which is described in more detail in the following section.
4.2. Film-Elicited Conversational Narrative
To explore the dynamics of the use of a PVS in relation to the four components of conversational control in more detail, the conversational narrative between Mike and his co-narrators was recorded, transcribed and coded. The conversation began without PVS support and continued with PVS support the moment that Mike started to playback his film. The results are illustrated with the support of several tables and figures: (a) DUs coded according to the four components of conversational control are displayed in Table 2, with corresponding graphical sketches in Figure 2; (b) an excerpt of (sub)topics is displayed in Table 3 and Figure 3; and (c) an illustration of multimodal resources adopted by Mike during the conversation is presented in Figure 4. All figures and tables are provided in higher resolution online as an infographic, available at the accompanying website (see Appendix B, Appendix C and Appendix D).
The proportion of DUs devoted to topic development, conversational structure, conversational repair and maintenance, as shown in Table 2, provides an insight into how the conversation unfolded upon use of the PVS. Figure 2, extracted from Table 2, represents topic development (asynchronous/synchronous), conversational structure (question and comment) and conversational repair (hint/guess/confirmation/breakdown), using colored dots or vertices, where the size of each vertex directly corresponds to the number of observations for the respective variable (i.e., more observations result in a correspondingly bigger vertex). Two vertices share an edge if they were observed simultaneously and occurred in the same line of the transcript.
Conversational maintenance was not included in the sketches since verbal and nonverbal emotional expressions are often open to multiple interpretations, still influencing the conversation atmosphere. For this reason, only instances of laughter were included in Table 2.
Table 3 presents an excerpt of the (sub)topics of the conversation. All of the (sub)topics during the conversation were directly or indirectly featured in the PVS. Figure 3 shows coded transcript scenes to describe/display in greater detail the use of multimodal resources and the four components of conversational control. An illustration of the multimodal (AAC) resources Mike used during the conversation is provided in Figure 4.
Each DU was coded according to its relationship with the four components of conversational control. Nevertheless, these components do not function independently, but rather are interconnected and influence one another. In addition, multimodal resource use (e.g., spoken/written/signed language, finger-/eye-pointing and the PVS) create within one single or between multiple DUs so-called multimodal wholes [7]. To represent these interconnections visually, a first draft of the Image–Conversation–Interaction Model (ICIM) was conceptualized, incorporating the PVS as a resource (see Figure 5). The ICIM model draws on the work of Kress [7], Müller and Soto [33] and Laakso and Klippi [32].
In the ICIM, the PVS is positioned in the middle, as an additional resource, becoming one of the building blocks of the conversation, potentially influencing the four components of conversational control. Topic development, conversational structure and conversational repair are placed within conversational maintenance (as an overlay). Conversational maintenance is influenced by the dynamics of the three units and the employment of semiotic resources. In the following sections, we will describe the conversational narrative related to the four components of conversational control, before and after the deployment of the PVS.
4.2.1. Topic Development, Conversational Structure and Repair Without PVS
At the start of the conversation, without PVS support, Mike initiated the topic of his school trip by operating his SGD to type out the words: “SS Rotterdam is super big”. Later, as can be seen in scene 1, shown in Figure 3, at 10.05, Mike responded to an open-ended question from a co-narrator: “What could you see?”, with a solicited typed comment: “Everything from Rotterdam”. The co-narrators followed up with yes/no questions (YNQs) to suggest (sub)topics, such as Hotel New York and the football stadium, The Kuip. About one-third of these questions led to a yes/no answer (YNA) from Mike. These suggestions from the co-narrators were made to elicit further information and create common ground for topic development. While Mike is capable of typing more detailed responses, he was limited to YNAs in this context, particularly due to the challenges with typing speed and grammatical accuracy.
Earlier, in transcript scene 2, at 4.51, Mike incorrectly typed: “Well guys there you had a tatater”, which served as a hint. This prompted the co-narrators to attempt interpreting what Mike meant using YNQs and comments (guesses), thus creating an HGS. The co-narrators asked additional YNQs to assist in the repair and problem solving process. After six guesses, Mike was able to confirm the intended meaning, a theater. However, when the co-narrators inquired whether there was a cinema on the ship or if it was a luxurious boat, the conversation stalled. It might have been that what they wanted to know was unclear for Mike, or that he was unable to describe the situation, leading to a communication breakdown.
4.2.2. Resource Use and Conversational Maintenance Without PVS Use
Figure 4 illustrates that Mike’s multimodal resource use without the PVS was limited, as evidenced by the lower number of DUs in. Although Mike used his high-tech AAC device slightly more frequently (in number of turns) without the PVS, his use of natural speech and gestures (his preferred modes of communication) was very restricted. The conversational breakdowns and relatively long moments of silence without PVS support seemed to influence the conversation maintenance and the overall atmosphere in a negative way. Mike became frustrated when he was unable to describe what he saw on the ship or when his typed words were grammatically incorrect, as demonstrated in the scene “Tatatater” (scene 2, in Figure 3). Additionally, prior to PVS use, all narrators displayed fewer instances of laughter, even though Mike enthusiastically started the conversation with the typed words “I have two hours of film”.
4.2.3. Topic Development, Conversational Structure and Repair with PVS Use
After 14 min of conversation, Mike activated his PVS, making it available as a resource for all narrators. As shown in transcript scene 3, he paused the PVS to initiate a new topic with his SGD: “That is the deck”. “That” directly referred to the PVS, using the audiovisual resource as a referent. The co-narrators used a YNQ to confirm with Mike that they understood him and his (PVS) comment correctly. These YNQs with PVS seemed to function more as backchanneling than repair. Many YNQs were asked, with the equivalent number of YNAs by Mike. At 12.39, when the PVS displayed « Driving on the deck», Mike raised his arm to signal a pause in the film. He then contributed to the image by typing: “I was looking for a spot”. Instead of mentioning only “everything from Rotterdam”, as in scene 1, triggering only YNQs, Mike now used his PVS close-ups, such as of the Euromast and a water taxi, as referents, thereby successfully triggering topic development. Next to the topic of the steamboat, many more (sub)topics were observed in the conversation with the PVS, as illustrated in Table 3, initiated by all narrators. It seems that all topics, directly or indirectly related to the PVS, triggered DUs that allowed narrators to delve into details, thoughts and emotions in greater depth.
4.2.4. Resource Use and Conversational Maintenance with PVS
All narrators used Mike’s PVS in combination with other resources to construct a DU, such as a comment/question, a gesture or finger-/eye-pointing to the screen with the PVS, creating a multimodal whole. This elaboration and enhanced intensity of resource use resulted in approximately double the number of DUs with PVS support produced by each narrator, as illustrated in Figure 2 and Figure 4. Mike also increased his preferred use of gestures and natural speech. The sentences he constructed with his SGD referenced or incorporated information from his PVS. At 20.28 (available to view online), Mike filmed close-ups by lowering and raising his wheelchair to capture the banister of the ballroom stairs in detail, some small fish and a sea monster. Initially, his co-narrators responded with a joke: “Oh are you going to take the stairs?” After Mike highlighted his intentions through close-up imagery and finger-pointing at the screen, they responded with: “Oh, there are all little fish in the stairs”.
The PVS resource seemed to assist Mike in making his DUs clearer and the co-narrators’ guesses/suggestions more accurate, evidenced by the marked increase in HGSs, concluding with a confirmation from Mike. These verbal and nonverbal confirmations seem to indicate that common ground and joint attention had been reached. Each narrator shared more instances of laughter when they were watching the PVS, and each contributed more jokes, indicating enhanced engagement with the story. At 28.12 (online), Mike’s PVS displayed the dining room, where the lights lit up at the moment Mike made a 360-degree recording. His mother commented with the Y/N question: “Did you put the lights on?” Mike indicated surprise, laughing and signing: “NO, ME, HOW?” Then, everyone started to laugh. Even though open questions were limited in frequency during the conversation, Mike’s mother asked Mike what he liked the most. Mike answered using a multimodal combination of semiotic resources: “the interior” (typed/spoken words), and, with a combination of gestures and finger-/eye-pointing, he indicated in his PVS the following subtopics: <lights, stairway banner, walls and decorations>. He explained, again by deploying a diversity of multimodal resources, that the ballroom with the lights, the sea monster in the stair banister and the birds woven in the wall-carpet were his favorite places.
4.3. Summary
At the beginning of the conversation, without PVS support, limited topic development was observed, suggesting that the narrators were restricted to the basics of the experience. Mike introduced the main topic using his SGD, “SS Rotterdam super big”, but his co-narrators prompted further discussion by posing questions and making comments, suggesting (sub)topics. However, Mike could only respond with yes/no answers to his co-narrators’ constructed yes/no questions. He appeared motivated, and initiated comments, in four cases leading to an HGS, of which half led to a breakdown, since the co-narrators seemed to lack the required information to assist Mike in attaining mutual understanding. In brief, Mike made some important contributions before the deployment of the PVS, but there was a tendency towards restricted topic development with fewer DUs, a higher frequency of instances of repair (including communication breakdowns) and limited resource use, influencing the conversation trajectory. Mike had eight DUs in which he used his speech-generating device, without PVS support. He typed 35 clear words, of which 10 were deemed unclear. One DU contained natural speech, although no understandable words could be distinguished. With the available audiovisual information of his PVS, Mike used his SGD in six DUs, with 22 clear words and 0 unclear words. Furthermore, he contributed 18 words using natural speech successfully in combination with the PVS information. As is often observed with aided narrators who depend on AAC, there is limited utility in analyzing differences in the number of spoken or written words. Rather, it is about an evaluation of the whole process and the dynamics between the narrators, focusing on qualitative aspects pertaining to the employment of a diverse mix of multimodal resources to make meaning.
The conversation dynamics changed noticeably with the introduction of the PVS, as might be the case with the addition of any new resource. When Mike and his co-narrators incorporated his PVS into their conversation, there was a marked increase in both new topics, from 15 to 140, and ongoing subtopics, from 79 to 673. Mike’s initiations of a new topic increased from 7 to 11, and the number of ongoing subtopics increased from 13 to 125. All topics during the conversation were related directly or indirectly to the visit to the SS Rotterdam. There were no off-topic exchanges, except for questions of whether anyone wanted coffee.
The PVS, functioning as a prepared audiovisual story timeline by Mike, seemed to assist him in initiating and sustaining the (sub)topics of his personal experience, rather than mostly his co-narrators, as was observed at the start of the conversation without PVS. The PVS seemed to assist Mike in employing a more diverse multimodality of semiotic resources, to construct 262 coded DUs, compared to the 33 DUs without PVS support. Also, his SGD-typed sentences were longer, in combination with other verbal and nonverbal resources (e.g., gestures, finger-/eye-pointing, speech) and the audiovisual information of the PVS. This increase in Mike’s DUs led to more HGSs than without PVS support, with none resulting in a conversation breakdown. Both Mike and his co-narrators employed Mike’s self-prepared PVS to construct comments and questions, to reach mutual understanding and to initiate and sustain topics. The audiovisual information, in combination with all other resources (eye-pointing, gestures and computerized spoken/written language), merged into multimodal wholes. Co-narrators checked whether they understood Mike and his PVS well by repeating his DUs in spoken words, which Mike could then confirm, as shown in the transcript scenes provided (Figure 3).
The marked increase in YNQs from co-narrators, with PVS support, was followed by a corresponding increase in YNAs by Mike. Still, these YNQs from co-narrators were prompts for quick confirmation as to whether they had understood Mike correctly (backchanneling). As a referent, the PVS seemed to facilitate the establishment of common ground and joint attention between narrators, as observed in (1) its incorporation into, or use as, a DU and (2) the required confirmation by Mike. Utilizing his PVS, Mike demonstrated more frequent use of his preferred natural speech and eye-/finger-pointing to construct DUs more successfully; since again there were no communication breakdowns, the conversation was sustained for an hour, with more enjoyment.
In brief, the deconstruction of the conversational dynamics and the multimodal whole of resources into separate units, for example, to count spoken words, is not the intended goal, since narrators with CCNs have little or no spoken words. That is why the focus of this study is upon the broader dynamics of the conversational flow. The strength of multimodality in AAC is the synergy that results from combining diverse technologies and resources together into a multimodal whole, which makes each individual resource stronger in combination than when used in isolation. In this case study, the coding model seemed to assist the capture of the conversation dynamics, first in the absence of Mike’s self-created PVS and then with PVS support, which we interpret as promising for a novel AAC approach—and which we hope to explore further in future research, as visualized below in Figure 6.
5. Discussion
This paper describes a case study that introduces a multimodal coding model, developed to display and study the employment of self-created film to use as one of the resources during the social interaction of a conversational narrative. In this case study, Mike shared his experience of a school trip to a steamship using his self-created film, a PVS, following the three steps of the FaOC method [19]. This study is not intended to assess Mike’s communicative skills, but instead focuses on the social interaction and co-construction of the conversational narrative that unfolded between him and his co-narrators, as reflected in the conversation transcript. The goal was to explore whether self-created film, a PVS, produced through assistive audiovisual technology, can assist narrators with CCNs in achieving greater conversational control, as evidenced by the metrics gathered of topic development, conversational structure, repair and maintenance [7,32]. To describe the dynamics between the four components with the PVS, a first draft of the ICIM was developed (Figure 5), referencing the work of Müller and Soto [33]. The PVS is at the center of the ICIM, since the film media resource appears to interact with all four components of conversational control. Underpinned by other studies, the PVS seems to function in three ways: (1) operating as a DU; (2) integrating within a DU and (3) prompting and assisting another DU [7,14]. In this case study, the deployment of a PVS resulted in (1) the expanded adoption of semiotic resources, such as finger-/eye-pointing, natural speech, computerized typed sentences and manual signing, creating multimodal wholes [7], (2) which appeared to boost DU generation (e.g., comments and questions) [21], (3) which, in turn, seemed to assist in making the DUs more clear, as evidenced by fewer incidents of repair required to reach common ground/joint attention [8,15], (4) boosting conversational maintenance and (5) encouraging topic growth [1,23].
Mike’s PVS appeared to facilitate a more thorough exploration of topics, as its film frames triggered (sub)topics, resulting in many yes/no questions by the co-narrators. However, with the PVS available, the (sub)topics were prompted by Mike and his PVS, instead of just topic suggestions via yes/no questions from his co-narrators, as observed in the absence of PVS support. This highlights an interesting point for further research: it seems that yes/no questions from co-narrators have at least three distinct functions: (1) to ask a closed question: “Did you eat chips?”; (2) to initiate a (sub)topic: “Did you see the football stadium?” and (3) to request timely confirmation, following a (PVS) comment from the aided narrator: “Did you say/mean (that) bird?” This last type of yes/no question is the opposite of the first and second, since the aided narrator, Mike, now introduces the (sub)topics (via PVS), with the co-narrators asking yes/no questions for quick confirmation to confirm mutual understanding, namely, backchanneling [14].
An HGS was initiated in the absence of a prompt confirmation, a phenomenon observed both without and with the PVS. With the availability of the PVS, there was a marked increase in DUs, resulting in many more HGSs, but with none resulting in breakdowns. This suggests, in the case of Mike, that the use of a PVS might assist in reaching joint attention and common ground. The ability to repair communication breakdowns is identified by Martinsen and Von Tetzchner [40] as a foundational communication strategy that both individuals with CCNs and their conversation partners should possess [41], alongside the maintenance of communication, turn taking and the ability to change topics. However, it is observed that narrators seldom spend time repairing such breakdowns, since this can involve a considerable amount of effort [8,23]. The present study shows that the PVS may provide conversation partners with enough detailed or contextual information to overcome breakdowns more easily and quickly, as it might boost the combined use of multiple resources and strategies by all participants in the conversation. Using the PVS improved interactions and facilitated understanding of what Mike wanted to share, according to his account and that of his co-narrators. This suggests that the PVS, produced through audiovisual technology, provides richer insights into the experiences and interests of the aided narrator, as evidenced by Mike’s documenting of interiors. Still, further research, with a larger cohort of participants, is warranted to corroborate these modest insights and to investigate how a PVS in a conversational narrative can lead to changes in communication resources, topic development, conversational structure, repair and maintenance.
5.1. Educational and Clinical Implications
The PVS is not an AAC device, tool or strategy. Rather, it serves as a content and context resource that can be used in conjunction with assistive devices and strategies to construct a hybrid multimodal whole, multiplying the potential of AAC solutions. To ensure the benefits of multimodality are available to people with CCNs, it seems essential to provide them with access to a constellation of AAC assistive technology, resources and strategies, including a PVS, as part of a holistic approach for supporting each narrator in private, school or care settings [11,42]. Based on the findings of the present study, we hope schools and practitioners will be motivated to expand their multimodal literacy and AAC programs and methods by incorporating self-created audiovisual materials created by aided film-/sign-makers [27,28]. The FaOC method can be used by AAC users themselves, parents, teachers and speech–language therapists as a tool to create, use and explore self-created film as a resource in story-sharing [19]. Also, the transcription-coding format and ICIM remain under development but may assist in understanding dynamics during film-elicited conversations.
5.2. Limitations and Future Directions
Some limitations in this paper should be recognized, closely related to future directions for this research. First, the findings are based upon a case study and are therefore not robustly generalizable. Second, the involvement of the first author as the ethnographic researcher could be seen as an influencing factor of the research design and outcomes, since this participant observation method might influence the process and outcomes. However, given the exploratory and descriptive nature of the present study, this method also offers the opportunity to glean close insights and build trust in the participants, to explore their creation and resource use in sharing a personal experience. Third, the present paper focuses predominantly on the third step in sign production, multimodal story-sharing, even though decision-making in the creation of the PVS influences subsequent conversations. Fourth, transcribing and coding a complete conversation of approximately one hour’s duration elicited a large volume of data, but details may still be missed. Finally, fifth, the approach of commencing the story-sharing in a conversation sans PVS support and subsequently introducing the PVS upon one of the narrators’ requests has disadvantages, for example, not accounting for extraneous influences from the first to the second phase of the conversation. However, maintaining the same conversation setting on the same day, with the same conversational partners, presented a chance to observe the dynamics of a conversation including the PVS as one of the resources. All narrators confirmed that the conversation without PVS support was typical. The joint viewing of the PVS seemed to enhance the conversation, generating more topics, more DUs and more laughter and supporting a longer duration. Yet, duration remains open for discussion, since watching an entire PVS and making oneself understood via the use of AAC takes time.
Some issues arose related to the transcription-coding processes. First, although the goal was to record any semiotic resource, the complexity of the aided narrators’ use of paralinguistic communication modalities could not always be registered fully on video. This relates to the paucity of publicly available dynamic transcription formats (with moving film) to date, which could provide a complete insight into the multimodality of the conversation. Since the PVS is only transcribed when it prompts or forms part of a DU, a question or comment, it is possible that other uses of the PVS are not included in the transcription. This choice was made to describe the use of the PVS as a conscious (deliberate) action such as employing it as a referent and/or a point of joint attention. This links to the theories of social semiotics and visual anthropology, where a photograph or video is acknowledged as never standing alone but rather is accompanied by a description, employing supplementary resources such as natural/synthetic speech and signed and/or written words.
Second, coding new topics (asynchronous) and ongoing subtopics (synchronous) was one of the challenges, similar to those identified in Müller and Soto’s (2002) study. The raters indicated this category to be the most challenging because of nuanced interpretations, even though inter-rater consensus remained high. The excerpt with the descriptive (sub)topics displays some more insights but warrants further study. Also, other questions were triggered, such as “How to code rhetorical questions?” or “How to code backchanneling?”.
Third, our FaOC transcript-coding model is an exploratory concept and requires further research related to PVS narrative content. The transcript scenes with still frames document these dynamics but are limited and therefore also warrant more study.
Fourth, with the introduction of a first draft of the ICIM, we hope to assist other researchers in further exploration of the dynamics of self-created film as a resource in daily multimodal conversations or other image resources, but further fine-tuning is required.
Lastly, fifth, the focus of this paper was to explore whether self-created film, produced with today’s available audiovisual technologies, could assist a narrator with CCNs in gaining greater conversational control, rather than the merits of the PVS as a film, although the latter remains an interesting topic worthy of further investigation.
6. Conclusions
Today, anyone can capture moments from their perspective with relative ease, utilizing the audiovisual technologies available on their phones and/or tablets for use as a resource in their daily story-telling, for example, at home about a holiday or football match or at work about a complex technical procedure. This in itself is not novel, since most people do these things daily, young and older, but often subconsciously. For persons with compromised communication abilities who struggle with naturally spoken or written speech, (self-created) film may be an especially useful resource in this context, as shown in several studies in prior research [8,28]. In visual anthropology and social semiotics, images are incorporated into theories and methods to cross language barriers during social interactions between people, to make, share and understand meaning [6,38].
Narrators create, select and use the most suitable resources to make meaning and, through social interactions, share their thoughts, ideas and experiences, identified as essential in film/sign production and story-sharing [7,23]. Today’s audiovisual and assistive technology delivers the opportunity for aided narrators to create their own personal (film) resource, a PVS, instead of using and depending on resources prepared by others, as is often the case with many AAC solutions and technologies. The PVS’s contextual and detailed documentation of a past moment, such as the locations, persons, objects and/or actions, captured from the aided narrator’s perspective, can be made available as one of the resources during a multimodal conversational narrative. Promoting self-created film as a resource in daily face-to-face conversations, for example, in multimodal literacy and AAC activities, requires a better understanding of how (self-created) film functions in a social interaction as “visual words”. This paper hopes to deliver a “proof of concept” to assist in further research related to the potential of self-created film as a resource in daily conversations by a diversity of film-/sign-makers, but especially for persons who have little or no natural speech, whose stories have often been unheard or unseen.
In the present study, the aided narrator, Mike, appears empowered to convey his messages more successfully in the presence of his PVS, and his co-narrators are observed to respond with more accurate guesses, as shown in the positives resulting from faster confirmations. The PVS seemed to encourage the adoption of augmenting resources such as gestures and finger-/eye-pointing but also more linguistic, physically or technically challenging resources such as natural and computerized speech, signing and written language. This is consistent with the work of Kress [7], Legel et al. [19] and Von Tetzchner [21] for promoting multimodal resource use, to provide narrators with CCNs greater conversation control. The more clearly rendered comments with PVS support, requiring fewer instances of repair and more topic development, relate to the common principle of “least effort”. Given that a PVS is individually captured and produced subjectively by the aided narrator, as a film-/sign-maker, to underpin their multimodal conversations and on the foregoing modest evidence of this case study, it seems reasonable to conclude that self-created film provides them with opportunities for more enhanced conversational control.
The present study was conducted as part of the first author’s doctoral dissertation. M.L. initiated and conceptualized the FaOC method and coding model. The co-authors assisted with ideas, knowledge and/or writing. Conceptualization, M.L., G.S., S.R.J.M.D., A.W., N.G., H.v.B., R.S. and B.S.; methodology, M.L. and B.S.; software, M.L. and C.S.N.; validation, M.L., B.S., H.v.B., R.S. and S.R.J.M.D.; investigation, M.L. and B.S.; resources, M.L. and B.S.; data curation, M.L. and B.S.; writing—original draft preparation, M.L.; writing—review and editing, M.L., B.S., G.S., N.G., A.W., H.v.B., R.S., S.R.J.M.D. and C.S.N.; visualization, M.L.; supervision, B.S.; project administration, M.L. and B.S.; funding acquisition, M.L. and B.S. All authors have read and agreed to the published version of the manuscript.
Ethics procedures have been adhered to, the present study being approved by the independent Ethical Board of Social Science, Radboud University, Nijmegen, the Netherlands, ECSW2015-0903-301 Legel-Steenbergen. This work has been conducted as part of the first author’s doctoral dissertation.
Informed consent was obtained from all persons involved in the study.
Any data presented in this study are available upon request. The information is available in higher resolution at
The present study was conducted as part of the first author’s doctoral dissertation. M. Legel initiated and conceptualized the FaOC method and is this paper’s first author. The co-authors assisted with ideas, knowledge and/or writing. We wish to thank Mike and his family for their kind and valuable collaboration during the project.
Our results are relevant for people with complex communication needs, researchers and clinicians in the field of AAC. By means of this statement, we would also like to confirm that the manuscript has not been published previously; that it is not under consideration for publication elsewhere; that it will not be published elsewhere in the same format without the written consent of the copyright holder(s) and that publication is approved by all authors. The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses or interpretation of data; in the writing of the manuscript or in the decision to publish the results.
The following abbreviations are used in this manuscript:
| (a) | asynchronous topics |
| (bd) | communication breakdown |
| AAC | augmentative and alternative communication |
| (C) | comment |
| CCN | complex communication need |
| (cf) | confirmation |
| DU | discourse unit |
| FaOC | Film as Observable Communication |
| (g) | guess |
| GDPR | General Data Protection Regulation |
| (h) | hint |
| HGS | hint–guess sequence |
| ICIM | Image–Conversation–Interaction Model |
| PVS | personal-video-scene |
| (O) | question/oblige |
| (OQ) | open question |
| (ot) | off-topic |
| (s) | synchronous subtopics |
| SGD | speech-generating device |
| (si) | solicited |
| (ui) | unsolicited information (comment) |
| (YNA) | yes/no answer |
| (YNQ) | yes/no question |
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Enlarge this image.Figure 6. Aided film-/sign-makers filming and sharing their stories utilizing assistive audiovisual technology.
Inter-rater reliability topic development, conversational structure, HGS.
| Rater 1 vs. Rater 2 vs. Rater 3 | ||||||
| General analysis | KALPHA | 95%CI | 95%CI | N | N | N |
| LB | UB | (units) | (observers) | (pairs) | ||
| Mike without PVS | 0.9865 | 0.9783 | 0.9933 | 23 | 3 | 69 |
| Mike with PVS | 0.9816 | 0.9724 | 0.9894 | 24 | 3 | 72 |
| Hint–guess sequence | ||||||
| Rater 1 vs. Rater 4 | ||||||
| General analysis | KALPHA | 95%CI | 95%CI | N | N | N |
| LB | UB | (units) | (observers) | (pairs) | ||
| Mike without PVS | 0.9760 | 0.9281 | 1 | 4 | 2 | 4 |
| Mike with PVS | 0.9671 | 0.9146 | 1 | 6 | 2 | 6 |
Four components of conversational control without and with PVS.
| Conversational Narrative | Mike | |
|---|---|---|
| Without PVS | With PVS | |
| Conversation time (min, s) | 14.20 | 37.47 |
| Employing PVS | 100 | |
| Coded DUs (Mike) | 33 | 262 |
| Coded DUs (co-narrators) | 83 | 679 |
| Asynchronous (Mike) | 7 | 11 |
| Asynchronous (co-narrators) | 8 | 129 |
| Synchronous (Mike) | 13 | 125 |
| Synchronous (co-narrators) | 66 | 548 |
| Yes/no question | 38 | 106 |
| Open question | 15 | 21 |
| Solicited information (Mike) | 8 | 27 |
| Solicited information (co-narrators) | 18 | 423 |
| Unsolicited information (Mike) | 0 | 7 |
| Unsolicited information (co-narrators) | 1 | 116 |
| Yes/no answer (Mike) | 12 | 101 |
| Yes/no answer (co-narrators) | 1 | 10 |
| HGS confirmation | 2 | 10 |
| HGS breakdown | 2 | 0 |
| Laughing (Mike) | 0 | 18 |
| Laughing (co-narrators) | 2 | 5 |
Excerpt of conversational narrative (sub)topics.
| (Sub)Topics | ||
|---|---|---|
| Without | ||
| The steamboat | Film, theater, view, lunch, Rotterdam | |
| With | ||
| School trip | School bus, bus driver, classmates, old ship | |
| The steamboat | In the boat itself, the size of the boat, the floors, | |
| Halls and rooms | The hall, room, luxurious halls, library, | |
| Making a film | Eye-level film, “seen from your eyes” | |
| Elevator | Elevator doors, mirror | |
| Deck (inside and | Spot on the deck, view, railing, front ship, | |
| Cabinets and things, | Lamps and glass ceiling, details of the room | |
| Murals | Murals with letters, squares, prepared for the | |
| Guide | Queen’s visit, cost of decorations | |
| Elevator 2 | Elevator fit and measure, collision with elevator | |
| Ballroom hall | Dancing, bar, walls, murals, drawings/images, | |
| Party room disco | Murals, different ceiling, sound proof | |
| Stairs with bird images | Detailed birds, fish on the banister, zoom | |
| Different rooms, | Lamps, modern room | |
| Restaurant | Table, chairs, retro style, hall paintings, | |
| Display cabinet | SS Rotterdam ship and all details | |
Appendix A
The FaOC method and the related transcription-coding procedure were developed during the “My Film, My Story” project, which took place between 2012 and 2024, in collaboration with people who use AAC, their families, teachers and speech–language therapists (SLTs). The method can be utilized by the target group themselves, their families, teachers and SLTs in both school and home settings. The project culminated in 2018, in the film production house Cam on Wheels (in Dutch Com in Beeld,
Appendix B
The paper is related to the FaOC method website:
Appendix C
Transcription scenes and abbreviations (
Appendix D
The infographic is available to view at the following URL:
References
1. Norrick, N.R. Conversational Narrative; John Benjamins: Amsterdam, The Netherlands, 2000.
2. Ochs, E.; Capps, L. Living Narrative: Creating Lives in Everyday Storytelling; Harvard University Press: Cambridge, MA, USA, 2009.
3. Buckingham, D. Media Education: Literacy, Learning and Contemporary Culture; John Wiley & Sons: Hoboken, NJ, USA, 2013.
4. Smith, M.M.; Murray, J. The Silent Partner? Language, Interaction and Aided Communication; J&R Press Limited: Havant, UK, 2016.
5. Light, J.; Collier, B.; Parnes, P. Communicative interaction between young nonspeaking physically disabled children and their primary caregivers: Part II—Communicative function. Augment. Altern. Commun.; 1985; 1, pp. 98-107. [DOI: https://dx.doi.org/10.1080/07434618512331273591]
6. Pink, S. Mobilising visual ethnography: Making routes, making place and making images. Forum Qual. Sozialforschung/Forum Qual. Soc. Res.; 2008; 9, [DOI: https://dx.doi.org/10.17169/fqs-9.3.1166]
7. Kress, G. Semiotic work: Applied linguistics and a social semiotic account of multimodality. Aila Rev.; 2015; 28, pp. 49-71. [DOI: https://dx.doi.org/10.1075/aila.28.03kre]
8. Ibrahim, S.; Clarke, M.; Vasalou, A.; Bezemer, J. Common ground in AAC: How children who use AAC and teaching staff shape interaction in the multimodal classroom. Augment. Altern. Commun.; 2024; 40, pp. 74-85. [DOI: https://dx.doi.org/10.1080/07434618.2023.2283853]
9. Burn, A.; Durran, J. Making The Moving Image: Teaching and Learning With Digital Video Production. Learning to Teach Using ICT in the Secondary School; Routledge: London, UK, 2013; pp. 146-160.
10. Nijland, D. Schaduwen en Werkelijkheid. Cultureel Bepaald Kennen, Waarnemen, Nonverbaal Communiceren, en Onderzoek Daarvan via de Terugvertoning van de Etnografische Film ‘Tobelo Marriage’. Ph.D. Thesis; University of Leiden: Leiden, The Netherlands, 1989.
11. Li, B.Y.; Ho, R.T.H. Unveiling the unspeakable: Integrating video elicitation focus group interviews and participatory video in an action research project on dementia care development. Int. J. Qual. Methods; 2019; 18, 1609406919830561. [DOI: https://dx.doi.org/10.1177/1609406919830561]
12. O’Brien, D. Visual Research with Young d/Deaf people-An Investigation of the Transitional Experiences of d/Deaf Young People from Mainstream Schools using Auto-driven Photo-elicitation Interviews. Grad. J. Soc. Sci.; 2013; 10, 152.
13. Clark, C.D. The autodriven interview: A photographic viewfinder into children’s experience. Vis. Stud.; 1999; 14, pp. 39-50. [DOI: https://dx.doi.org/10.1080/14725869908583801]
14. Clark, H. Grounding in communication. Perspectives on Socially Shared Cognition/American Psychological Association; American Psychological Association: Washington, DC, USA, 1991.
15. Clarke, M. Eye-pointing, joint attention, interpersonal interaction and language development in children provided with AAC. Proceedings of the Research Symposium of International Society for Augmentative and Alternative Communication. International Society for Augmentative and Alternative Communication (ISAAC); Lisboa, Portugal, 25–26 July 2014; Volume 16.
16. Harper, D. Visual ethnography. Visual Sociology; Routledge: London, UK, 2023; pp. 50-95.
17. Sitter, K.C. Participatory video analysis in disability research. Disabil. Soc.; 2015; 30, pp. 910-923. [DOI: https://dx.doi.org/10.1080/09687599.2015.1057319]
18. Chalfen, R.; Sherman, L.; Rich, M. VIA’s visual voices: The awareness of a dedicated audience for voices in patient video narratives. Vis. Stud.; 2010; 25, pp. 201-209. [DOI: https://dx.doi.org/10.1080/1472586X.2010.523271]
19. Legel, M.; Soto, G.; Grove, N.; Waller, A.; Deckers, S.; van Balkom, H.; Spanjers, R.; Norrie, C.; Steenbergen, B. A tutorial: Self-created film as a semiotic resource in AAC. Augment. Altern. Commun.; 2024; pp. 1-12. [DOI: https://dx.doi.org/10.1080/07434618.2024.2387573]
20. Higginbotham, D.J.; Engelke, C.R. A primer for doing talk-in-interaction research in augmentative and alternative communication. Augment. Altern. Commun.; 2013; 29, pp. 3-19. [DOI: https://dx.doi.org/10.3109/07434618.2013.767556] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23521349]
21. von Tetzchner, S. The semiotics of aided language development. Cogn. Dev.; 2015; 36, pp. 180-190. [DOI: https://dx.doi.org/10.1016/j.cogdev.2015.09.009]
22. Blackstone, S.; Light, J.; Beukelman, D.; Shane, H. Visual scene displays. Augment. Commun. News; 2004; 16, pp. 1-16.
23. Higginbotham, D.J. In-person interaction in AAC: New perspectives on utterances, multimodality, timing, and device design. Perspect. Augment. Altern. Commun.; 2009; 18, pp. 154-160. [DOI: https://dx.doi.org/10.1044/aac18.4.154]
24. Higginbotham, D.J.; Caves, K. AAC performance and usability issues: The effect of AAC technology on the communicative process. Assist. Technol.; 2002; 14, pp. 45-57. [DOI: https://dx.doi.org/10.1080/10400435.2002.10132054]
25. Grove, N. Storytelling, Special Needs and Disabilities: Practical Approaches for Children and Adults; Routledge: London, UK, 2021.
26. Grace, E.; Raghavendra, P. Cross-age peer E-mentoring to support social media use: A new focus for intervention research. Commun. Disord. Q.; 2019; 40, pp. 167-175. [DOI: https://dx.doi.org/10.1177/1525740118760216]
27. Lim, F.V. A design-based research approach to the teaching and learning of multiliteracies. Asia-Pac. Educ. Res.; 2023; 32, pp. 641-653. [DOI: https://dx.doi.org/10.1007/s40299-022-00683-0]
28. Zhu, D.; Al Mahmud, A.; Liu, W.; Wang, D. Digital Storytelling for People with Cognitive Impairment Using Available Mobile Apps: Systematic Search in App Stores and Content Analysis. JMIR Aging; 2024; 7, e64525. [DOI: https://dx.doi.org/10.2196/64525]
29. Babb, S.; McNaughton, D.; Light, J.; Caron, J.; Wydner, K.; Jung, S. Using AAC video visual scene displays to increase participation and communication within a volunteer activity for adolescents with complex communication needs. Augment. Altern. Commun.; 2020; 36, pp. 31-42. [DOI: https://dx.doi.org/10.1080/07434618.2020.1737966]
30. Waller, A. Telling tales: Unlocking the potential of AAC technologies. Int. J. Lang. Commun. Disord.; 2019; 54, pp. 159-169. [DOI: https://dx.doi.org/10.1111/1460-6984.12449]
31. Fulcher-Rood, K.; Higginbotham, J. Interacting with persons who have ALS: Time, media, modality, and collaboration via speech-generating devices. Top. Lang. Disord.; 2019; 39, pp. 370-388. [DOI: https://dx.doi.org/10.1097/TLD.0000000000000195]
32. Laakso, M.; Klippi, A. A closer look at the’hint and guess’ sequences in aphasic conversation. Aphasiology; 1999; 13, pp. 345-363. [DOI: https://dx.doi.org/10.1080/026870399402136]
33. Müller, E.; Soto, G. Conversation patterns of three adults using aided speech: Variations across partners. Augment. Altern. Commun.; 2002; 18, pp. 77-90. [DOI: https://dx.doi.org/10.1080/07434610212331281181]
34. Stalnaker, R. Common ground. Linguist. Philos.; 2002; 25, pp. 701-721. [DOI: https://dx.doi.org/10.1023/A:1020867916902]
35. Soto, G.; Clarke, M.T.; Nelson, K.; Starowicz, R.; Savaldi-Harussi, G. Recast type, repair, and acquisition in AAC mediated interaction. J. Child Lang.; 2020; 47, pp. 250-264. [DOI: https://dx.doi.org/10.1017/S0305000919000436]
36. Sperber, D.; Wilson, D. Beyond speaker’s meaning. CRoatian J. Philos.; 2015; 15, pp. 117-149.
37. Norrie, C.S.; Deckers, S.R.; Radstaake, M.; van Balkom, H. A Narrative Review of the Sociotechnical Landscape and Potential of Computer-Assisted Dynamic Assessment for Children with Communication Support Needs. Multimodal Technol. Interact.; 2024; 8, 38. [DOI: https://dx.doi.org/10.3390/mti8050038]
38. Kress, G.; Van Leeuwen, T. Reading Images: The Grammar of Visual Design; Routledge: London, UK, 2020.
39. Hayes, A.F.; Krippendorff, K. Answering the call for a standard reliability measure for coding data. Commun. Methods Meas.; 2007; 1, pp. 77-89. [DOI: https://dx.doi.org/10.1080/19312450709336664]
40. Martinsen, H.; von Tetzchner, S. Situating augmentative and alternative communication intervention. Augmentative and Alternative Communication. European Perspectives; Wiley: Hoboken, NJ, USA, 1996; pp. 37-48.
41. Midtlin, H.S.; Næss, K.A.B.; Taxt, T.; Karlsen, A.V. What communication strategies do AAC users want their communication partners to use? A preliminary study. Disabil. Rehabil.; 2015; 37, pp. 1260-1267. [DOI: https://dx.doi.org/10.3109/09638288.2014.961659]
42. Norrie, C.S.; Waller, A.; Hannah, E.F. Establishing context: AAC device adoption and support in a special-education setting. ACM Trans. Comput.-Hum. Interact. (TOCHI); 2021; 28, pp. 1-30. [DOI: https://dx.doi.org/10.1145/3446205]
43. MacDougall, D. The Corporeal Image: Film, Ethnography, and the Senses; Princeton University Press: Princeton, NJ, USA, 2005.
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Stijn R J M Deckers 2
; Soto, Gloria 3
; Grove, Nicola 4
; Waller, Annalu 5
; Hans van Balkom 6
; Spanjers, Ronald 7 ; Norrie, Christopher S 8
; Steenbergen, Bert 9
1 Behavioural Science Institute, Radboud University, 6525 GD Nijmegen, The Netherlands;
2 Stichting Milo, 5482 JH Schijndel, The Netherlands;
3 Department of Speech, Language and Hearing Sciences, San Francisco State University, San Francisco, CA 94132, USA;
4 Rix Centre, University of East London, London E16 2RD, UK;
5 Assistive Technology and AAC, University of Dundee, Dundee DD1 4HB, UK;
6 Behavioural Science Institute, Radboud University, 6525 GD Nijmegen, The Netherlands;
7 Roessingh Rehabilitation, 7522 AH Enschede, The Netherlands;
8 Stichting OOK-OC, Research Center for AAC, 5482 JH Schijndel, The Netherlands;
9 Behavioural Science Institute, Radboud University, 6525 GD Nijmegen, The Netherlands;