Failure to modulate reward prediction errors in declarative learning with theta (6 Hz) frequency transcranial alternating current stimulation

Abstract

Recent evidence suggests that reward prediction errors (RPEs) play an important role in declarative learning, but its neurophysiological mechanism remains unclear. Here, we tested the hypothesis that RPEs modulate declarative learning via theta-frequency oscillations, which have been related to memory encoding in prior work. For that purpose, we examined the interaction between RPE and transcranial Alternating Current Stimulation (tACS) in declarative learning. Using a between-subject (real versus sham stimulation group), single-blind stimulation design, 76 participants learned 60 Dutch-Swahili word pairs, while theta-frequency (6 Hz) tACS was administered over the medial frontal cortex (MFC). Previous studies have implicated MFC in memory encoding. We replicated our previous finding of signed RPEs (SRPEs) boosting declarative learning; with larger and more positive RPEs enhancing memory performance. However, tACS failed to modulate the SRPE effect in declarative learning and did not affect memory performance. Bayesian statistics supported evidence for an absence of effect. Our study confirms a role of RPE in declarative learning, but also calls for standardized procedures in transcranial electrical stimulation.

About the Authors:

Kate Ergo

Roles Conceptualization, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation: Department of Experimental Psychology, Ghent University, Ghent, Belgium

ORCID logo https://orcid.org/0000-0003-3714-0325

Esther De Loof

Roles Conceptualization, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing

Affiliation: Department of Experimental Psychology, Ghent University, Ghent, Belgium

Gillian Debra

Roles Conceptualization, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft

Affiliation: Department of Experimental Psychology, Ghent University, Ghent, Belgium

Bernhard Pastötter

Roles Formal analysis, Writing – original draft, Writing – review & editing

Affiliation: Department of Psychology, University of Trier, Trier, Germany

ORCID logo https://orcid.org/0000-0001-7364-4702

Tom Verguts

Roles Conceptualization, Investigation, Methodology, Resources, Supervision, Writing – original draft, Writing – review & editing

Affiliation: Department of Experimental Psychology, Ghent University, Ghent, Belgium

Introduction

Declarative memory consists of memory for facts and events that can be consciously recalled [1, 2]. Memoranda are learned rapidly, often after a single exposure [3]. The process of acquiring such memories is called declarative learning. Declarative memory differs from procedural memory, where a skill is learned slowly and by means of repeated practice (e.g., learning how to drive a car). Research has firmly established that prediction errors modulate declarative memory [4], just like they do in procedural memory [5]. Recent research shows that reward prediction errors (RPE; i.e., mismatches between reward outcome and reward prediction) specifically may facilitate memory formation. RPEs were primarily studied within procedural learning (e.g., [6]). However, recent evidence suggests that RPEs are crucial for declarative learning as well [7–9].

One robust experimental paradigm to test this RPE effect on declarative memory, was proposed in [10]. Here, a variable-choice experimental paradigm was used where participants learned Dutch-Swahili word pairs. On each trial, participants were presented with one Dutch word and four Swahili translations. By fixing a priori the number of eligible Swahili translations and whether a choice was rewarded or not, each trial was associated with a different RPE. As a consequence, participants did not learn the actual Swahili translations for the Dutch words. This manipulation allowed verifying whether declarative learning was driven by unsigned RPEs (URPE; signifying that the outcome is different than expected) or instead by signed RPEs (SRPE; indicating that the outcome is better or worse than expected). If URPEs boost declarative learning, recognition of word pairs should be enhanced for large positive and large negative RPE values, exhibiting a U-shaped effect of RPE on memory. Instead, if SRPEs drive declarative learning, recognition should be increased only for large, positive RPEs. The data revealed a SRPE effect. Larger and more positive RPEs during study improved subsequent declarative memory during testing. The effect of RPEs in this experimental paradigm was further substantiated in a follow-up EEG study, where oscillatory signatures at reward feedback were detected in the theta (4–8 Hz), high-beta (20–30 Hz) and high-alpha (10–15 Hz) frequency ranges, suggesting the experience of RPEs by the participants [11]. Further validation came from an fMRI study using a similar paradigm in which famous faces were associated with Swahili village names [12]. This study revealed that RPE responses in the ventral striatum (VS) at reward feedback predicted memory performance. These findings lend further support to the notion that RPE is a key factor in the formation of new declarative memories, and that RPEs are characterized by distinctive neural signatures.

It remains unclear, however, how RPEs boost declarative memory. It is well established that RPEs are encoded by dopaminergic neurons in the midbrain (i.e., ventral tegmental area and substantia nigra) [5]. These neurons change their firing rate in relation to RPEs. From the midbrain, RPEs are projected to several other subcortical and cortical brain regions, such as the VS [13], the hippocampus (HC) [14], and the medial frontal cortex (MFC) [15]. Within these brain structures, dopamine release functions as a neuromodulatory signal. One potential neuromodulatory influence of dopamine occurs via modulating neural oscillations in a wide range of frequency bands [16]. Neural activity in the theta frequency band (4–8 Hz) seems to be of particular importance in memory encoding [17]. Indeed, oscillations in the theta frequency allow communication between distant brain regions, promote encoding of novel information [18], enable learning [19], and have been linked to improved declarative memory [20–22].

One possible mechanism through which theta frequency improves memory is theta phase synchronization. Synchronization in declarative memory can be observed locally, for example, using intracranial electrodes placed in the medial temporal lobe. With this method [23], found increased theta phase locking during the encoding of words. Theta phase synchronization can also be observed non-locally. When multimodal (audio-visual) stimuli are synchronously presented in theta phase, episodic memory is enhanced; with stronger theta phase synchronization between the visual and auditory cortex predicting better memory performance [24, 25]. Furthermore [26], observed increased theta phase synchronization between HC and prefrontal cortex (PFC) during the presentation of unexpected items. Interestingly, the PFC, and in particular the MFC, has been ascribed an important role in memory encoding [27–29]. It is also strongly implicated in reward [30, 31] and RPE [32, 33] processing. We hypothesize that during declarative learning, RPEs project to the MFC [15], where they are used to optimize future behavior [34]. Specifically, RPEs may (by means of neuromodulatory signaling) increase theta (phase) synchronization between relevant brain areas (e.g., MFC and HC), therefore allowing associative memories to be glued together more efficiently [35], facilitating (multimodal) memory formation [36].

Unfortunately, the evidence for theta modulation of RPEs in declarative memory thus far remains correlational only. With the rise of non-invasive brain stimulation (NIBS) techniques, the causal role of neural oscillations and their relation to behavior can be explicitly tested [37]. More specifically, transcranial Alternating Current Stimulation (tACS) allows modulating neural oscillations [38]. It is hypothesized that tACS causes underlying brain networks to synchronize or desynchronize. Although tACS has rather low temporal and spatial resolution, its frequency resolution is high. By applying a weak sinusoidal current to the scalp, the likelihood of neural firing is increased or decreased, depending on the stimulation parameters [39]. Ongoing neural oscillations can thus be entrained at specific frequencies of interest [39]. This synchronization modulates brain activity and alters cognitive processes, leading to behavioral changes, which can be measured through, for example, memory performance [40].

Whereas several tACS experiments entraining oscillations at theta frequency looked at its effects on working memory [41–46], a few studies have investigated its effects on declarative memory [47]. applied theta-frequency tACS over the right fusiform cortex while face and scene pairs were encoded. Here, stimulation enhanced memory performance measured after a 24-hour delay. Similarly [48], also found enhanced long-term memory performance after applying theta-frequency tACS over the right posterior cortex while participants learned face-monetary value pairs. To the best of our knowledge, no study examined the effects of theta-frequency tACS over MFC in relation to declarative learning.

Together, these findings suggest that RPEs are projected from brainstem to MFC; elicit theta phase synchronization between several neural areas; and thus boost declarative learning. As such, the goal of the current study was to use theta-frequency (6 Hz) tACS to entrain neural oscillations whilst encoding new word pairs associated with RPEs of different sizes and values. To this end, tACS was applied over the MFC while participants acquired 60 Dutch-Swahili word pairs using the variable-choice experimental paradigm. We hypothesized that if declarative learning is modulated by theta oscillations in MFC, then subsequent memory performance and certainty ratings should be modulated by tACS (i.e., higher recognition accuracies and certainty ratings in the real compared to sham stimulation group); and if theta oscillations are driven by RPE, as the literature review suggests, tACS and RPE should interact.

Methods

Participants

We tested a total of 77 healthy, Dutch-speaking participants. One participant was excluded from further analysis due to below chance level performance on the recognition test. The analyses were run on the remaining 76 participants (57 females, range = 18–29 years, Mage = 20.8 years, SDage = 2.4 years). All participants had no prior knowledge of Swahili, gave written informed consent, were randomly assigned to a real (N = 38) or sham (N = 38) stimulation group, and were paid €17.5. The study was approved by the Medical Ethics Review Board of the Ghent University Hospital and was carried out in accordance with the Declaration of Helsinki.

Material

A total of 330 words (66 Dutch, 24 Japanese and 240 Swahili words) (S1–S4 Tables) were used. Each participant memorized 60 Dutch-Swahili word pairs. The experiment was run on an HP ProBook 6560b laptop with a 15.6” screen size running PsychoPy software (version 1.85.4) [49].

Experimental paradigm

Familiarization task.

Participants started with a familiarization task using the stimuli in the experiment, to control for the novelty of the foreign Swahili words. All Dutch (N = 60) and Swahili (N = 240) words were randomly and sequentially presented on the screen for a duration of two seconds. Participants were asked to press the space bar whenever a Dutch word was presented.

Acquisition task.

Prior to the actual acquisition task, a total of six practice trials with Dutch (N = 6) and Japanese (N = 24) words was presented. After successfully finishing the practice set, participants were presented with the acquisition task. Here, the aim was to learn 60 unique Dutch-Swahili word pair associations. On each trial, one Dutch word was shown together with four Swahili translations (Fig 1A). After four seconds, frames surrounded the eligible Swahili translations. Either one, two or four Swahili translations were framed. In the one-option condition, one Swahili translation was framed and participants could only choose this Swahili word as the translation for the Dutch word. In the two-option condition, two Swahili translations were framed and participants could choose between two options. In the four-option condition trials, all four Swahili translations were framed and participants could choose among these four options. The probability of choosing the correct Swahili translation was therefore 100% (in one-option condition trials), 50% (in two-option condition trials), or 25% (in four-option condition trials). Importantly, each trial was associated with a specific RPE value by fixing a priori whether a trial was rewarded or not and the number of eligible Swahili translations. As a result, participants did not learn the actual Swahili translations of the Dutch words. They were unaware of this manipulation during the experiment, but were debriefed afterwards. Note also that although not explicitly communicated to the participants, there was a clear, normatively correct choice that had to be remembered on each trial. The intention of the experiment was also made clear by the colors (i.e., red/green) and the feedback (i.e., wrong/correct) that were used in the acquisition task. Participants responded with the index and middle finger of the right and left hand. For stimulation purposes, trial duration was controlled by instructing participants to make their choice as soon as the fixation cross turned blue. If no choice was made after two seconds, the fixation cross turned red, urging participants to choose as soon as possible. To ensure that stimulation was given throughout the entire duration of the acquisition task, total time spent in the acquisition task was equated for each participant. Specifically, if participants made a choice less than two seconds after the fixation cross turned blue, feedback was presented after [two seconds—choice duration] seconds. After participants made their choice, the fixation cross turned into a blue “o” indicating that their response had been registered. They were then provided with feedback where they saw the Dutch word, an equation sign, and the to-be-learned Swahili translation (in green for correct choices and in red for incorrect choices) for a duration of five seconds. This was followed by reward feedback (+0.5 Euros for correct choices and +0 Euros for incorrect choices) and a reward update telling them how much money they earned up until the last completed trial (two seconds). After every ten trials, the acquisition task was briefly paused for ten seconds to allow an impedance check.

[Figure omitted. See PDF.]

Fig 1. Experimental paradigm and tACS setup.

(A) Example trial of the acquisition task and recognition test. In the acquisition task, participants choose between 1, 2 or 4 Swahili translations. The two-option condition with rewarded choice is illustrated. (B) Experimental design. The 2 (rewarded or unrewarded choice) x 3 (number of options) experimental design showing the number of trials and associated RPE value in each cell. SRPEs were calculated by subtracting the probability of reward from the obtained reward; URPE is the absolute value of SRPE. (C) tACS setup. Theta-frequency (6 Hz) tACS was applied over the MFC. The stimulation electrode (i.e., red electrode) was placed over FCz, while the reference electrode (i.e., blue electrode) was placed on the neck. (D) Simulation of the electric field with the ROAST toolbox.

https://doi.org/10.1371/journal.pone.0237829.g001

Design. Parametric modulation of RPEs was accomplished by fixing a priori the number of options (one, two or four) and reward on each trial (reward/no reward). This allowed the computation of an RPE for each cell of the design (Fig 1B). In addition, the proportion of trials in each cell of the design matched the reward expectation (i.e., 100% rewarded trials in the one-option condition, 50% rewarded and 50% non-rewarded trials in the two-option condition, and 25% rewarded and 75% non-rewarded trials in the four-option condition).

SRPEs were obtained by subtracting reward probability from reward outcome. For rewarded trials, reward outcome is equal to one, whereas reward outcome is equal to zero for unrewarded trials. Reward probability is determined by the number of options. URPEs are computed by taking the absolute value of the SRPE.

Recognition test.

In the recognition test, participants’ recognition was tested on 60 Dutch-Swahili word pairs that were acquired during the acquisition task (Fig 1A). On each trial, one Dutch word was shown together with the same four Swahili translations from the acquisition task. Spatial positions of the Swahili translations were randomly shuffled relative to the acquisition task to avoid that participants would respond based on the spatial position instead of the learned translation of the Dutch word. In contrast to the acquisition task, no frames surrounded the Swahili translations, and no feedback was provided. No time limit was imposed. At the end of each trial, participants rated their certainty on a four-point scale (“very certain”, “rather certain”, “rather uncertain”, “very uncertain”).

Sensations questionnaire

A subset of participants (N = 61) filled out a sensations questionnaire [50] (S1 File). Participants rated seven sensations (itching, pain, burning, warmth/heat, pinching, metallic/iron taste and fatigue) on a five-point scale (none, mild, moderate, considerable, strong). They were also asked when the discomfort began, how long the discomfort lasted and how much these sensations affected their performance. The sensations questionnaire was used to verify whether participants in the real and sham stimulation group report a difference in sensations.

tACS stimulation

tACS stimulation was applied using a DC-stimulator Plus device (NeuroConn GmbH, Ilmenau, Germany). Two saline-soaked sponge electrodes (5 x 6.5 cm2) were placed on the scalp and neck. The stimulation (red) electrode was positioned at FCz (according to the 10–20 positioning system), targeting the MFC, while the reference (blue) electrode was placed on the neck (Fig 1C). The sponge electrodes were fixed onto the participant’s head with elastic fabric bands. Impedance between electrodes was kept below 15 kΩ. Participants received tACS stimulation at the theta (6 Hz) frequency with an intensity of 2 mA (peak-to-peak; mean 0 mA). A sinusoidal stimulation waveform was used with no DC offset and a phase shift of zero degrees. A fade-in and fade-out period of 5 seconds (30 cycles) was used. tACS was administered during the entire acquisition task for a duration of 16.6 minutes (6000 cycles) in the real stimulation group, while the sham stimulation group received 40 seconds (240 cycles) of stimulation at the beginning of the acquisition task only. Sham stimulation duration was deliberately kept short to avoid changes in cortical excitability [51, 52]. Current flow was simulated using the ROAST (Realistic vOlumetric Approach to Simulate Transcranial electric stimulation) toolbox [53] in MATLAB (Fig 1D).

Data analysis

Both frequentist and Bayesian statistics were calculated. With regard to frequentist statistics, all data were analyzed within the linear mixed effects framework in R software [54], unless mentioned otherwise. For continuous dependent variables (i.e., certainty ratings in the recognition test) linear mixed effects models were used, while for categorical dependent variables (i.e., recognition accuracy) generalized linear mixed effects models were applied. A random intercept for participants was included in each model, while all predictors (i.e., accuracy, SRPE and stimulation) were mean-centered. Note that SRPEs were treated as a continuous predictor allowing the inclusion of all 60 trials per participant to estimate its regression coefficient, with the exception of invalid trials (i.e., trials on which a non-framed Swahili translation was chosen during the acquisition task). We report the χ2 statistics from the ANOVA Type III tests. All data are made publicly available at OSF (DOI 10.17605/OSF.IO/ZXHQ4).

In addition to frequentist statistics, Bayesian repeated measures analyses of variance (ANOVAs) are reported that were performed in JASP (version 0.11.1; [55]). In Bayesian ANOVAs, recognition accuracy and certainty ratings were analyzed as a function of SRPE and stimulation. Bayes factors (BFs) quantify the evidence in favor of the null hypothesis (BF01; e.g., tACS does not influence memory performance) or the alternative hypothesis (BF10 = 1/BF01; e.g., tACS influences memory performance). BF01 is reported when the Bayesian analysis provides relatively more evidence for the null hypothesis; BF10 is instead reported when the analysis provides relatively more evidence for the alternative hypothesis. We used default prior settings for all analyses [56]. To determine the strength of evidence, we used Jeffreys’ benchmarks [57], with BFs corresponding to anecdotal (0–3), substantial (3–10), strong (10–30), very strong (30–100) or decisive (>100) evidence.

Results

Sensations questionnaire

Independent samples t-tests were used to verify whether sensations varied between the two stimulation groups. Participants in the real and sham stimulation groups did not report a significant difference for any of the sensations probed (itching, pain, burning, warmth/heat, pinching, metallic/iron taste and fatigue) (all p > .06). Furthermore, there were no significant differences between stimulation groups with regard to when the discomfort began, t(58.90) = 0.48, p = .63 (real: M = 1.23, SD = 0.50, range = 0−2; sham: M = 1.17, SD = 0.46, range = 0−2), and how much these sensations affected their performance, t(53.77) = 1.13, p = .26 (real: M = 1.39, SD = 0.62, range = 0−4; sham: M = 1.23, SD = 0.43, range = 0−4). Participants in the real stimulation group did report that the discomfort lasted significantly longer compared to the sham stimulation group, t(40.33) = 3.35, p = .002 (real: M = 1.68, SD = 0.83, range = 0−2; sham: M = 1.13, SD = 0.35, range = 0−2).

Recognition accuracy

Here, we verified whether recognition accuracy linearly increased with SRPEs. Replicating earlier research, frequentist statistics revealed a significant positive effect of SRPE, χ2(1, N = 76) = 9.13, p = .003, with larger and more positive RPEs leading to increased recognition accuracy (Fig 2A and 2B). There was no main effect of stimulation on recognition accuracy, χ2(1, N = 76) = 1.42, p = .23. The interaction between SRPE and stimulation was also not significant, χ2(1, N = 76) = .004, p = 0.95.

[Figure omitted. See PDF.]

Fig 2. Results.

(A-B) Recognition accuracy as a function of SRPE in the real and sham stimulation group, respectively. The average recognition and its 95% confidence interval were estimated and superimposed. Gray dots represent data points for individual subjects. Recognition accuracy increases linearly with larger and more positive RPEs in the two stimulation groups, suggesting a SRPE effect. (C-D) Certainty rating for correct recognitions in the real and sham stimulation group, respectively. The average certainty and its 95% confidence interval were estimated and superimposed. Gray dots and rectangles represent data of individual subjects for correct recognitions. In the two stimulation groups, SRPE significantly predicted certainty for correctly recognized word pairs. (E-F) Certainty rating for incorrect recognitions in the real and sham stimulation group, respectively. The average certainty and its 95% confidence interval were estimated and superimposed. Gray dots and rectangles represent data of individual subjects for incorrect recognitions. In the two stimulation groups, SRPE did not significantly predict certainty for incorrectly recognized word pairs.

https://doi.org/10.1371/journal.pone.0237829.g002

Bayesian repeated measures ANOVA provided substantial evidence for the absence of a stimulation effect (BF01 = 3.02, evidence for null versus alternative model). Thus, the observed data were about 3 times more likely under the model that included no stimulation than under the alternative model that did. The evidence for the SRPE effect was decisive (BF10 > 100, evidence for alternative versus null model). In addition, there was strong evidence against the interaction of SRPE and stimulation (BF01 = 54.66, evidence for main-effects-only relative to main-effects-plus-interaction model).

Certainty ratings

For the certainty ratings there was a significant main effect of recognition accuracy, χ2(1, N = 76) = 1170, p < .001, indicating that participants were more certain of correctly recognized word pairs compared to incorrectly recognized word pairs (see (S1–S4 Figs) for within-subject behavioral responses for the certainty ratings). In addition, there was a significant interaction between SRPE and recognition accuracy, χ2(1, N = 76) = 7.63, p = .006. Follow-up analysis revealed that, as expected, SRPE increased certainty for correctly recognized word pairs, χ2(1, N = 76) = 9.14, p = .002, but did not affect certainty for false recognitions, i.e., incorrectly recognized word pairs, χ2(1, N = 76) = 2.16, p = .14 (Fig 2C and 2D). In addition, the data revealed a significant interaction between stimulation and recognition accuracy on certainty ratings, χ2(1, N = 76) = 5.37, p = .02. Follow-up analysis revealed a main effect of stimulation for the correctly recognized word pairs, χ2(1, N = 76) = 5.03, p = .02, but not for incorrectly recognized word pairs, χ2(1, N = 76) = 0.11, p = .75. Participants in the sham stimulation group were more certain of correctly recognized word pairs, compared to participants in the real stimulation group. Importantly, although participants in the real stimulation group reported increased discomfort duration, the effect of discomfort duration did not significantly affect certainty in the real, χ2(1, N = 31) = 0.93, p = .33, and sham, χ2(1, N = 30) = 0.19, p = .66, stimulation groups. This suggests that discomfort in itself did not influence the certainty rating. Finally, the interaction between SRPE and stimulation was not significant, χ2(1, N = 76) = 1.61, p = .20.

A Bayesian repeated measures ANOVA revealed anecdotal evidence for the absence of a stimulation effect (BF01 = 1.33, null model relative to model including stimulation). For the SRPE effect, the evidence was decisive (BF10 > 100, model including SRPE compared to null model). We also found strong evidence against the interaction of SRPE and stimulation (BF01 = 19.74, compared to two-main-effects model).

Discussion

The main objective of our study was to examine if theta-frequency (6 Hz) tACS can modulate the effect of RPEs in declarative learning. For this purpose, participants acquired 60 Dutch-Swahili word pairs, associated with RPEs of different sizes and values, while the MFC was stimulated. We replicated our earlier finding of SRPEs driving declarative learning [10]. Word pair recognition increased for large and positive RPEs. However, contrary to our hypothesis, theta-frequency (6 Hz) tACS did not successfully improve memory nor modulate the effect of RPEs on declarative learning. There was a small effect of stimulation on certainty in the correctly recognized words, but this effect requires replication and must currently be interpreted with caution.

Whereas the importance of RPEs in procedural learning has been well established, its role in declarative learning has remained elusive until recently. One of the first experimental paradigms examining the effect of RPEs in declarative learning was put forward by [58]. Although this RPE effect on declarative memory could not be replicated [59, 60], several research labs have since then used a range of experimental paradigms to investigate the role of RPEs in declarative learning. Most of these studies revealed positive effects of RPEs on declarative memory [8, 9, 61], but one study also reported negative effects [62] (for review see [7]). Overall, these studies (including the current one) support the claim that RPEs are a key factor in the formation of declarative memory.

Prior research has repeatedly shown a role of theta frequency in (reward) prediction error processing [63–66] as well as memory performance [21]. In particular [25], provided direct evidence for a causal role of theta frequency in memory. Memory for multimodal (audio-visual) stimuli was enhanced only when these stimuli were modulated at the theta frequency and not at other frequencies. Furthermore, in an earlier EEG study from our lab, we examined the neural signatures of RPEs in declarative learning and found increased theta (4–8 Hz) power during reward feedback [11]. However, it must be noted that in this particular EEG study, theta frequency followed an unsigned RPE (URPE) pattern during reward feedback. Theta power thus increased for both large negative and large positive RPEs. This URPE pattern evolved into a SRPE pattern during reward feedback and was accompanied by power increases in the high-beta (20–30 Hz) and high-alpha (10–17 Hz) frequency bands. Although beta and alpha power followed a clear SRPE pattern, we opted not to stimulate at these frequencies as there is more inter-individual variability with regard to peak-frequency [67].

We hypothesized that declarative learning is facilitated by theta frequency synchronization. Neurons are synchronized when their activation is locked to a common (slow-wave) phase. In such case, spikes of pre- and postsynaptic neurons are highly correlated, enabling synaptic learning between pairs of neurons because synaptic plasticity relies on the precise spike-timing of neurons [68]. Theta phase may modulate spike-timing-dependent plasticity by ensuring that (anatomically distant) neurons fire in synchrony [69, 70]. As tACS modulates the spike-timing of neurons [71–73], it is a promising tool to causally manipulate neural oscillations related to RPE-processing in declarative learning. For this reason, theta-frequency tACS was used to stimulate the MFC. Unfortunately, however, our tACS manipulation did not affect memory performance.

In the following section, we speculate why we found no effect of theta-frequency (6Hz) tACS and provide suggestions for future research. First, tACS has a relatively low spatial resolution. As a consequence, current flow is not focal, but distributed across the entire scalp. In Fig 1D, we simulated the electric field in our paradigm. The distribution of current flow is indeed very broad, encompassing several brain areas. Therefore, it is conceivable that our tACS manipulation did not exclusively stimulate the MFC. Due to a complex interplay of brain networks, it remains possible that other brain regions were stimulated as well, potentially interacting or interfering with our RPE effect in declarative learning. Second, tACS only generates weak electrical fields. The simulation in Fig 1D shows that using a stimulation intensity of 2mA caused, at best, an electric field strength of 0.3 V/m, which is on the weak side. The induction of weak electrical fields makes it difficult to entrain endogenous oscillations. This is especially the case if the brain regions that need to be stimulated are located deeper within the brain. For instance [74], reported that low frequency tACS did not modulate ongoing brain activity during resting wakefulness [75]. also found that conventional stimulation parameters are insufficient to induce measurable effects. However, the use of stronger currents might be accompanied by increased discomfort. Third, some researchers raised the issue of brain-state-dependent effects [76–80]. More specifically, tACS effects might depend on the current brain state of the participant. If a participant is in an optimal brain state where brain networks are synchronized enabling high encoding efficiency, stimulating the learning brain might impair learning. If, however, a participant is in a non-optimal brain state where synchronization is less pronounced and accompanied by decreased encoding efficiency, then applying stimulation could facilitate learning and improve memory performance. Importantly [81], have shown that endogenous brain oscillations are entrained only when phase-alignment is achieved between the applied stimulation and the ongoing brain activity (see also [72]). Therefore, stimulation should ideally be phase-aligned to participants’ internal brain states [82]. As we could not measure participants’ brain states in our study, it is possible that tACS interacted with ongoing endogenous brain states. Fourth, it remains possible that theta frequency has no effect on RPEs in declarative learning and declarative memory per se. For instance [83], applied theta-frequency (5 Hz) tACS over the ventrolateral prefrontal cortex during the acquisition of face-occupation pairs in older adults. In line with our study, theta-frequency tACS did not affect memory performance. Fifth, due to logistical constraints, a between-subjects design was used. By doing so, individual differences are not easily controlled. This could be mitigated by using a within-subjects design, where each participant is subjected to a real and a sham stimulation condition. Finally, due to the lack of standardized tACS procedures across studies, it remains difficult to draw definitive conclusions. The absence of an effect highlights the importance for understanding its underlying mechanisms [84], and setting up general procedural guidelines with regard to neurostimulation studies [51, 85].

Taken together these issues, we argue that the lack of strong, localized, and phase-dependent stimulation is the most important factor contributing to our null result. Therefore, a follow-up of our study would be to use rhythmic Transcranial Magnetic Stimulation (TMS) to improve spatial resolution and induce stronger electrical fields [86] while simultaneously measuring EEG. Even though the spatial resolution of TMS remains debated [87], it is more focal than tACS. By using a closed-loop approach, brain states are continuously monitored and stimulation can be phase-aligned to individual theta oscillations. As such, we would be in a better position to influence learning. Interestingly, in the same experimental paradigm where rTMS at beta frequency modulated declarative memory [88], tACS at beta frequency did not successfully modulate memory formation [89]. This finding thus further validates the use of (rhythmic) TMS over tACS. To further increase stimulation strength, instead of delivering single pulses at theta frequency, another procedure would be to deliver high-frequency bursts at theta frequency. This procedure has also been shown to increase memory performance and certainty ratings [90, 91] and thus is also a viable alternative for future research.

In summary, the current study examined whether applying theta-frequency (6 Hz) tACS over the MFC modulates the RPE effect in declarative learning. Previous behavioral results were replicated, with SRPEs driving declarative learning. However, theta tACS over the MFC did not modulate the effect of RPEs on declarative learning, and we proposed guidelines for future neuromodulation studies in declarative memory.

Supporting information

[Figure omitted. See PDF.]

S1 Fig. Certainty ratings for subjects 1 to 20.

https://doi.org/10.1371/journal.pone.0237829.s001

(TIFF)

S2 Fig. Certainty ratings for subjects 21 to 41.

https://doi.org/10.1371/journal.pone.0237829.s002

(TIFF)

S3 Fig. Certainty ratings for subjects 42 to 61.

https://doi.org/10.1371/journal.pone.0237829.s003

(TIFF)

S4 Fig. Certainty ratings for subjects 62 to 77.

https://doi.org/10.1371/journal.pone.0237829.s004

(TIFF)

S1 Table. Stimulus material practice set: 6 Dutch words.

https://doi.org/10.1371/journal.pone.0237829.s005

(DOCX)

S2 Table. Stimulus material practice set: 24 Japanese words.

https://doi.org/10.1371/journal.pone.0237829.s006

(DOCX)

S3 Table. Stimulus material: 60 Dutch words.

https://doi.org/10.1371/journal.pone.0237829.s007

(DOCX)

S4 Table. Stimulus material: 240 Swahili words.

https://doi.org/10.1371/journal.pone.0237829.s008

(DOCX)

S1 File. Sensations questionnaire.

https://doi.org/10.1371/journal.pone.0237829.s009

(DOCX)

Acknowledgments

We thank Lara Bardi for help with the tACS startup.

Citation: Ergo K, De Loof E, Debra G, Pastötter B, Verguts T (2020) Failure to modulate reward prediction errors in declarative learning with theta (6 Hz) frequency transcranial alternating current stimulation. PLoS ONE 15(12): e0237829. https://doi.org/10.1371/journal.pone.0237829

References

1. Squire LR. Memory systems of the brain: A brief history and current perspective. Neurobiol Learn Mem. 2004;82(3):171–7. pmid:15464402

2. Bastin C, Van der Linden M. The contribution of recollection and familiarity to recognition memory: A study of the effects of test format and aging. Neuropsychology. 2003;17(1):14. pmid:12597069

3. Eichenbaum H. Hippocampus: Cognitive processes and neural representations that underlie declarative memory. Neuron. 2004;44(1):109–20. pmid:15450164

4. Greve A, Cooper E, Kaula A, Anderson MC, Henson R. Does prediction error drive one-shot declarative learning? J Mem Lang. 2017;94:149–65. pmid:28579691

5. Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science (80-). 1997;275(5306):1593–9.

6. Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH. A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci. 2013;16(7):966–73. pmid:23708143

7. Ergo K, De Loof E, Verguts T. Reward prediction error and declarative memory. Trends Cogn Sci. 2020;24(5):388–97. pmid:32298624

8. Rouhani N, Norman KA, Niv Y. Dissociable effects of surprising rewards on learning and memory. J Exp Psychol Learn Mem Cogn. 2018;44(9):1430–43. pmid:29553767

9. Jang AI, Nassar MR, Dillon DG, Frank MJ. Positive reward prediction errors during decision-making strengthen memory encoding. Nat Hum Behav. 2019;3(7):719–32. pmid:31061490

10. De Loof E, Ergo K, Naert L, Janssens C, Talsma D, Van Opstal F, et al. Signed reward prediction errors drive declarative learning. Ito E, editor. PLoS One. 2018 Jan;13(1):e0189212. pmid:29293493

11. Ergo K, De Loof E, Janssens C, Verguts T. Oscillatory signatures of reward prediction errors in declarative learning. Neuroimage. 2019;186:137–45. pmid:30391561

12. Calderon CB, De Loof E, Ergo K, Snoeck A, Boehler CN, Verguts T. Signed reward prediction errors in the ventral striatum drive episodic memory. bioRxiv. 2020 Jan 3;2020.01.03.893578.

13. Watabe-Uchida M, Eshel N, Uchida N. Neural circuitry of reward prediction error. Annu Rev Neurosci. 2017;40(1):373–94. pmid:28441114

14. Shohamy D, Adcock RA. Dopamine and adaptive memory. Trends Cogn Sci. 2010;14(10):464–72. pmid:20829095

15. Nieuwenhuis S, Holroyd CB, Mol N, Coles MGH. Reinforcement-related brain potentials from medial frontal cortex: Origins and functional significance. Neurosci Biobehav Rev. 2004 Jul 1;28(4):441–8. pmid:15289008

16. Lisman JE, Grace AA, Duzel E. A neoHebbian framework for episodic memory; role of dopamine-dependent late LTP. Trends Neurosci. 2011 Oct;34(10):536–47. pmid:21851992

17. Herweg NA, Solomon EA, Kahana MJ. Theta oscillations in human memory. Trends Cogn Sci. 2019;24(3):20.

18. Klimesch W. EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Res Rev. 1999;29(2):169–95. pmid:10209231

19. Fries P. A mechanism for cognitive dynamics: Neuronal communication through neuronal coherence. Trends Cogn Sci. 2005;9(10):474–80. pmid:16150631

20. Osipova D, Takashima A, Oostenveld R, Fernández G, Maris E, Jensen O. Theta and gamma oscillations predict encoding and retrieval of declarative memory. J Neurosci. 2006;26(28):7523–31. pmid:16837600

21. Sederberg PB, Kahana MJ, Howard MW, Donner EJ, Madsen JR. Theta and gamma oscillations during encoding predict subsequent recall. J Neurosci. 2003;23(34):10809–14. pmid:14645473

22. Summerfield C, Mangels JA. Coherent theta-band EEG activity predicts item-context binding during encoding. Neuroimage. 2005;24:692–703. pmid:15652304

23. Solomon EA, Stein JM, Das S, Gorniak R, Sperling MR, Worrell G, et al. Dynamic theta networks in the human medial temporal lobe support episodic memory. Curr Biol. 2019 Apr 1;29(7):1100–11. pmid:30905609

24. Wang D, Clouter A, Chen Q, Shapiro KL, Hanslmayr S. Single-trial phase entrainment of theta oscillations in sensory regions predicts human associative memory performance. J Neurosci. 2018 Jun 13;38(28):6299–309. pmid:29899027

25. Clouter A, Shapiro KL, Hanslmayr S. Theta phase synchronization is the glue that binds human associative memory. Curr Biol. 2017;27(23):3143–8. pmid:28988860

26. Gruber MJ, Hsieh L-T, Staresina BP, Elger CE, Fell J, Axmacher N, et al. Theta phase synchronization between the human hippocampus and the prefrontal cortex supports learning of unexpected information. J Cogn Neurosci. 2018;30(11):1646–56. pmid:29952700

27. De La Vega A, Chang LJ, Banich MT, Wager TD, Yarkoni T. Large-scale meta-analysis of human medial frontal cortex reveals tripartite functional organization. J Neurosci. 2016 Jun 15;36(24):6553–62. pmid:27307242

28. Euston DR, Gruber AJ, Mcnaughton BL. The role of medial prefrontal cortex in memory and decision making. Neuron. 2012;76(6):1057–70. pmid:23259943

29. Preston AR, Eichenbaum H. Interplay of hippocampus and prefrontal cortex in memory. Curr Biol. 2013;23(17):R764–73. pmid:24028960

30. Gehring WJ, Willoughby AR. The medial frontal cortex and the rapid processing of monetary gains and losses. Science (80-). 2002;295(5563):2279–82. pmid:11910116

31. Vassena E, Silvetti M, Boehler CN, Achten E, Fias W, Verguts T. Overlapping neural systems represent cognitive effort and reward anticipation. PLoS One. 2014 Mar 7;9(3). pmid:24608867

32. Oliveira FTP, McDonald JJ, Goodman D. Performance monitoring in the anterior cingulate is not all error related: Expectancy deviation and the representation of action-outcome associations. J Cogn Neurosci. 2007;19(12):1994–2004. pmid:17892382

33. Jessup RK, Busemeyer JR, Brown JW. Error effects in anterior cingulate cortex reverse when error likelihood is high. J Neurosci. 2010 Mar 3;30(9):3467–72. pmid:20203206

34. Silvetti M, Vassena E, Abrahamse E, Verguts T. Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner. PLoS Comput Biol. 2018 Aug 1;14(8):e1006370. pmid:30142152

35. Berens SC, Horner AJ. Theta rhythm: Temporal glue for episodic memory. Curr Biol. 2017;27(20):R1110–2. pmid:29065291

36. Backus AR, Schoffelen J-M, Szebényi S, Hanslmayr S, Doeller CF. Hippocampal-prefrontal theta oscillations support memory integration. Curr Biol. 2016;26(4):450–7. pmid:26832442

37. Thut G, Bergmann TO, Fröhlich F, Soekadar SR, Brittain JS, Valero-Cabré A, et al. Guiding transcranial brain stimulation by EEG/MEG to interact with ongoing brain activity and associated functions: A position paper. Vol. 128, Clinical Neurophysiology. Elsevier Ireland Ltd; 2017. p. 843–57. pmid:28233641

38. Zaehle T, Rach S, Herrmann CS. Transcranial alternating current stimulation enhances individual alpha activity in human EEG. PLoS One. 2010;5(11):1–7. pmid:21072168

39. Antal A, Paulus W. Transcranial alternating current stimulation (tACS). Front Hum Neurosci. 2013;7:317. pmid:23825454

40. Ambrus GG, Pisoni A, Primaßin A, Turi Z, Paulus W, Antal A. Bi-frontal transcranial alternating current stimulation in the ripple range reduced overnight forgetting. Front Cell Neurosci. 2015;9:374. pmid:26441544

41. Jaušovec N, Jaušovec K. Increasing working memory capacity with theta transcranial alternating current stimulation (tACS). Biol Psychol. 2014;96:42–7. pmid:24291565

42. Pahor A, Jaušovec N. The effects of theta and gamma tACS on working memory and electrophysiology. Front Hum Neurosci. 2018;11:651. pmid:29375347

43. Chander BS, Witkowski M, Braun C, Robinson SE, Born J, Cohen LG, et al. tACS phase locking of frontal midline theta oscillations disrupts working memory performance. Front Cell Neurosci. 2016;10:120. pmid:27199669

44. Alekseichuk , Turi Z, Lara G de, Antal A, Biology WP-C, 2016 U. Spatial working memory in humans depends on theta and high gamma synchronization in the prefrontal cortex. Curr Biol. 2016;26(12):1513–21. pmid:27238283

45. Albouy P, Weiss A, Baillet S, Zatorre RJ. Selective entrainment of theta oscillations in the dorsal stream causally enhances auditory working memory performance. Neuron. 2017 Apr 5;94(1):193–206. pmid:28343866

46. Vosskuhl J, Huster RJ, Herrmann CS. Increase in short-term memory capacity induced by down-regulating individual theta frequency via transcranial alternating current stimulation. Front Hum Neurosci. 2015;9:257. pmid:26005411

47. Lang S, Gan LS, Alrazi T, Monchi O. Theta band high definition transcranial alternating current stimulation, but not transcranial direct current stimulation, improves associative memory performance. Sci Rep. 2019;9(1):8562. pmid:31189985

48. Alekseichuk I, Turi Z, Veit S, Paulus W. Model-driven neuromodulation of the right posterior region promotes encoding of long-term memories. Brain Stimul. 2020 Mar 1;13(2):474–83. pmid:31882373

49. Peirce JW. PsychoPy—psychophysics software in Python. J Neurosci Methods. 2007;162(1–2):8–13. pmid:17254636

50. Fertonani A, Ferrari C, Miniussi C. What do you feel if I apply transcranial electric stimulation? Safety, sensations and secondary induced effects. Clin Neurophysiol. 2015 Nov 1;126(11):2181–8. pmid:25922128

51. Nitsche MA, Cohen LG, Wassermann EM, Priori A, Lang N, Antal A, et al. Transcranial direct current stimulation: State of the art 2008. Brain Stimul. 2008;1(3):206–23. pmid:20633386

52. Gandiga PC, Hummel FC, Cohen LG. Transcranial DC stimulation (tDCS): a tool for double-blind sham-controlled clinical studies in brain stimulation. Clin Neurophysiol. 2006;117(4):845–50. pmid:16427357

53. Huang Y, Datta A, Bikson M, Parra LC. Realistic volumetric-approach to simulate transcranial electric stimulation-ROAST-a fully automated open-source pipeline. J Neural Eng. 2019;16(5). pmid:31071686

54. R Core Team. R: A language and environment for statistical computing. R Found Stat Comput Vienna, Austria. 2014;

55. JASP Team. JASP (Version 0.13)[Computer software] [Internet]. 2020. Available from: https://jasp-stats.org/

56. Rouder JN, Morey RD, Speckman PL, Province JM. Default Bayes factors for ANOVA designs. J Math Psychol. 2012;56:356–74.

57. Jeffreys H. The theory of probability. 3rd ed. New York, NY: Oxford University Press.; 1961.

58. Bunzeck N, Dayan P, Dolan RJ, Duzel E. A common mechanism for adaptive scaling of reward and novelty. Hum Brain Mapp. 2010;31(9):1380–94. pmid:20091793

59. Mason A, Ludwig C, Farrell S. Adaptive scaling of reward in episodic memory: A replication study. Q J Exp Psychol. 2017;70(11):2306–18. pmid:27603181

60. Mason A, Farrell S, Howard-Jones P, Ludwig CJH. The role of reward and reward uncertainty in episodic memory. J Mem Lang. 2017;96:62–77.

61. Davidow JY, Foerde K, Galvan A, Shohamy D. An upside to reward sensitivity: The hippocampus supports enhanced Reinforcement Learning in adolescence. Neuron. 2016;92(1):93–9. pmid:27710793

62. Wimmer GE, Braun EK, Daw ND, Shohamy D. Episodic memory encoding interferes with reward learning and decreases striatal prediction errors. J Neurosci. 2014;34(45):14901–12. pmid:25378157

63. Cavanagh JF, Figueroa CM, Cohen MX, Frank MJ. Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb Cortex. 2012 Nov;22(11):2575–86. pmid:22120491

64. Cavanagh JF, Frank MJ, Klein TJ, Allen JJB. Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. Neuroimage. 2010 Feb;49(4):3198–209. pmid:19969093

65. Mas-Herrero E, Marco-Pallarés J. Frontal theta oscillatory activity is a common mechanism for the computation of unexpected outcomes and learning rate. J Cogn Neurosci. 2014;26(3):447–58. pmid:24188368

66. Cavanagh JF, Cohen MX, Allen JJB. Prelude to and resolution of an error: EEG phase synchrony reveals cognitive control dynamics during action monitoring. Neuroscience. 2009;29(1):98–105. pmid:19129388

67. Haegens S, Cousijn H, Wallis G, Harrison PJ, Nobre AC. Inter- and intra-individual variability in alpha peak frequency. Neuroimage. 2014;92:46–55. pmid:24508648

68. Caporale N, Dan Y. Spike timing–dependent plasticity: A Hebbian learning rule. Annu Rev Neurosci. 2008;31:25–46. pmid:18275283

69. Fell J, Axmacher N. The role of phase synchronization in memory processes. Nat Rev Neurosci. 2011;12(2):105–18. pmid:21248789

70. Rutishauser U, Ross IB, Mamelak AN, Schuman EM. Human memory strength is predicted by theta-frequency phase-locking of single neurons. Nature. 2010;464(7290):903–7. pmid:20336071

71. Reato D, Rahman A, Bikson M, Parra LC. Low-intensity electrical stimulation affects network dynamics by modulating population rate and spike timing. J Neurosci. 2010;30(45):15067–79. pmid:21068312

72. Krause MR, Vieira PG, Csorba BA, Pilly PK, Pack CC. Transcranial alternating current stimulation entrains single-neuron activity in the primate brain. Proc Natl Acad Sci U S A. 2019 Mar 19;116(12):5747–55. pmid:30833389

73. Johnson L, Alekseichuk I, Krieg J, Doyle A, Yu Y, Vitek J, et al. Dose-dependent effects of transcranial alternating current stimulation on spike timing in awake nonhuman primates. Sci Adv. 2020 Sep 2;6(36):eaaz2747. pmid:32917605

74. Lafon B, Henin S, Huang Y, Friedman D, Melloni L, Thesen T, et al. Low frequency transcranial electrical stimulation does not entrain sleep rhythms measured by human intracranial recordings. Nat Commun. 2017 Dec 1;8(1):1–14. pmid:28232747

75. Vöröslakos M, Takeuchi Y, Brinyiczki K, Zombori T, Oliva A, Fernández-Ruiz A, et al. Direct effects of transcranial electric stimulation on brain circuits in rats and humans. Nat Commun. 2018 Dec 1;9(1):1–17. pmid:29317637

76. Ezzyat Y, Kragel JE, Burke JF, Gorniak R, Rizzuto DS, Kahana MJ, et al. Direct brain stimulation modulates encoding states and memory performance in humans. Curr Biol. 2017;27:1251–8. pmid:28434860

77. Hanslmayr S, Roux F. Human memory: Brain-state-dependent effects of stimulation. Curr Biol. 2017;27(10):385–7.

78. Bergmann TO. Brain state-dependent brain stimulation. Front Psychol. 2018;9:2108. pmid:30443236

79. Nguyen J, Deng Y, Reinhart RMG. Brain-state determines learning improvements after transcranial alternating-current stimulation to frontal cortex. Brain Stimul. 2018;11(4):723–6. pmid:29482970

80. Silvanto J, Muggleton N, Walsh V. State-dependency in brain stimulation studies of perception and cognition. Trends Cogn Sci. 2008 Dec 1;12(12):447–54. pmid:18951833

81. Thut G, Schyns PG, Gross J. Entrainment of perceptually relevant brain oscillations by non-invasive rhythmic stimulation of the human brain. Front Psychol. 2011 Jul 20;2(JUL):170. pmid:21811485

82. Kim K, Ekstrom AD, Tandon N. A network approach for modulating memory processes via direct and indirect brain stimulation: Toward a causal approach for the neural basis of memory. Neurobiol Learn Mem. 2016 Oct 1;134:162–77. pmid:27066987

83. Klink K, Peter J, Wyss P, Klöppel S. Transcranial electric current stimulation during associative memory encoding: Comparing tACS and tDCS effects in healthy aging. Front Aging Neurosci. 2020;12:66. pmid:32256337

84. Hanslmayr S, Axmacher N, Inman CS. Modulating human memory via entrainment of brain oscillations. Trends Neurosci. 2019 Jul 1;42(7):485–99. pmid:31178076

85. Tavakoli A V, Yun K. Transcranial alternating current stimulation (tACS) mechanisms and protocols. Front Cell Neurosci. 2017;11:214. pmid:28928634

86. Thut G, Veniero D, Romei V, Miniussi C, Schyns P, Gross J. Rhythmic TMS causes local entrainment of natural oscillatory signatures. Curr Biol. 2011;21(14):1176–85. pmid:21723129

87. Slotnick S. Controversies in cognitive neuroscience. Basingstoke: New York: Palgrave Macmillan; 2013.

88. Hanslmayr S, Matuschek J, Fellner MC. Entrainment of prefrontal beta oscillations induces an endogenous echo and impairs memory formation. Curr Biol. 2014;24(8):904–9. pmid:24684933

89. Braun V, Sokoliuk R, Hanslmayr S. On the effectiveness of event-related beta tACS on episodic memory formation and motor cortex excitability. Brain Stimul. 2017 Sep 1;10(5):910–8. pmid:28528736

90. Demeter E, Mirdamadi JL, Meehan SK, Taylor SF. Short theta burst stimulation to left frontal cortex prior to encoding enhances subsequent recognition memory. Cogn Affect Behav Neurosci. 2016;16(4):724–35. pmid:27098772

91. Tambini A, Nee DE, D’Esposito M. Hippocampal-targeted theta-burst stimulation enhances associative memory formation. J Cogn Neurosci. 2017;30(10):1452–72.