Behavioural ‘nudging’ interventions to reduce low-value care for low back pain in the emergency department (NUDG-ED): protocol for a 2×2 factorial, before-after, cluster randomised trial

Abstract

Introduction

Opioids and imaging are considered low-value care for most people with low back pain. Yet around one in three people presenting to the emergency department (ED) will receive imaging, and two in three will receive an opioid. NUDG-ED aims to determine the effectiveness of two different behavioural ‘nudge’ interventions on low-value care for ED patients with low back pain.

Methods and analysis

NUDG-ED is a 2×2 factorial, open-label, before-after, cluster randomised controlled trial. The trial includes 8 ED sites in Sydney, Australia. Participants will be ED clinicians who manage back pain, and patients who are 18 years or over presenting to ED with musculoskeletal back pain. EDs will be randomly assigned to receive (i) patient nudges, (ii) clinician nudges, (iii) both interventions or (iv) no nudge control. The primary outcome will be the proportion of encounters in ED for musculoskeletal back pain where a person received a non-indicated lumbar imaging test, an opioid at discharge or both. We will require 2416 encounters over a 9-month study period (3-month before period and 6-month after period) to detect an absolute difference of 10% in use of low-value care due to either nudge, with 80% power, alpha set at 0.05 and assuming an intra-class correlation coefficient of 0.10, and an intraperiod correlation of 0.09. Patient-reported outcome measures will be collected in a subsample of patients (n≥456) 1 week after their initial ED visit. To estimate effects, we will use a multilevel regression model, with a random effect for cluster and patient, a fixed effect indicating the group assignment of each cluster and a fixed effect of time.

Ethics and dissemination

This study has ethical approval from Southwestern Sydney Local Health District Human Research Ethics Committee (2023/ETH00472). We will disseminate the results of this trial via media, presenting at conferences and scientific publications.

Trial registration number

ACTRN12623001000695.

Correspondence to Dr Adrian C Traeger; [email protected]

STRENGTHS AND LIMITATIONS OF THIS STUDY

• First randomised controlled trial to test the impact of clinician-directed and patient-directed behavioural nudges on reducing low-value care for back pain in the emergency department.

• Tests scalable interventions to align care with clinical guidelines.

• Study design informed by pilot studies, hospital administrative data, clinician focus groups, patient involvement and behavioural economic theory.

• Intervention period of 6 months may not capture potential seasonal variations in care.

• Clinicians will not be blinded to the nudge interventions.

• Including a ‘before-after’ component in the trial will allow us to investigate potential Hawthorne effects.

Background and rationale

The problem

There is increasing global recognition that use of low-value care—healthcare services that are ineffective or offer little patient benefit—is a pervasive problem.¹ An Australian systematic review identified 156 low-value health services listed on the Medical Benefits Schedule, that were either ineffective or unsafe.² Examples included routine diagnostic imaging for low back pain, opioid overuse for non-cancer pain and several types of spinal surgery.² In recognition of the impact on quality and safety of care and resource stewardship, reducing low-value care has become an international priority, motivating awareness campaigns across 25 countries.³

Low back pain is the leading cause of disability worldwide,⁴ is the fifth most common reason to visit the ED⁵ and is often associated with low-value care.⁶ Our recent analysis of hospital administrative data on 6393 back pain presentations to the ED found 23.6% received lumbar imaging and 69.6% received opioids.⁷ Neither of these services is recommended for management of uncomplicated low back pain because of their poor benefit-to-harm profile. Most low back pain is benign in nature and imaging adds little information to the clinical assessment.⁸ Non-indicated lumbar imaging can have immediate harms, such as exposure to radiation, increased patient anxiety, increased wait times (which can increase morbidity, decrease patient satisfaction and contribute to ED overcrowding)⁹ and increased length of stay.¹⁰ Potential long-term harms of imaging include the detection of irrelevant findings (eg, ‘degenerative discs’) that correlate poorly with symptoms, and trigger ineffective or unnecessary procedures such as spinal surgery.¹¹

Similarly, harms from opioids are well documented. In the short-term opioids can cause nausea, vomiting, constipation and dizziness.¹² Potential long-term harms include dependence, overdose and death.¹² Between 2007 and 2016, the number of Australians dying or hospitalised from poisoning by prescription opioids increased by 38% and 62%, respectively.¹² Evidence from a recent high-quality trial found that for people with acute low back pain, an opioid analgesic strategy provided no significant difference in pain severity compared with placebo.^13–15 Overuse of low-value care such as opioids at discharge is often coupled with underuse of high-value care options such as advice, non-drug care and simple analgesics.¹⁶

Past interventions

There is limited evidence on the effectiveness of strategies to reduce use of low-value imaging and opioids for low back pain. A 2015 systematic review and earlier Cochrane review concluded there was an absence of effective strategies to reduce low-value imaging.^{17 18} Similarly, a 2020 systematic review of opioid stewardship interventions showed that although several strategies have been implemented (eg, education, hospital policies, electronic medical record (eMR) changes, registries), there was a lack of high-quality randomised controlled trials (RCTs) to determine whether these strategies work.¹⁹ Recently, an intervention to reduce low-value care for low back pain was tested in a high-quality RCT in 4491 patients attending the ED.²⁰ The intervention, which focused on resource-intensive clinician education, provision of heat packs and department-level audit and feedback, reduced opioid initiation, but it did not change imaging rates. Tsega et al found adding cues to eMR request forms could reduce use of lumbar X-rays in ED, but use of advanced imaging was unchanged or increased.²¹ The success of behaviour change interventions to reduce low-value care has therefore been mixed, and the evidence for reliable strategies to implement in healthcare settings is uncertain. Key factors yet to be addressed in these interventions are the beliefs and biases that can drive decision making.²²

Insights from behavioural economics

Research from the field of behavioural economics suggests that decision making, including clinical decision making, can be influenced by context, heuristics and cognitive biases.^{23 24} For example, Prospect Theory proposes that despite clinicians being highly skilled and knowledgeable professionals, they are still susceptible to cognitive biases.²⁵ There are many examples of humans relying on heuristics, or rules of thumb, to make decisions.²⁵ In some contexts, these heuristics can lead to biases, systematic errors or suboptimal decision making. A systematic review of 213 studies identified 19 types of heuristics and biases that were present in patients’ and clinicians’ medical decision making.²⁶ Seventy-three studies looked at decision making in clinicians, with 80% (n=51) of these finding a heuristic or bias present in clinical decision making.²⁶ Similarly, 140 studies looked at patients, with 61% (n=86) of studies finding a heuristic or bias present in healthcare decisions.

Therefore, it is important to understand the heuristics and biases that may affect decision making in the ED when designing interventions to reduce low-value care. The heuristics and biases presented in table 1 have been researched in clinical decision making and are particularly relevant to imaging and opioid prescribing at discharge. These heuristics and biases could help explain the persistence of low-value care despite clinician education and awareness campaigns but may also offer possible solutions.

Table 1

Relevant heuristics and biases

Heuristic or bias	Description
Ambiguity aversion	Ambiguity aversion describes when the decision maker will pursue testing to increase certainty, even if not recommended, or avoid making decisions with ambiguity.23 56 For example, clinicians with a low tolerance to ambiguity may order unnecessary imaging tests for patients with uncomplicated back pain to increase clinical information before making a diagnosis.57
Commission bias	Commission bias describes when clinicians prefer to provide something (eg, a test or treatment), rather than nothing, even when nothing is the clinically appropriate decision.26 58 For example, a cross-sectional survey study emergency physicians found that 97% reported that they had personally ordered some unnecessary imaging.59 In addition, a survey of Australian hospital pharmacists (n=135) found that even when patients have not needed opioid analgesics in the past 48 hours before discharge, over 70% provided a take home opioid ‘just-in-case’.
Default or status quo bias	Default or status quo bias describes when the structure or complexity of a decision leaves clinicians opting to maintain the existing course of action, even when it is an inferior choice.27 For example, reducing the default quantity populated in the eMR for opioids can modestly reduce the number of opioids prescribed.60
Framing effect	Framing effect describes when decision makers’ preferences and judgements are influenced by how information is described or framed.27 For example, two hospital field experiments to increase handwashing found that framing hand hygiene in terms of benefits to the health of the patients was more effective than framing in hand hygiene in terms of benefits to the health of the doctors.44
Present bias	Present bias describes when people value immediate benefits or harms, over the long-term consequences of their decisions.61–64 For example, a clinician may request imaging and/or opioids because they overweight the immediate benefits (such as increased patient satisfaction and reduced pain scores) over the potential long-term harms (such as overdiagnosis and opioid dependence).22 65

eMR, electronic medical record.

The use of nudges

‘Nudges’ can leverage heuristics and biases to improve decision making to increase guideline concordant care. Nudges are changes to the way choices are presented or structured that can alter decision making without restricting or prohibiting options.²⁷ Nudges are a light-touch, low-cost and scalable way to align care with clinical guidelines.^{28 29} A systematic review that included 28 RCTs found that computer reminders delivered to clinicians during their routine activities improved protocol adherence by a median of 4.2%, with some trials reporting larger effects. In addition, a recent systematic review that included 42 RCTs examining the use of nudges such as social norms, defaults and reminders found that 86% of them were effective at improving guideline concordance.²⁹ Several of these techniques could be implemented in ED settings to align care with clinical guidelines.

Identifying the most relevant nudge strategies and understanding their implementation in a broader healthcare context is essential.³⁰ Jesse and Jannach³¹ proposed a taxonomy of nudging mechanisms that included decision information, decision assistance, decision structure and social decision appeal.³¹ Decision information supports decision makers by simplifying complex information and highlighting decision consequences.²⁸ Decision assistance brings forward information that clinicians are likely aware of but may forget to consider in each clinical judgement.²⁹ Changing the decision structure can involve increasing the effort and friction for guideline discordant decisions and reducing it for guideline concordant decisions. It can also include suggesting guideline concordant alternatives or substitutes. Social decision appeal leverages the power of messenger effects and social norms. These nudging mechanisms should be considered in the broader decision-making context when designing nudge interventions.

Given both clinicians and patients can be influenced by decision-making biases, there is an important role for both patients and clinicians to be involved in improving healthcare decisions. However, there is little evidence about to what nudge strategies are effective at reducing low-value care in the ED. In addition, it is not known if directing nudges towards patients, clinicians or both is most effective. To our knowledge, this will be the first RCT to test the impact of scalable clinician-directed and patient-directed behavioural nudges on reducing low-value care for back pain in the emergency department (ED).

Study aims

Primary aim

To determine, for people with back pain due to a musculoskeletal condition presenting to ED, the effectiveness of patient nudges, clinician nudges or both interventions compared with no nudge intervention on reducing encounters involving low-value care (non-indicated lumbar imaging test, opioid at discharge or both).

Secondary aims

The secondary objectives of this study are:

• To determine if the interventions lead to non-inferior short-term patient-reported outcomes (satisfaction with care, worry, pain, function, quality of life) compared with a no nudge control group.

• To calculate the cost-effectiveness of the interventions compared with a no nudge control group.

• To evaluate unintended consequences (representation rate, readmission rate).

• To explore patient and clinician experiences of the interventions.

Methods and analysis

Study design

NUDG-ED is a 2×2 factorial, open-label, before-after, cluster RCT design (figure 1). This involves randomising eight hospital EDs to one of four groups after a 3-month period where all sites are in a no nudge control group. We will require 2416 encounters for back pain due to a musculoskeletal condition across 8 sites over a 9-month study period (3-month before period and 6-month after period). NUDG-ED is planned to commence in April 2024, ending in December 2024.

Figure 1. 2×2 factorial, open-label, before-after, cluster randomised controlled trial design.

We used the Standard Protocol Items: Recommendations for Interventional Trials checklist to report the protocol.³²

Study setting

The participating hospitals are located across three Local Health Districts (LHDs) in Sydney (Western Sydney LHD, Nepean Blue Mountains LHD, Southwestern Sydney LHD). These are public hospitals in culturally and linguistically diverse, metropolitan areas that provide emergency care for general medical conditions. Between 20% and 50% of residents across these LHDs were born overseas and approximately 50% speak a language other than English at home. The regions of Western and Southwestern Sydney have higher than average levels of socioeconomic disadvantage compared with the state of New South Wales.³³ Most EDs in Australia are publicly funded and have no out-of-pocket expenses for Australian citizens and most permanent residents to attend.

Site recruitment

We adopted a pragmatic approach to site selection. We recruited sites based on the capacity of the sites to make eMR changes. To participate in the trial, sites were required to have an eMR system that could identify patient participants based on their presenting problem and provide a specific eMR pop-up alert when a clinician orders imaging or opioids for these patients.

Randomisation and group allocation

We will use cluster randomisation because the intervention will be at the hospital level. We will randomly allocate the eight hospitals (clusters), into one of four groups: (i) patient nudges; (ii) clinician nudges; (iii) patient nudges and clinician nudges, (iv) no nudge control. To create the randomisation list, a trial statistician will use computer-generated random numbers.

Participants and recruitment

Clinician participants

Clinician participants will be ED clinicians at study sites who are involved in the care of patients presenting to the ED with a primary complaint of low back pain. This includes physicians (Junior Medical Officer, Registrar, Consultant, Career Medical Officers), nurses and physiotherapists. After the intervention period, ED Directors will invite all ED clinicians to complete a survey and a subset to participate in semi-structured interviews. We will be using a purposive sampling approach for clinician interviews to ensure a diverse range of clinician experience, age, gender and interest in the management of back pain. We do not have capacity to collect data on individual clinician characteristics because these details are not recorded in the current eMR system.

Patient participants

For the health service measures, including our primary outcome, patient participants will be adults aged 18 years or over who present to the ED during the study period with back pain due to a musculoskeletal condition. We will invite a sub-sample of patients who present to ED with low back pain during the study, to participate in a follow-up period via text message survey. To confirm eligibility, clinician investigators will use the eMR to identify people diagnosed with back pain due to a musculoskeletal condition using a list of codes from Systematised Nomenclature of Medicine Clinical Terms Australia (SNOMED-CT AU) (online supplemental table 1). Information about the study and how to opt out will be displayed on the 55-inch advertising screen located in the waiting room. Patient participants will not be invited to complete the survey if they were diagnosed with a non-musculoskeletal condition e.g. renal colic, if they did not have a valid mobile phone number on record, if they required a translator, or if they opted out of the patient survey in the ED waiting room.

Patient and public involvement

Patient nudges have been co-designed with patients, public and clinicians. Two consumer advisors have been appointed to support throughout key stages of the research including reviewing the protocol and patient-reported outcome measures, analysing patient survey feedback and interpreting results.

Sample size

A previous study of patients presenting to ED with low back pain suggests 31.7% (95% CI 22.9 to 41.6)³⁴ received opioid prescription at discharge and 30.3% (95% CI 23.7 to 38.0) received non-indicated lumbar imaging.³⁵ To detect an effect of the patient nudges or clinician nudges on the number and proportion of encounters involving low-value care, with an absolute difference of 10% (eg, event rate 30% in the control hospitals vs event rate 20% in intervention hospitals) and with 80% power, alpha set at 0.05, assuming an intraclass correlation coefficient (ICC) of 0.10 and an intraperiod correlation (IPC) of 0.09 (ie, between the before and after periods, within each site) and accounting for variable cluster sizes, we would require 2416 encounters for back pain due to a musculoskeletal condition across 8 sites, over a 9-month trial period (ie, ~302 encounters per site, over 3-month baseline and 6-month intervention period). Our sample is based on an achievable and conservative estimate that IPC is less than ICC and assumes no loss to follow-up. Losses to follow-up are very unlikely because our primary outcome is based on routinely collected health service data.

For patient-reported outcomes, we calculated power based on the mean of five items related to ‘Overall Assessment of ED Experience’ of the Press Ganey Survey (range 1–5), at 1-week follow-up. We chose this measure because patient experience is a key priority for hospitals in Australia and have powered the study to detect a meaningful drop in patient experience due to either intervention. We assumed a mean of 4.2 points, an SD of 1.9,³⁶ a non-inferiority margin of 0.5 point (ie, 0.5 units is the maximum acceptable drop in patient experience), an IPC and ICC of 0.01, the minimum required sample size for 80% power was 57 patients per site, over the 6-month intervention period (456 patients in total). The expected response rate of the survey is around 30-50%.²⁰

Blinding

It is not possible to blind patient participants to the nudge interventions, although measures will be taken to reduce performance bias, for example, masking patients to the study hypothesis. At least one member of the research team and a blinded statistician will be unaware of site allocation. Outcome assessors and trial staff reviewing clinical notes will also be blind to site allocation. Clinicians at all sites will receive an email from their ED director endorsing the trial and explaining their group allocation at the start of the trial. To investigate potential Hawthorne effects, we will compare use of low-value care in the no nudge control sites before and after the email from the ED director notifying staff of the trial (see ‘Intervention delivery’ section). Statistical analysis and interpretation will also be performed blind to group allocation. Unblinding of the statistician and independent trial staff will occur once data analysis and interpretation are complete.

Interventions

Patient nudges: making decision information salient

Patient nudges will include six digital decision information posters displayed on 55-inch LCD screens (patient nudge A—figure 2) and a decision information brochure (patient nudge B—figure 3) based on behavioural economics theory (see ‘Theory and evidence behind the patient nudges’ section). Patients can access patient nudge B using their smartphone (via the QR code on the digital posters) or a paper version that will be stocked in a brochure holder attached to the LCD screen. ‘Scan here for more information’ is on each poster to direct patients to the patient information brochure. A full list of intervention materials included is shown in online supplemental figure 1.

Figure 2. Patient nudge A—example of a decision information poster targeting opioids and imaging for back pain. These digital posters will be displayed on LCD screens, these will also include QR codes linking to patient nudge B, the decision information brochure.

Figure 3. Patient nudge B—example of smartphone-based decision information brochure. Patients will have access to this information via a QR code or printed decision information brochures.

All decision information posters will be translated into Arabic, simple Chinese and Vietnamese. Up to two non-English language posters will be included in rotation in each of the sites based on the relevant LHD data on languages most spoken.

Theory and evidence behind the patient nudges

These patient information materials were originally designed by an ad agency and further refined by investigators using pilot studies and behavioural economic theory.³⁷ The interventions use several behavioural science techniques that can be grouped under broad categories in the nudge taxonomy by Jesse and Jannach.³¹ The patient nudges are making decision information visible for patients in the ED waiting room. They provide simplified information that is directly relevant to immediate next steps for patients with low back pain. The salience of this information is increased by presenting the relevant messages on attractive, attention-grabbing posters that are presented on a large, prominently placed LCD screens in the ED waiting room.²⁸ The patient nudges will frame the decision consequences in terms of the lack of benefits and the immediate potential harms of imaging and opioids, rather than long-term harms (online supplemental figure 1).

Social norms and messenger effects can influence how people receive information.^{38 39} A systematic review of nudges to improve clinical decision making found that in 16 of the 17 RCTs that used social norm and messenger effect resulted in improved clinical decision making.²⁹ This could be due to the authority, trustworthiness, relatability or perceived knowledge of the messenger. However, there are contexts in which the messenger has no effect.⁴⁰ In addition, a meta-analysis that examined 297 studies found that the use of descriptive norms (describing what most other people do) can be effective at directly influencing behaviour.⁴¹ Attempting to leverage these potential messenger effects, two of the six decision information posters will include the image of two physicians (one male, one female) to convey credibility and trustworthiness of the information. In addition, two posters will leverage descriptive social norms. One poster that says ‘most people’ will experience as much pain relief from anti-inflammatories, as they do from opioids, with fewer side effects, and the other poster states that back scans are not helpful for ‘most people’ with low back pain.

Suggested alternatives involve increasing the salience of more preferable options that users can substitute the targeted behaviour with.³¹ We have suggested that instead of using opioids patients can use evidence-based management techniques including gentle movement, using heat, over-the-counter medication and giving time for recovery. These nudges aim to provide patients with information to help them to better understand the costs and benefits or imaging and opioids.³⁷ Our randomised proof-of-concept study found that, compared with standard care, these patient nudges reduced intention to request imaging by 1 point on a 10-point scale (95% CI −1.6 to −0.4).⁴²

The cues to action used in the patient nudge intervention aim to encourage patients to start a conversation about their care options with their ED clinician.³⁷ Information is provided at a time just before making care decisions with their clinician. Given its relevance and timing to patients with back pain, the information should be readily available in their memory and can be used to inform shared decision-making discussions with their clinician.

Clinician nudges: computer alerts

The clinician nudges were informed by a pilot study we conducted with primary care physicians.⁴³ The interventions include three small changes to the current computerised order systems that are triggered only for patients who are flagged in the workflow as presenting with low back pain.

Clinician nudge A (figure 4) is a behaviourally informed computer alert (see ‘Theory and evidence behind the clinician nudges’ section) that appears when a clinician attempts to order imaging test for patients recorded as presenting with back pain. This interrupts habitual ordering and provides decision assistance by reminding clinicians that imaging is not recommended without features of serious pathology and providing them clear guidance on what pathologies would warrant imaging. The prompt will also provide them the key information from the patient decision information brochure to support a conversation with the patient about preferred care for low back pain (see online supplemental figure 1). If the clinician does suspect serious pathology or wish to continue, they will be able to do so as normal. If they decide imaging is not necessary, they will be able to cancel the order.

Figure 4. Clinician nudge A—computer alert when clinicians try to order imaging for low back pain. The pop-up reminds clinicians imaging is not recommended and includes a clickable link to the decision information brochure.

Clinician nudge B (figure 5) is also a behaviourally informed computer alert that appears when a clinician attempts to administer an opioid medicine for a person with back pain. Before proceeding with the administration, the alert reminds clinicians that opioids are not recommended for back pain and provides a list of suggested non-steroidal anti-inflammatory drugs (NSAIDs) to choose from instead.¹³ The nudge for ordering an opioid while a patient is in the ED will be muted for every second order to reduce alert fatigue. Clinicians who attempt to order or prescribe opioids for a person diagnosed with uncomplicated back pain will receive an alert reminding them that take-home opioids are not recommended. It also provides evidence-based advice for the patients at home care (figure 5).

Figure 5. Clinician nudge B—computerised alert when clinicians try to administer an opioid medication for back pain. The pop-up reminds clinicians opioids are not recommended and provides suggested alternatives for more appropriate care. BD, two times a day; IM, intramuscular; IV, intravenous; NSAID, non-steroidal anti-inflammatory drug; PO, orally.

No nudges will restrict clinicians testing or treatment options; it is still within their scope to provide imaging or opioids if they deem it suitable. The nudges disrupt habitual ordering and provide decision information. Our outcome relies on routine recording practices of clinicians at participating hospitals and could be affected by differences in recording across the eight sites. We do not have capacity to collect data regarding other interventions patients received while in ED (eg, education or manual therapy) because these data are not provided in coded format for extraction from the eMR.

Theory and evidence behind the clinician nudges

The clinician nudges change the decision structure and provide assistance to clinicians at the time of decision making.²⁸ They take the form of active choice alerts that disrupt the habitual flow of test and/or opioid ordering, increase the option-related effort and provide just-in-time information to support decision making. Both nudges remind clinicians that opioids and imaging may not be appropriate for musculoskeletal back pain.⁴⁴ The opioid alert provides clinicians suggested alternatives to remind clinicians that safer care alternatives are available for them to provide which may alleviate commission bias effects (figure 5).^{31 45} The lumbar imaging alert reminds the clinician of the red flags that would indicate imaging is required and provides them information on the lack of benefits and safer care options and a cue to discuss it with the patient. It also provides a shared decision-making prompt and supporting information that can be discussed with patients on when imaging may be helpful, when it is not and what the evidence says works for treating uncomplicated back pain. Clinicians may override the alerts and continue to order imaging or opioids if clinically indicated.

No nudge control

The no nudge control group will receive an email communication from the ED Director at the start of the NUDG-ED intervention period notifying them of the trial. No nudge interventions will be delivered to these hospitals. Waiting room screens will display standard hospital messaging, without any content on back pain.

Intervention delivery

ED clinician participants in the intervention and control groups will be officially notified by their ED Director of the start date and aims of the trial, and to give explicit support for the trial. Nudges that aim to support improved clinical decision making may be more widely accepted and reduce backfire effects if there is transparency about the purpose and intent of the nudges.⁴⁶ A study by Loewenstein et al ⁴⁶ found that explicitly disclosing a nudge did not meaningfully affect the impact of the nudge. Clinicians may also value the nudges more knowing that ED Directors have been involved in co-designing the intervention materials to ensure they are suitable for their ED context.³⁷

An LCD advertising screen (55-inch) will be newly installed in the ED waiting room of each participating hospital. In hospitals randomised to receive the patient nudge intervention, the screen will begin displaying the decision information posters in the form of a slideshow on loop (~10 s per poster). At hospitals not allocated to the patient nudge intervention group, the screens will display standard hospital messaging. Investigators can launch and monitor the fidelity of the patient nudges remotely via web analytics and LCD advertising screen content management system. Hospitals randomised to the clinician nudges will launch the computer alerts in the ED eMR. Interventions will run for 6 months at each intervention site.

Outcome measures

Primary outcome

The primary outcome will be use of low-value care, defined as the proportion of encounters for back pain due to a musculoskeletal condition where a person received a non-indicated lumbar imaging test, an opioid at discharge or both, in the ED over a 9-month period. We chose this composite outcome because it is a meaningful metric to ED clinicians. Clinician researchers will perform chart reviews every month for all participants who present with low back pain and receive imaging to understand and code if it was non-indicated imaging (ie, imaging provided in the absence of clinical features of serious pathology) using reliable methods we have published.³⁵ Opioids provided or prescribed at discharge for patients diagnosed with ‘non-serious’ low back pain (ie, low back pain with non-specific cause or low back pain with neurological signs and symptoms—see online supplemental table 1) will be coded as low value.

For patient participants who received imaging, the following sections of the clinical chart will be examined, and used to code the primary outcome (online supplemental figure 2):

• Order comment

• Clinical or case notes

• Reason for imaging request

• ED discharge letter

• ED imaging report

• Hospital admission report

A clinician researcher will screen the clinical charts of participants who received lumbar imaging and identify if there were documented clinical indications. To do this, a checklist of guideline-endorsed imaging indications will be completed by the reviewing clinician via a standardised chart review in Research Electronic Data Capture (REDCap). Approved chart reviewers (clinician researchers) will receive a list of participants in the trial whose charts require review. Indications for imaging will be based on international clinical guidelines⁸ and our previous work on coding the appropriateness of imaging for low back pain in ED (online supplemental figure 2).³⁵

If patients are diagnosed with ‘non-serious’ musculoskeletal low back pain (i.e., they have a SNOMED-CT AU code corresponding with either category (1) low back pain with non-specific cause or category (2) low back pain with neurological signs and symptoms; see online supplemental table 1) and they receive opioids at discharge, this will be coded as low-value care. This decision was informed by advice from the ED clinicians in our team.

A 6-month intervention period could be a limitation of the study design as we will not capture potential seasonal variations in ED care. A time series analysis of Western Australia metropolitan EDs showed variation in the number and case mix of patient presentations over the course of the year, with peaks in winter.⁴⁷

Secondary outcomes

Patient-reported outcomes

A number of patient-reported outcome measures will be collected from a minimum of 456 patients up to 1 week after their index ED visit in the 3-month before period and 6-month after period. These measures include:

• Patient experience: two items related to ‘Overall Assessment of ED Experience’ and two items from ‘Medical Provider’ from 36-item Press Ganey ED Survey.⁴⁸

• Pain intensity: Numeric Pain Rating Scale,⁴⁹ the pain duration question from Orebro Musculoskeletal Pain Questionnaire⁵⁰ and disability measured using the 2008 adaptation of item 8 of the 36-Item Short Form Health Survey by Henschke et al.⁵¹

• Health-related quality of life: EQ-5D-5L health-related quality of life indicators.⁵²

• Reassurance: generic reassurance subscale from Consultation-based Reassurance Questionnaire.⁵³

• Patient participation in decision making: CollaboRATE Tool.⁵⁴

• Referrals to specialist.

• Intention to seek second opinion: one item from the National Patient Safety Foundation established by the American Medical Association.⁵⁵

• Patient beliefs about imaging and opioids: items 13 and 14 from the survey by Jenkins et al ¹⁷ and a new statement on patient beliefs on the effectiveness of opioids.

See online supplemental file 1 for full list of patient-reported outcome measures.

Process measures

We will be evaluating patient beliefs, patient reassurance and perceived helpfulness of the computer alerts as potential mediators of the intervention effect.

Service outcomes

• Proportion of patients admitted to hospital (excludes patients sent to the ED short stay units).

• Proportion of patients who receive advanced lumbar imaging tests (CT/MRI=yes, X-ray/no imaging=no).

• Time in the ED (triage time to the ED discharge or admission time, including the time in short stay units).

• Hospital costs (including intervention costs, ie, LCD screens, installation costs, staff time, IT support costs), cost-effectiveness.

• Use of opioids in the ED (eMeds).

Fidelity measures

• Clinicians’ awareness and opinion of interventions. See online supplemental file 2 for full list of clinician survey questions.

• Patient engagement (use of QR code on ED waiting room screen).

Unintended consequences

• Proportion of patients representing with low back pain to the index ED within 48 hours (this aligns with hospital performance indicators and is readily captured in hospital administrative data).

• Proportion of patients with unintended 30-day representation.

• Proportion of patients who are readmitted.

• Proportion of patients who left the ED without treatment.

• Proportion of patients diagnosed with non-musculoskeletal pain who were administered an opioid.

Experiences with interventions

Semi-structured interviews (15 min via Zoom) with patients and clinician participants at each site to discuss implementation (usage, reactions to and awareness of interventions) (n=~40); we will attempt to capture clinician beliefs about the usefulness and impact of the interventions. We are particularly interested if the nudges and information helped patients and clinicians in their decision making. We are asking clinicians if they noticed the alerts and if they found them appropriate and useful. We are also asking if clinicians had patients refuse imaging or opioids when they were offered.

Data collection methods

Data collection

We will use Discern Analytics to build a standardised query to extract the routinely collected health service delivery data at participating sites from Cerner software. In the 3-month control period and during the intervention period, the same health service delivery measures will be extracted from all sites, every week, until the end of the trial at 6 months of follow-up. This collection system has worked well in our previous trial²⁰ and avoids additional workloads for ED staff. We will collect patient-reported outcomes from a random subsample of patients who presented during the 9-month trial period (n=456 or ~12% of sample). We will use automated text messaging to contact patients 1 week after the index ED visit. Participants will be referred to a brief self-reported online questionnaire.

Health-service data

Using electronic clinical charts, a data manager at each of the participating LHDs will extract re-identifiable data on people presenting with back pain (see online supplemental figure 2 for specific items), to a REDCap database. REDCap is suitable for ‘highly protected’ data—data are stored securely, is encrypted in transit and at rest. Only approved study investigators will have access to identifiable data using a personal REDCap login and password. We will de-identify data for analysis and store the de-identified dataset in the University of Sydney’s Research Data Store.

Clinical notes and coding of appropriateness

Clinical notes will be accessed by approved NUDG-ED investigators using remote access to the eMR. For all other participants, only the discharge summary will be extracted from the eMR to the main study datasheet. Clinician investigators who review the appropriateness of imaging will use a standardised survey in REDCap to extract data and score appropriateness. This process is informed by our previous work.³⁵ We anticipate we will require 10–20 clinicians to assist with case note reviews.

Patient-reported data

We will collect patient-reported data via a secure web application, REDCap. We will send participants a text message invitation via Twilio to participate and a REDCap link to complete an online survey.

Statistical methods

Data analysis will be blinded, by intention-to-treat and guided by a published statistical analysis plan. Analysis will be conducted by an independent biostatistician and checked for accuracy.

• Primary analysis: to evaluate the effect of the intervention on the proportion of encounters for back pain due to a musculoskeletal condition where low-value care was provided, we will use a multilevel regression model, with a random effect for cluster and patient (assuming some patients may have several encounters during the study period), a fixed effect indicating the group assignment of each cluster and a fixed effect of time.

• Secondary analysis: dichotomous outcomes will be compared between groups using generalised estimating equations (GEE) considering clustering effects. Continuous secondary outcomes will be analysed using the same GEE model with appropriate link function.

• Cost-effectiveness analysis: cost-effectiveness analysis of the NUDG-ED interventions compared with current emergency care will be done from the health system perspective. For this, we will measure all costs related to the delivery of the intervention (ie, LCD screens, installation costs, staff time, printed resources, IT support). We will also calculate the costs of imaging and opioid use in control and intervention groups. We will present the incremental cost-effectiveness ratio as the incremental cost per patient avoiding low-value care. We will also estimate the incremental cost per quality-adjusted life year gained, using utility weights from the EQ-5D-5L.

• Qualitative analysis: we will conduct a framework analysis to explore experiences with the nudge interventions.

Auditing

We have not planned a formal audit. However, we will arrange independent auditing of the trial process and documents if needed. This trial is registered with ANZCTR (ACTRN12623001000695) and subject to the usual audits for clinical trials in Australia.

Data monitoring committee

This study does not require a data monitoring committee because it does not focus on life-threatening diseases, vulnerable populations or potentially harmful experimental interventions.

Ethics and dissemination

This study has ethical approval from Southwestern Sydney LHD Human Research Ethics Committee (2023/ETH00472). The trial received a waiver of informed consent because: (a) it is impractical to consent all the clinicians and patients and the project could not practicably be done if informed consent were required; (b) as a behavioural intervention, the requirement for patients or clinicians to consent to participation would invalidate the scientific validity of the experiment or (c) both. Participant information and consent forms are included in the online supplemental files 3 and 4. Any important protocol modifications will be reported to investigators, research ethics committee/institutional review boards, trial registries, journals and trial regulators. All authors will be required to meet the International Committee of Medical Journal Editors criteria. We will disseminate the results of this trial via media and presenting at conferences and publications in scientific journals.

Thank you to all those from the NHMRC Clinical Trials Centre who have contributed to NUDG-ED.

Data availability statement

No data are available.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study has ethical approval from Southwestern Sydney Local Health District Human Research Ethics Committee (2023/ETH00472).

Footnote

Twitter @GemmaAltinger, @SweekritiSharma, @jeffreylinder, @Scientosis

Collaborators NUDG-ED Study Group members: Jeremy Lawrence, Kevin Pile, Richard Cracknell, Arsalan Hermiz, Francisco Moncada, Daryn Mitford, Mark Salter, James Mallows, Raymond Morgan, Cindy Hastings, Richard McNulty, Alexandra Frost, Belinda Burns, Kelly Bivona, Jordan Fenech, Helen Zaouk, Matthew Smith, Ahilan Parameswaran, Jenny Morris, Brendon Shapter, Daniel Van Vorst, Peter Squire, Jim Basilakis, Michael Meller, Elise Tcharkhedian, Gustavo Machado, James McAuley, Janet Harrison, Zoe Michaleff, Aidan van Wyk, Wade McKeown, Jo Davis, Eric Ho.

Contributors GA, SS and ACT drafted the protocol. CM, LC, KMcC, JL, RB, IH, EC, QL, KH, AC, PMM, NG, IF, TC, KT and AV provided feedback and approved the protocol. QL provided statistical advice. KT and AV also provided consumer perspectives on the research documents and the nudges.

Funding This work is supported by the Clinical Trials and Cohort Studies Grant, Australian National Health and Medical Research Council (ID 2015173). JL is supported by grants from the United States National Institute on Aging (P30AG059988, R01AG069762, R01AG074245, P30AG024968, R01AG070054, R33AG057395), National Heart, Lung and Blood Institute (R01HL167023) and the Agency for Healthcare Research and Quality (R01HS026506, R01HS028127).

Competing interests None declared.

Patient and public involvement Patients were involved in the design of this research. Refer to the 'Methods' section for further details.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Brownlee S, Chalkidou K, Doust J, et al. Evidence for overuse of medical services around the world. Lancet 2017; 390: 156–68. doi:10.1016/S0140-6736(16)32585-5

² Elshaug AG, Watt AM, Mundy L, et al. Over 150 potentially low-value health care practices: an Australian study. Med J Aust 2012; 197: 556–60. doi:10.5694/mja12.11083

³ Furlan L, Francesco PD, Costantino G, et al. Choosing wisely in clinical practice: embracing critical thinking, striving for safer care. J Intern Med 2022; 291: 397–407. doi:10.1111/joim.13472

⁴ James SL, Abate D, Abate KH, et al. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the global burden of disease study 2017. The Lancet 2018; 392: 1789–858. doi:10.1016/S0140-6736(18)32279-7

⁵ Edwards J, Hayden J, Asbridge M, et al. Prevalence of low back pain in emergency settings: a systematic review and meta-analysis. BMC Musculoskelet Disord 2017; 18: 143. doi:10.1186/s12891-017-1511-7

⁶ Buchbinder R, van Tulder M, Öberg B, et al. Low back pain: a call for action. The Lancet 2018; 391: 2384–8. doi:10.1016/S0140-6736(18)30488-4

⁷ Ferreira GE, Machado GC, Abdel Shaheed C, et al. Management of low back pain in Australian emergency departments. BMJ Qual Saf 2019; 28: 826–34. doi:10.1136/bmjqs-2019-009383

⁸ Chou R. Diagnostic imaging for low back pain: advice for high-value health care from the American college of physicians. Ann Intern Med 2011; 154: 181. doi:10.7326/0003-4819-154-3-201102010-00008

⁹ McCaughey EJ, Li L, Georgiou A, et al. Imaging for patients presenting to an emergency Department with back pain: impact on patient pathway. Emerg Med Australas 2016; 28: 412–8. doi:10.1111/1742-6723.12602

¹⁰ Kyi L, Kandane-Rathnayake R, Morand E, et al. Outcomes of patients admitted to hospital medical units with back pain. Intern Med J 2019; 49: 316–22. doi:10.1111/imj.14075

¹¹ Jarvik JG, Hollingworth W, Martin B, et al. Rapid magnetic resonance imaging vs Radiographs for patients with low back pain: a randomized controlled trial. JAMA 2003; 289: 2810–8. doi:10.1001/jama.289.21.2810

¹² AIHW. Opioid harm in Australia: and comparisons between Australia and Canada. 2018. Available: https://tinyurl.com/4eawe96f

¹³ Abdel Shaheed C, Maher CG, Williams KA, et al. Tolerability, and dose-dependent effects of opioid Analgesics for low back pain: A systematic review and meta-analysis. JAMA Intern Med 2016; 176: 958–68. doi:10.1001/jamainternmed.2016.1251

¹⁴ Qaseem A, McLean RM, O’Gurek D, et al. Nonpharmacologic and pharmacologic management of acute pain from non-low back, musculoskeletal injuries in adults: A clinical guideline from the American college of physicians and American Academy of family physicians. Ann Intern Med 2020; 173: 739–48. doi:10.7326/M19-3602

¹⁵ Jones CMP, Day RO, Koes BW, et al. Opioid analgesia for acute low back pain and neck pain (the OPAL trial): a randomised placebo-controlled trial. Lancet 2023; 402: 304–12. doi:10.1016/S0140-6736(23)00404-X

¹⁶ Traeger AC, Buchbinder R, Elshaug AG, et al. Care for low back pain: can health systems deliver Bull World Health Organ 2019; 97: 423–33. doi:10.2471/BLT.18.226050

¹⁷ Jenkins HJ, Hancock MJ, French SD, et al. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ 2015; 187: 401–8. doi:10.1503/cmaj.141183

¹⁸ French SD, Green S, Buchbinder R, et al. Interventions for improving the appropriate use of imaging in people with musculoskeletal conditions. Cochrane Database Syst Rev 2010: CD006094. doi:10.1002/14651858.CD006094.pub2

¹⁹ Shoemaker-Hunt SJ, Wyant BE. The effect of opioid stewardship interventions on key outcomes: a systematic review. J Patient Saf 2020; 16: S36–41. doi:10.1097/PTS.0000000000000710

²⁰ Coombs DM, Machado GC, Richards B, et al. Effectiveness of a multifaceted intervention to improve emergency department care of low back pain: a stepped-wedge, cluster-randomised trial. BMJ Qual Saf 2021; 30: 825–35. doi:10.1136/bmjqs-2020-012337

²¹ Tsega S, Krouss M, Alaiev D, et al. Imaging wisely campaign: initiative to reduce imaging for low back pain across a large safety net system. J Am Coll Radiol 2024; 21: 165–74. doi:10.1016/j.jacr.2023.07.012

²² Sharma S, Traeger AC, Reed B, et al. Clinician and patient beliefs about diagnostic imaging for low back pain: a systematic qualitative evidence synthesis. BMJ Open 2020; 10: e037820. doi:10.1136/bmjopen-2020-037820

²³ Scott IA, Soon J, Elshaug AG, et al. Countering cognitive biases in minimising low value care. Med J Aust 2017; 206: 407–11. doi:10.5694/mja16.00999

²⁴ Oakes AH, Radomski TR. Reducing low-value care and improving health care value. JAMA 2021; 325: 1715–6. doi:10.1001/jama.2021.3308

²⁵ Verma AA, Razak F, Detsky AS. Understanding choice: why physicians should learn prospect theory. JAMA 2014; 311: 571–2. doi:10.1001/jama.2013.285245

²⁶ Blumenthal-Barby JS, Krieger H. Cognitive biases and Heuristics in medical decision making: A critical review using a systematic search strategy. Med Decis Making 2015; 35: 539–57. doi:10.1177/0272989X14547740

²⁷ Goldstein D. The Behavioral Economics Guide 2022.

²⁸ Münscher R, Vetter M, Scheuerle T. A review and Taxonomy of choice architecture techniques. Behavioral Decision Making 2016; 29: 511–24. doi:10.1002/bdm.1897 Available: https://onlinelibrary.wiley.com/toc/10990771/29/5

²⁹ Yoong SL, Hall A, Stacey F, et al. Nudge strategies to improve Healthcare providers’ implementation of evidence-based guidelines, policies and practices: a systematic review of trials included within Cochrane systematic reviews. Implement Sci 2020; 15: 50. doi:10.1186/s13012-020-01011-0

³⁰ Fox CR, Doctor JN, Goldstein NJ, et al. Details matter: predicting when nudging Clinicians will succeed or fail. BMJ 2020; 370: m3256. doi:10.1136/bmj.m3256

³¹ Jesse M, Jannach D. Digital nudging with Recommender systems: survey and future directions. Computers in Human Behavior Reports 2021; 3: 100052. doi:10.1016/j.chbr.2020.100052

³² Chan A-W, Tetzlaff JM, Gøtzsche PC, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 2013; 346: e7586. doi:10.1136/bmj.e7586

³³ Sydney SW. South West Sydney: Our Health in Brief

³⁴ Coombs DM, Maher CG, Collett M, et al. Continued opioid use following an emergency Department presentation for low back pain. Emerg Med Australas 2022; 34: 694–7. doi:10.1111/1742-6723.13979

³⁵ Traeger AC, Machado GC, Bath S, et al. Appropriateness of imaging decisions for low back pain presenting to the emergency Department: a retrospective chart review study. Int J Qual Health Care 2021; 33: mzab103. doi:10.1093/intqhc/mzab103

³⁶ Schwartz TM, Tai M, Babu KM, et al. Lack of association between press Ganey emergency Department patient satisfaction scores and emergency Department administration of analgesic medications. Ann Emerg Med 2014; 64: 469–81. doi:10.1016/j.annemergmed.2014.02.010

³⁷ Sharma S, Traeger AC, Tcharkhedian E, et al. Effect of a waiting room communication strategy on imaging rates and recall of public health messages for low back pain. Int J Qual Health Care 2021; 33: mzab129. doi:10.1093/intqhc/mzab129

³⁸ Cialdini RB, Goldstein NJ. Social influence: compliance and conformity. Annu Rev Psychol 2004; 55: 591–621. doi:10.1146/annurev.psych.55.090902.142015

³⁹ Lipari F. This is how we do it: how social norms and social identity shape decision making under uncertainty. Games 2018; 9: 99. doi:10.3390/g9040099

⁴⁰ Favero N, Jilke S, Wolfson JA, et al. Messenger effects in COVID-19 communication: does the level of government matter Health Policy Open 2021; 2: 100027. doi:10.1016/j.hpopen.2020.100027

⁴¹ Melnyk V, van Herpen E, Jak S, et al. The mechanisms of social norms' influence on consumer decision making: A meta-analysis. Zeitschrift Für Psychologie 2019; 227: 4–17. doi:10.1027/2151-2604/a000352

⁴² Sharma S, Traeger AC, O’Keeffe M, et al. Effect of information format on intentions and beliefs regarding diagnostic imaging for non-specific low back pain: a randomised controlled trial in members of the public. Patient Educ Couns 2021; 104: 595–602. doi:10.1016/j.pec.2020.08.021

⁴³ Soon J, Traeger AC, Elshaug AG, et al. “Effect of two behavioural 'nudging' interventions on management decisions for low back pain: a randomised vignette-based study in general practitioners”. BMJ Qual Saf 2019; 28: 547–55. doi:10.1136/bmjqs-2018-008659

⁴⁴ Grant AM, Hofmann DA. It’s not all about me: motivating hand hygiene among health care professionals by focusing on patients. Psychol Sci 2011; 22: 1494–9. doi:10.1177/0956797611419172

⁴⁵ Richards AR, Linder JA. Behavioral economics and ambulatory antibiotic stewardship: A narrative review. Clin Ther 2021; 43: 1654–67. doi:10.1016/j.clinthera.2021.08.004

⁴⁶ Loewenstein G, Bryce C, Hagmann D, et al. Warning: you are about to be nudged. Behavioral Science & Policy 2015; 1: 35–42. doi:10.1353/bsp.2015.0000

⁴⁷ Aboagye-Sarfo P, Mai Q. Seasonal analysis of emergency Department presentations in Western Australia, 2009/10–2014/15. Journal of Applied Statistics 2018; 45: 2819–30. doi:10.1080/02664763.2018.1441384

⁴⁸ Press Ganey. Emergency Department Survey

⁴⁹ Jensen MP, McFarland CA. Increasing the Reliability and validity of pain intensity measurement in chronic pain patients. PAIN 1993; 55: 195–203. doi:10.1016/0304-3959(93)90148-I

⁵⁰ Brown G. The Örebro musculoskeletal pain questionnaire. Occupational Medicine 2008; 58: 447–8. doi:10.1093/occmed/kqn077

⁵¹ Henschke N, Maher CG, Refshauge KM, et al. Prognosis in patients with recent onset low back pain in Australian primary care: inception cohort study. BMJ 2008; 337: 337/jul07_1/a171. doi:10.1136/bmj.a171

⁵² Foundation ER. EQ-5D-5L User Guide 2019.

⁵³ Holt N, Pincus T. Developing and testing a measure of consultation-based reassurance for people with low back pain in primary care: a cross-sectional study. BMC Musculoskelet Disord 2016; 17: 277.: 277. doi:10.1186/s12891-016-1144-2

⁵⁴ Brodney S, Fowler FJ Jr, Barry MJ, et al. Comparison of three measures of shared decision making: SDM Process_4, collaborate, and SURE scales. Med Decis Making 2019; 39: 673–80. doi:10.1177/0272989X19855951

⁵⁵ Harris L. Public opinion of patient safety issues research findings. National Patient Safety Foundation, 1997.

⁵⁶ Berger L, Bleichrodt H, Eeckhoudt L. Treatment decisions under ambiguity. J Health Econ 2013; 32: 559–69. doi:10.1016/j.jhealeco.2013.02.001

⁵⁷ Saposnik G, Redelmeier D, Ruff CC, et al. Cognitive biases associated with medical decisions: a systematic review. BMC Med Inform Decis Mak 2016; 16: 138. doi:10.1186/s12911-016-0377-1

⁵⁸ Australia T, S.o.H.P.o. Reducing opioid related harm: A hospital Pharmacy landscape paper. 2018.

⁵⁹ Kanzaria HK, Hoffman JR, Probst MA, et al. Emergency physician perceptions of medically unnecessary advanced diagnostic imaging. Academic Emergency Medicine 2015; 22: 390–8. doi:10.1111/acem.12625 Available: https://onlinelibrary.wiley.com/toc/15532712/22/4

⁶⁰ Bachhuber MA, Nash D, Southern WN, et al. Effect of changing electronic health record opioid analgesic dispense quantity defaults on the quantity prescribed: A cluster randomized clinical trial. JAMA Netw Open 2021; 4: e217481. doi:10.1001/jamanetworkopen.2021.7481

⁶¹ Zauberman G, Kim BK, Malkoc SA, et al. Discounting time and time discounting: subjective time perception and Intertemporal preferences. Journal of Marketing Research 2009; 46: 543–56. doi:10.1509/jmkr.46.4.543

⁶² O’Donoghue T, Rabin M. Doing it now or later. American Economic Review 1999; 89: 103–24. doi:10.1257/aer.89.1.103

⁶³ Rozbroj T, Haas R, O’Connor D, et al. How do people understand Overtesting and Overdiagnosis? systematic review and meta-synthesis of qualitative research. Social Science & Medicine 2021; 285: 114255. doi:10.1016/j.socscimed.2021.114255

⁶⁴ Meredith SE, Petry NM. Improving medication adherence with behavioral economics., in Behavioral economics and healthy behaviors: Key concepts and current research A.J.B. Routledge/Taylor & Francis Group, 2017: 109–26.

⁶⁵ Ballantyne JC, Kalso E, Stannard C. WHO analgesic ladder: a good concept gone astray. BMJ 2016; 352: i20. doi:10.1136/bmj.i20