Content area
Background
In response to rural medical workforce shortages in Western and Southwestern Victoria, Australia, Deakin University’s School of Medicine established a dedicated Rural Training Stream (RTS) within its graduate entry Doctor of Medicine course in 2022. Consistent with the school’s intention to promote social accountability, the RTS embodies a place-based approach, aiming to recruit 30 local applicants annually and train them in situ, thereby maximising opportunities for deeper connections with their communities and enhancing the likelihood of future local workforce retention. To underpin and facilitate collaborative program evaluation and research conceptually, a Program Logic Model (PLM) was developed. A PLM was chosen as the basis of this evaluation framework due to its foundations in systems and complexity theory and its amenity to collaborative development.
Methods
The PLM was developed using a stepwise, participatory approach, incorporating the contributions of a broad range of educators involved in the RTS program. Over a period of three months, participant input was provided through individual submissions and group workshops, sequentially developing each component of the model. Data gathered were summarised, reviewed and validated by the group periodically.
Results
The development of the PLM brought together a diverse group of academic and professional educators during the early stages of delivery of a new program, providing an opportunity for consensus building and the development of a shared understanding of the goals and objectives of the program. The developed PLM provides three clear objectives related to recruitment, training and professional identity formation. Short, medium and long-term outcomes aligned with each of these areas provide key reference points for the design of a comprehensive program of evaluation and research that is clearly linked to the program’s goals.
Conclusion
The development of a PLM for Deakin’s new RTS has provided an accessible roadmap that transparently communicates the design of the program to enhance local workforce outcomes for the rural communities that Deakin serves. The PLM provides a shared foundation for program-related evaluation and research, that is aligned with the strategic goals and desired outcomes.
Trial registration
Clinical trial number: not applicable.
Background
In response to rural medical workforce challenges in Western Victoria, Australia, the Deakin University School of Medicine established a dedicated Rural Training Stream (RTS) within the graduate entry Doctor of Medicine (MD) course in 2022. Rural areas of Western Victoria comprise approximately one third of the area of the state yet experience a medical workforce maldistribution common to many areas of rural Australia. In 2023, only 5.3% (430) of the states 8141 General Practitioners (GPs) were working in rural areas of the Western Victoria Primary Healthcare Network, with 64.2% (276/430) of these GPs over 45 years of age [1]. Consistent with the school’s focus on enhancing its social accountability, the RTS aims to recruit local applicants and train them in situ, maximising opportunities for deeper connections with their communities and increasing the likelihood of future local workforce retention. The design of the RTS is based on substantial evidence and andragogy demonstrating the effectiveness of similar approaches in achieving robust graduate rural workforce outcomes [2, 3–4].
The RTS has adopted a place-based approach to rural medical education to address rural workforce shortages, whilst promoting and providing educational opportunities for underserved communities [5]. The rural training footprint of the medical school has been precisely defined and refers to a geographical area recognised as medically underserved [6]. Applicants meeting Australia’s Rural Health Multidisciplinary Training Program’s rural background criteria [7], whose rural background address is within Deakin’s rural footprint, are prioritised for entry to the RTS, ahead of rural background applicants from other parts of rural Victoria or rural Australia. From 2024, RTS students (30 per year) can complete the entire four-year MD within the rural footprint, without having to relocate to a metropolitan campus. The two foundational (pre-clinical) years of the course are undertaken through a program of blended learning, with face-to-face activities based primarily at one of two rural learning campuses that are centrally located within each sub-region of the rural footprint (Fig. 1). Clinical training (years 3 and 4) is undertaken at one of three rural clinical schools, also located in the rural training footprint.
Program evaluation is a critical component of comprehensive medical education programs, providing feedback on the outcomes of a program and facilitating continuous improvement [3, 8]. As the RTS commenced, members of the geographically distributed team, contributing to both strategic and operational aspects of implementation, were invited to join a new RTS evaluation and research group. The purpose of this group was to develop a transparent and coordinated approach to the evaluation of the program collectively, along with identification of research opportunities.
An early decision of the group was to develop a Program Logic Model (PLM) to plan and coordinate research and evaluation activities, through the clarification of agreed program outcomes. A PLM graphically illustrates how an initiative is intended to work by explicitly outlining the intended links between resources, activities, and outcomes, linking these back to the program’s goals and objectives [9–13]. A PLM was deemed to provide a suitable evaluation framework for a rural medical education program due to its foundations in systems and complexity theory and its ability to incorporate the relationships between program components and the program’s context [8]. Frameworks that recognise complexity are particularly relevant in rural medical education, which exists within a complex nexus of stakeholder relationships, training providers and within national and state level workforce and funding policy contexts.
A further reason for choosing a PLM was the suitability of this model for collaborative development and facilitation of communication, to develop collective agreement on the RTS program’s purpose, structure and intended outcomes [9]. The model’s focus on the change process and logical connections between activities and outcomes offers an opportunity to deepen engagement through the shared development of the program’s ‘road map’ [8, 9].
It has been reported that the links between medical schools’ social accountability goals and elements of program design are often not evident [14, 15]. This paper provides an example of how explicit links between elements of program design can be made transparent through a process of collaborative logic model development.
[See PDF for image]
Fig. 1
Deakin School of Medicine sites. © Deakin University, reproduced with permission. All rights reserved
Methods
Our PLM was designed using a participatory approach [16]. A PLM toolkit was used to guide the process of development, supported by other resources and adapted to suit our context, requirements and timeframe [9–11].
Participants
Participation in the RTS research and evaluation group was open to any School of Medicine educators involved in the development or delivery of the RTS. Following expressions of interest, the group commenced meeting regularly, approximately every six weeks, with most meetings conducted via Zoom™, enhancing access to these activities for our geographically dispersed participants.
Data collection and analysis
Step 1: designing the PLM framework and development process
While PLMs are simple and linear, they can vary in terms of the elements designers choose to include [11]. The initial action of the group was to agree on the PLM elements required for our context and how the model would be developed collaboratively. Three authors (JB, VR and LF) reviewed the literature and examples of PLMs, sharing the findings, with recommendations, to the group. The group agreed on the elements for inclusion in the model and then followed a systematic stepwise approach that iteratively developed these in the following sequence: (i) goal, objectives and assumptions, (ii) inputs, (iii) activities (outputs), (iv) outcomes (short, medium and long term), (v) external factors [12]. At each stage of development, consultation processes were included to provide an opportunity for de-identified independent input by all participants, followed by group discussion, refinement and achievement of consensus.
Step 2. Developing the program goal and objectives
The first component of the model, developed by consensus, was the program’s goal. This was a deductive process, as the program had already been developed, but a clearly defined goal statement had yet to be established. A Qualtrics survey was developed (January 2024) and distributed, providing research group members the opportunity to submit their interpretation of the RTS program goal and objectives, anonymously (Supplementary file 1) [17]. Three authors (JB, LF, VR) independently read and deductively coded all responses, then met to discuss and develop common themes. The results were presented, discussed and refined during a four-hour hybrid workshop to achieve consensus amongst the broader group (synchronous in-person and online (February 2024)). This process allowed for a common understanding to be developed and for objectives to be transformed into user-friendly statements of change that were achievable, and representative of the key components required to accomplish the RTS’s goal.
During this stage, participants were also asked to submit their assumptions about the program. Assumptions are the underlying reasons, or the conditions believed necessary, for the program to succeed [10]. In essence, they are assumptions underpinning why it is believed the program will be successful.
Step 3. Identifying inputs, outputs and outcomes
The identification of inputs is necessary to map both the available resources and those required to achieve the program’s goal [10]. Many resources to support the RTS were already in place, but as rural delivery of the foundational years was commencing in 2024, this activity was an opportunity to review resourcing and prioritise needs. During the workshop, participants were asked to contribute to the creation of lists of both existing and required resources, under headings including human resources, funding, infrastructure and external partnerships. These were later synthesised and added to the draft model, which was then reviewed and discussed by the group.
The outputs (activities) represent the actions required to deliver the program successfully. They depict how the resources will be utilised to achieve the program outcomes and ultimately, the goal of redressing the rural workforce shortage [10]. Given many of the program activities had already been planned prior to the commencement of the RTS, the development of this part of the model primarily involved alignment with the program objectives.
During the workshop, participants individually generated an open-ended array of outcomes which were entered into a matrix table, aligning them with one of the three program objectives (row), as well as short, medium and long-term timeframes (columns). Group discussion followed, collating similar outcomes and determining the appropriate time-frame category until consensus was achieved. Following the workshop, three authors individually and then collectively refined the group’s contributions into a series of specific outcome statements (JB, LF, VR).
Step 4. Data validation
A de-identified Qualtrics survey (March 2024) was sent to participants that presented a summary of the components of the PLM generated through the group’s input [17]. Survey actions were to (i) confirm or reject the goal and objectives, (ii) select, from a list of inputs, those they considered relevant to the RTS, (iii) select from a list of outputs, those that applied to the program, and (iv) select from a list of outcome statements and place them in the appropriate time frame.
To ensure outcomes were achievable, participants were directed to examine outcomes critically for (i) alignment with the goal and objectives of the program and (ii) ability to be measured. During this stage, there was also an opportunity to submit free text for alternate wording, or reasons for rejecting a component of the PLM.
Step 5. Drafting the PLM
Survey responses were collated (JB), and a draft PLM was developed (JB, LF, VR) incorporating all elements. This was then reviewed and refined by the whole group at an online meeting (March 2024). During this process, wording was slightly amended to enhance understanding of some components of the PLM. A further program outcome was identified and included at this stage, that aligned with measuring one of the objectives, highlighting the iterative nature of constructing the PLM over time. External factors were also included in the PLM, recognising the contextual elements in the medical workforce environment that are beyond the program’s scope, but may influence the achievement of our goal (either positively or negatively).
A final draft of the model (April 2024) was distributed for comment, with re-drafting continuing until a consensus was reached. The PLM is a living document and will be revised as required and reviewed annually.
Results
Twenty-two participants engaged in the development of the PLM including a broad range of educators involved in both strategic and operational aspects of the RTS program. There were academic and professional educators from the three rural clinical schools, course and school leadership, foundational teaching team, Indigenous Health team and Deakin University’s Damion Drapac Centre for Equity in Health Professional Education [18].
Figure 2 depicts the final PLM. The model includes an overarching goal, three objectives, inputs, outputs, outcomes (short, medium and long term), assumptions and external factors. The developed PLM provides a clear visual representation of how the program activities (outputs) relate to the goal and objectives of the RTS, the desired outcomes over time, and some influential factors in the external context in which the program operates.
Goal
The overarching goal is:
To ameliorate the medical workforce crisis within the Deakin University School of Medicine rural training footprint.
In PLMs, sometimes the problem the program is seeking to address is defined as the overarching concept. Our high-level problem, the maldistribution of the medical workforce, is situated at a societal level, is multifactorial and is deemed to be a ‘wicked problem’ (i.e. difficult to address and resistant to change) [19]. Our goal needed to be something closely aligned with our local impacts and what the RTS can realistically achieve. The term ‘ameliorate’ was chosen to describe an attainable goal, acknowledging that one program cannot solve the pervasive rural medical workforce problem alone. Furthermore, the focus on the school’s rural training footprint reflects its social accountability intention to graduate doctors with the attributes required to deliver contextually appropriate care for the communities the school serves [4, 6].
Objectives
Analysis of staff surveys and discussions identified three fundamental objectives of the program that needed to be addressed to achieve the overarching goal. Objectives were included in the model as statements of change.
Recruitment: To remove admission barriers and prioritise the recruitment of students from our rural training footprint, who have the attributes needed to be effective doctors for their communities.
Training: To facilitate longitudinal training within the rural training footprint, to decrease the dislocation of students from their rural communities.
Professional identity formation: To foster professional identity formation, through the delivery of learning opportunities that nurture and promote careers in rural general practice and rural generalism.
The development of the third objective generated important discussion around the types of ‘generalist practice’ that the program was hoping to encourage, allowing us to clarify that careers in rural General Practice (GP), Rural Generalism (rural GP with advanced qualification) [20] and generalist practice within a medical or surgical specialty in a rural context, were all viewed as desirable career trajectories for graduates of the program.
Inputs and outputs
As the delivery of the RTS had already commenced, defining inputs was straightforward. The group’s contributions, through creating lists of available resources (inputs), identified the broad range of potential avenues of support available for the program, which was relevant at the point of commencing rural delivery of the foundational curriculum.
A review of the RTS planned activities (outputs) by the group provided an important opportunity to reflect on their alignment with the three objectives. Development of a shared understanding of the importance of activities or aspects of program design enhanced the project’s strategic direction. For example, the opportunity for early exposure to rural GP experience was able to be linked to the goal of fostering professional identity formation for generalist rural practice.
Outcomes
The process used to develop a list of program outcomes generated an extensive array of ideas. Through collaboration, ideas were refined, aligned with one of the three objectives and assigned to a short, medium or long-term timeframe. Key outcome statements were developed that captured the breadth of desired program outcomes and would inform research and evaluation activities.
Short-term outcomes were allocated a timeframe of 0–4 years, reflecting the duration of the MD course. A natural focus during this period was assessing the effectiveness of the changes to admission strategies on student recruitment to the RTS, monitoring student progression and the provision of student support within the new program design. Medium-term outcomes (4–10 years) focussed on the sustainability of the program, building rural training capacity, partnerships and the influence of the program on graduates choosing to train in generalist rural specialities in the rural footprint. Long-term outcomes (> 10 years) focussed on the retention of graduates and the impact on the availability of healthcare in the region.
Assumptions and external factors
A fundamental assumption of the program is that there are residents within our rural training footprint who are suitably qualified, wish to undertake medical studies in their community and will remain in those communities once they are qualified. This assumption is based on previous formal and anecdotal community consultation within the rural footprint, empirical knowledge related to the academic disadvantage that rural applicants face when applying to medical school and the barriers that dislocation from their rural community presents [2, 14, 21, 22]. Thus far, this assumption has proven correct, with applications to the Deakin MD RTS entry pathway (2024 and 2025 admission) from the rural footprint outnumbering the places available. It is anticipated that this interest may increase further as the program gains visibility within the region.
The development of the PLM provided an opportunity to identify the factors beyond the program that may have a significant influence on the outcomes. The availability of post-graduate training pathways within the rural footprint is an essential requirement for retaining graduates in the region. The RTS delivery is dependent on partnerships with health services in the region where students complete their clinical placements. The availability of both funding and supervision capacity are external factors identified in the PLM, fundamental to the successful delivery of this program.
[See PDF for image]
Fig. 2
Rural training stream project logic model
Discussion
The initial stimulus for developing the PLM was to coordinate RTS research and evaluation activities, however, the benefits of this process went well beyond this. The development of the PLM brought together a diverse group of academic and professional educators during the early stages of delivery of a new program providing an opportunity for consensus building and the development of a shared understanding of the goals and objectives of the program. Gathering input from the group through online and face-to-face meetings, as well as asynchronous anonymous surveys allowed for equitable sharing and discussion of ideas, valuing, and consideration of all contributions. This inclusive approach bridged departmental divides, opened networking channels and mitigated power dynamics based on organisational or personal characteristics [23]. The group reflected that the collaborative process of developing the PLM was instrumental in clarifying and deepening engagement with the RTS program’s goals.
The developed PLM provides an accessible reference document for all stakeholders, with a simple heuristic and iterative nature that allows for adjustments, as needed [23]. It provides an effective tool for communication of program design that will be useful to compare with other similar programs and to examine the program’s applicability and generalisability to other settings and populations [24, 25]. In recognition of the need for a place-based approach to program evaluation, the PLM allows for the inclusion of contextual elements such as assumptions and external factors, that can be adapted to suit the specific circumstances of other programs.
Developing a robust plan to evaluate a new education process, or a suite of research projects associated with a new program, is a complex undertaking. However, the use of a PLM framework made this process efficient, comprehensive and inclusive, and established the foundation for an effective and supportive team to commence planning and undertaking the research and evaluation activities for the RTS. Having agreed program outcomes provides a clear reference point for the alignment of proposed research and evaluation activities. In the spirit of ongoing open collaboration, RTS research group members are developing research and/or evaluation projects under each of the project outcome headings. For example, to evaluate the achievement of the first two listed short-term outcomes in the PLM, a project is currently under development to compare the demographic and geographic profile of RTS students with other MD student cohorts. The educators who developed the model will continue to meet to coordinate its activities, ensuring that research proposals align with the goals of the program and do not overburden the enrolled students.
The division of outcomes into three distinct timeframes has provided a helpful basis for sequencing and prioritisation of research and evaluation activities. In addition to clarifying the most time-critical projects, having visibility of longer-term goals allows for the early collection of baseline data that may be required, such as RTS student admission, demographic and training information. The process of defining measurable outcomes has stimulated discussion on the availability and sharing of existing data sources. For example, we are currently involved in several national projects that collect information ranging from graduates’ practice locations and medical specialities, students’ intentions at exit from the MD and survey data on students who have undertaken rural training. Visibility of the data already available creates efficiencies, mitigates the duplication of activities and has highlighted data gaps to be addressed.
Moreover, the development of a PLM allows for the inclusion of fundamental program reporting requirements, ensuring these deliverables are met. Funding through Australia’s Rural Health Multidisciplinary Training program requires engagement in research into rural health workforce development and the impact of rural training strategies (parameter 5) [7]. Evaluation of the outcomes of different course entry streams is an accreditation requirement of the Australian Medical Council. The PLM provides a shared framework to conduct evaluation and research to meet these program requirements, as well as a foundation to encourage broader collaborative rural research activity.
Limitations
The PLM presented represents the output from the initial development process and will be further refined and adapted over time as the RTS matures. We are aware of several assumptions that were explicitly included in the PLM but acknowledge there are likely to be further blind spots requiring identification in future. Although the logic model framework has been successfully adopted across a range of different program contexts, the components of the PLM require adaptation to the context and requirements of each setting. Having been developed by educators within the medical school, our model would benefit from transdisciplinary validation by broader stakeholder and rural community consultation.
Conclusion
The development of a PLM for Deakin’s new RTS has provided an accessible roadmap that transparently communicates how the program is designed to enhance local workforce outcomes for the rural communities that Deakin serves. The process of PLM co-creation provided an important opportunity for communication and clarification amongst educators regarding the program’s goals and design logic. Importantly, the PLM provides a shared foundation for a comprehensive program of collaborative evaluation and research, aligned with the strategic goals and desired outcomes.
Acknowledgements
The authors wish to acknowledge all participants in Deakin’s Rural Training Stream research group who contributed to the development of the program logic model.
Author contributions
LF, JB and VR led the conceptualisation of the Project Logic Model (PLM) development project, project design and methodology, collection, analysis and synthesis of the study data and contributed to writing the first draft of the manuscript. LF, JB, VR, TM, LG, DB, GR, NM, EM, BC, JM, TW, KB participated in data generation and PLM development through project workshops and meetings and reviewed draft manuscripts. LF revised all draft manuscripts and authored the final version. All authors approved the final version of the manuscript.
Funding
The authors acknowledge the Australian Government’s Rural Health Multidisciplinary Training funding that supports this work.
Data availability
The datasets generated and/or analysed during the current study are not publicly available due to the mixed methods and iterative nature of data collection but are available from the corresponding author on reasonable request.
Declarations
Ethics approval and consent to participate
Ethics exemption granted by Deakin University Human Research Ethics Committee, Project ID: 2025/HE000480. Informed consent was obtained from participants for data collection and dissemination in accordance with Australian National Health and Medical Research Council guidelines for ethical conduct in human research.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Abbreviations
General Practice
Doctor of Medicine
Program Logic Model
Rural Training Stream
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
1. Australian Government Department of Health and Aged Care. Data Tool. https://hwd.health.gov.au/datatool/. Accessed 26 Mar 2025.
2. McGrail, MR; O’Sullivan, BG. Increasing Doctors working in specific rural regions through selection from and training in the same region: National evidence from Australia. Hum Resour Health; 2021; 19,
3. McGrail, MR; Doyle, Z; Fuller, L; Gupta, TS; Shires, L; Walters, L. The pathway to more rural Doctors: the role of universities. Med J Aust; 2023; 219,
4. Fuller, L; Beattie, J; Versace, V. Graduate rural work outcomes of the first 8 years of a medical school: what can we learn about student selection and clinical school training pathways?. Aust J Rural Health; 2021; 29,
5. Medical Deans Australia and New Zealand. Position Paper: Thriving Rural Doctors: Place-Based Solutions to Medical Workforce Challenges in Remote, Rural and Regional Australia and New Zealand. Sydney, Australia. 2024. https://medicaldeans.org.au/md/2024/06/Thriving-Rural-Doctors-Medical-Deans-Position-Paper-Jun-2024.pdf. Accessed 9 Dec 2024.
6. Fuller, L; Beattie, J; Versace, VL; Rogers, GD; McGrail, MR. Describing a medical school’s rural activity footprint: setting selection and workforce distribution priorities. Rural Remote Health; 2024; 24,
7. Commonwealth of Australia. Rural Health Multidisciplinary Training (RHMT) program framework (2019-20). Department of Health and Aged Care, 2021. Accessed 30 Aug 2024.
8. Frye, AW; Hemmer, PA. Program evaluation models and related theories: AMEE guide 67. Med Teach; 2012; 34,
9. Eaton SE. Logic models: What they are and how to prepare one. Onate Press, 2010. https://www.researchgate.net/publication/236222297_Logic_Models_What_they_are_and_how_to_use_them. Accessed 13 Nov 2024.
10. New South Wales Ministry of Health, Centre for Epidemiology and Evidence. Developing and using program logic: a guide. Population and Public Health Division. 2023. https://www.health.nsw.gov.au/research/Publications/developing-program-logic.pdf. Accessed 10th Jul 2024.
11. Australian, Government. Australian Institite of Family Studies. How to develop a program logic for planning and evaluation. Canberra 2016. https://aifs.gov.au/resources/practice-guides/how-develop-program-logic-planning-and-evaluation. Accessed 10 July 2024.
12. Innovation network: Transforming evaluation for social change. Logic Model Workbook. Washington. 2005. https://www.innonet.org/news-insights/resources/logic-model-workbook/. Accessed 10 Jul 2024.
13. O’Keefe, CM; Head, RJ. Application of logic models in a large scientific research program. Eval Program Plann; 2011; 34,
14. Ellaway, RH; Malhi, R; Bajaj, S; Walker, I; Myhre, D. A critical scoping review of the connections between social mission and medical school admissions: BEME guide 47. Med Teach; 2018; 40,
15. O’Sullivan, BG; McGrail, MR; Russell, D; Chambers, H; Major, L. A review of characteristics and outcomes of Australia’s undergraduate medical education rural immersion programs. Hum Resour Health; 2018; 16,
16. Greenhalgh, T; Hinton, L; Finlay, T; Macfarlane, A; Fahy, N; Clyde, B et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect; 2019; 22,
17. Qualtrics, inventor., Provo, editors. Utah, USA2020.
18. Deakin University. The Damion Drapac Centre for Equity in Health Professions Education, Geelong 2024. https://www.deakin.edu.au/faculty-of-health/school-of-medicine/the-damion-drapac-centre. Accessed 2 Dec 2024.
19. Walls, HL. Wicked problems and a ‘wicked’ solution. Globalization Health; 2018; 14,
20. Royal Australian College of General Practitioners. The Collingrove Agreement and Rural Generalist Recognition. 2024. https://www.racgp.org.au/the-racgp/faculties/rural/rural-generalist-fellowship/about-the-rural-generalist-fellowship/recognition-of-rural-generalist-medicine. Accessed 7 Nov 2024.
21. Beattie, J; D’Souza, K; Mc Leod, J; Versace, V. Rural origin students match metropolitan origin students’ academic performance once admitted to bachelor of medicine bachelor of surgery course. Aust J Rural Health; 2019; 27,
22. Wilson, IG; Roberts, C; Flynn, EM; Griffin, B. Only the best: medical student selection in Australia. Med J Aust; 2012; 196,
23. Fielden, SJ; Rusch, ML; Masinda, MT; Sands, J; Frankish, J; Evoy, B. Key considerations for logic model development in research partnerships: a Canadian case study. Eval Program Plann; 2007; 30,
24. Page, M; Parker, SH; Renger, R. How using a logic model refined our program to ensure success. Health Promot Pract; 2009; 10,
25. Savaya, R; Waysman, M. The logic model. Adm Social Work; 2005; 29,
© The Author(s) 2025. This work is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.