Introduction
The UK lags behind other G7 countries in the productivity of its small- and medium-sized enterprises (SMEs) and suffers, in particular, from a long tail of low-productivity SMEs (LSE Growth Commission, 2017). The Business Basics Fund (Department for Business, Energy and Industrial Strategy, 2019–21) was set up to discover if technological and management practices commonly used in corporate organizations would also raise productivity if adopted by SMEs.
A large and multidisciplinary literature base points to gains in the popularity of coaching as a personal development intervention. There are still few studies that evaluate the practice of coaching beyond the idea of conducting episodic Executive Coaching sessions to examine the aspects that might encourage adoption of coaching-related behaviors into everyday management practices. Previous research studies of coaching efficacy (Institute of Coaching, 2010) and the evaluation of coaching effectiveness (Osatuke et al., 2017) examined particular groups of people enjoying coaching sessions with a coach toward the improved accomplishment of varied objectives, such as goal attainment (Spence et al., 2008; Bandura and Lyons, 2017; Morgeson et al., 2010); higher team playing behavior (Sue-Chan and Latham, 2004) and improved teamwork performance (Aldrin and Utama, 2019); improvements in self-efficacy and resilience (Franklin and Doran, 2009); enhanced goal attainment resilience and workplace well-being, reduced depression and stress and helped participants deal with organizational change (Grant et al., 2009) and improved well-being and engagement (Green and Spence, 2014; McQuaid et al., 2018).
It has been noted in research conducted by the UK's Office for National Statistics (2018) that very small changes in a derived management score have a direct correlation to increases in productivity (for each 0.1 movement in the derived score, a 9–10% improvement in productivity). The opportunity then to equip leaders and managers with coaching skills as a means of fostering continuous and higher levels of engagement, focusing their efforts on enabling others to step up and improve their performance on an ongoing basis, could pay an enormous productivity dividend. There is a paucity of evidence that coaching can meaningfully change workforce behaviors and performance and boost overall business productivity. Notion is an award-winning management performance improvement consultancy of some 20 years' standing. In their work with corporate clients, the embedding of Notion's STAR Coaching Model® (created by Notion in 2010; Intellectual Property Office, 2014) to support wider adoption of an Operational Coaching™ style of management has been measured by Notion (2011–2019) to improve levels of engagement, performance and productivity (Royal Mail [2011–12], National Express [2014–15], J Sainsbury plc [2015–16], Stepstone [2017–19] and Amino Communications [2019–2022]).
This Business Basics–funded study sought to discover if the same productivity gains could be achieved by SMEs when their managers (learners) adopted this same set of coaching-related behaviors. It also aimed to test a controlled intervention. We had two main objectives. First, we wanted to examine the outcomes from applying the STAR® model to encourage the adoption of an Operational Coaching™ style of management within SMEs, particularly if the STAR® model offers the potential for changing management behavior (measured in terms of the proportion of time in their average working day spent by the manager coaching and leading rather than managing and doing). Second, we wanted to assess whether that change in behavior could contribute to an improvement in productivity.
Our article unfolds as follows. First, we briefly review the workplace control literature with a specific emphasis on coaching intervention research. Then we assess the role that adopting an Operational Coaching™ style of management likely plays in the process of changing employee behaviors and performance. This review forms the basis for our hypotheses concerning the effects that managers adopting a distinct set of coaching-related behaviors (an Operational Coaching™ style) can have on employee behaviors and performance designed to augment business productivity. We then describe the intervention (the STAR® Manager program) that teaches managers a four-step model (Notion's STAR Coaching Model®) to adopt Operational Coaching™ behaviors and report the results of a randomized field experiment that evaluated the effects of the intervention on several managers (individual learners) and the SME outcomes.
Architecture of the STAR® Manager program (the intervention)
The overall design of Notion's STAR® Manager training program is to create a learning experience which encourages small adjustments in behavior so that managers begin to use Notion's STAR Coaching Model® more frequently in their day-to-day workplace interactions and thus adopt an Operational Coaching™ style of management. The STAR® Manager program is hosted on an online learning platform that can be accessed at any time via any device. It is a programmatic, sequential learning programme made up of 20 modules that engage the learner (the manager) in varied activities based around a “Learn, Do, Review” model. The first 10 modules of the STAR® Manager program focus on teaching the manager how to put Notion's STAR Coaching Model® into practice to adopt more of an Operational Coaching™ style of management by the greater use of enquiry and other coaching-related behaviors, such as active listening and providing appreciative and developmental feedback. The second 10 modules focus on teaching the manager how to use an Operational Coaching™ approach to intentionally enable and develop others, especially their direct reports, through more planned developmental conversations. More details on the intervention are in supplementary material 1.
Logic model
The logic model in Figure 1 sets out the STAR® Manager program intervention (input), how it is delivered (activity) and the assumptions about how the STAR® Manager program works in terms of outputs and outcomes (immediate, intermediate and ultimate). A hierarchical approach (Pilbeam and Corbridge, 2010) shown in Figure 2 was also used, to focus on objective measurements so that the costs and benefits of the STAR® Manager program for the business can be measured and cost–benefit analyses calculated. Every level covers a different evaluation strategy, and it provides a measure of the transfer and application of learning contents embedded in the STAR® Manager program in practice in SMEs. For example,
If participants enjoy the program (level 1), they are likely to learn (level 2);
If participants learn (level 2), they are more predisposed to change their on-the-job performance and behavior (level 3);
If participants improve their work performance and behavior (level 3), the learning is likely to have a business impact on the SMEs (levels 4 and 5).
Methodology
A cluster randomized controlled trial (RCT) was set up to provide a robust evaluation study to prove the effectiveness of the STAR® Manager program in changing learner behavior against a control. The study was registered in the American Economic Association's registry for RCTs (AEA RCT Registry: AEARCTR-0005008 - Adopting Operational Coaching as a Management Style to Drive SME Productivity, 2019). SMEs (Cluster) were recruited and supplied a number of managers (individual learners) to participate in the RCT. Ethics approval was obtained from the London School of Economics and Political Science (LSE; Care Policy Evaluation Centre Research Ethics Committee) prior to commencement.
The RCT's main research question was “For SMEs (the population), does access to a blended learning program in Operational Coaching™ (the STAR® Manager program) (the Intervention) lead to greater adoption of coaching-related management behaviors that drive performance and productivity increases (the outcome) than no access at all (the control)?”
Impacts were analyzed against the individual and cluster level. We determined a range of impacts arising from receiving training in the use of Notion's STAR Coaching Model® by taking the STAR® Manager program and that the adoption of an Operational Coaching™ style of management brought about by the application of newly learned skills.
Strategic partner networking and sampling
The RCT was conducted in consortium with Notion, the LSE and the Coventry and Warwickshire Growth Hub. Notion and, initially, the Coventry and Warwickshire Growth Hub were responsible for the recruitment of SMEs to the trial. Notion created a database of 274 contacts in Local Enterprise Partnerships and Growth Hubs and contacted each one to create awareness of the RCT program and the opportunities for local SMEs. The STAR® Manager program (the intervention) was delivered by Notion, and the results were independently evaluated by the LSE.
Randomization and blinding
Once SMEs signed up to participate in the program, SMEs were randomly allocated (2:1) to the intervention or control group using a computer-generated sequence using three SME-level stratification factors: SME size, SME industry sector and SME organizational experience (years since incorporation). The randomization process was conducted by Notion immediately after all SMEs had been recruited for each cohort, but each SME's allocated group (intervention or control) was not communicated to the SMEs and their learners until after baseline measurements had been taken. Notion ensured that numbers of control and intervention SMEs were 2:1 in both cohorts to facilitate study delivery. Due to the nature of the intervention (managers pursuing a development program), SME staff members, learners and individuals delivering the intervention could not be blinded to group allocation. The RCT evaluator (LSE) was blinded regarding the SMEs, learners and study groups.
Experimental design
The experimental design was a two-arm, pragmatic, SME-based, cluster RCT with masked outcome assessment and was performed in several SMEs in England randomized 2:1 to receive Notion's STAR® Manager program or a control condition. The cluster level was defined as the SME. Randomization took place at a cluster level after eligibility for the intervention was confirmed and initial data collection carried out at cluster (SME) and individual levels. Randomization was performed using a computer-generated sequence using three SME-level stratification factors. When SMEs confirmed their participation in the RCT, they were given a sequentially numbered alpha-numeric code. LSE performed the randomization. Once this had been completed, and all SME-level and learner-level baseline data collection had been completed, Notion informed each SME whether they had been allocated to the intervention. Each SME had previously selected the learners that it wanted to participate in the RCT. There were no rules or criteria by which the SMEs had to select participating managers, and each SME used its own selection criteria and process. Consent was sought from each SME as part of the recruitment process. Consent was sought from each learner as part of the baseline data collection process.
All organizations were motivated to participate in the RCT as all SMEs had paid for their places on the study (the fee per learner was the same for intervention and control group SMEs). Those SMEs and their learners who were randomized into the intervention group completed the STAR® Manager program, and those randomized into the control group were able to access the STAR® Manager program after the follow-up data collection had been completed. Any SMEs randomized into the control group, who did not wish to proceed with accessing the STAR® Manager program, did not have to pay for the program. More details on the RCT are in supplementary material 2.
Measures
The primary outcome of the trial was calculated on the basis of the percentage of respondents in the self-assessment survey who reported an increased proportion of their daily time spent coaching as opposed to leading, managing or doing. The increase is calculated based on the difference between the results of the self-assessment survey taken before the start of the program and the results of the same self-assessment taken at the end of the trial period. This is not an “outcome measure” strictly speaking, but it gives a measure that the treatment is taken. An increase in the proportion of time that the respondent spends coaching should then create secondary outcomes that were also measured, such as
Proportion of time leading;
On a qualitative level, we analyzed changes in their perceptions of coaching.
In order to understand better how respondents were using the increased time that they reported that they were spending coaching, other secondary outcomes were measured covering four different outcome results:
Reaction (and well-being) outcomes: participants' perceptions, emotions and subjective interpretations of their learning experience with the STAR® Manager program.
Learning outcomes: how well the learning objectives are achieved.
Behavior outcomes: the degree to which the STAR® Manager program changes participants' behavior in their actual job.
Organizational outcomes: information on the change in organizational performance.
Wider societal impacts were measured in terms of individual well-being at work (see Short Smith Wellbeing Questionnaire). A list of the individual outcomes is provided in supplementary material 3.
Learner outcome analysis
The analysis population consisted of all randomized learners (and SMEs). The primary analysis for each outcome was undertaken on an “intention-to-treat” basis, i.e., all learners with a recorded outcome were included in the analysis and analyzed according to the group to which they were allocated. The full analysis population for the primary outcome analysis consisted of all randomized learners for whom baseline data were collected and for whom 6-month data were also available. If a learner left the SME, they were invited to continue with the data collection over the remaining period of the trial. Data were analyzed using linear regression modeling. More details are in supplementary material 4.
SME impact analysis
Additional ordinary least squares (OLS) regressions were performed to uncover which factors predict positive SME impacts. Controlled comparative data with baseline and follow-up measurements were analyzed using an OLS regression model. The main analysis included covariate-adjusted analysis, with the statistical models including the stratification variables (SME-related factors: SME size, industry sector and organizational experience), cohort, and baseline values for the outcome under consideration, where available. Unadjusted between-group differences were presented for completeness.
The main SME impact measures included the number of unexcused absences in the last 6 months; average number of employees (last 6 months); total working days (last 6 months); turnover rate for the last 6 months (%); employee losses and the average number of employees on the payroll for the same period. For the purpose of the cost–benefit analysis, we considered the Gross Asset Value (GAV) for the SME, in the last 6 months (economic impacts) vs. the cost of delivering the intervention per delegate (completed 20 modules) (economic costs). Missing data and presentation of comparative analyses followed the same protocol used for the outcome evaluation.
Intervention delivery costs
The SME total and per learner cost for providing this intervention package for the 6-month study period was calculated. The costs of the intervention were estimated based on material and facilitator costs to implement the intervention (program license cost per delegate). We assumed that no new equipment would be purchased by the SME/delegate to access the program, as this is standard office equipment. We included the cost of employee time (based on 233 working days, 7.5 h/day: Junior – £36,501; Middle – £50,669; Senior – £73,466; Glassdoor, 2019). Cost of overheads was considered as 20% of employee time cost (Notion data). Unit cost data are summarized in supplementary material 5.
Baseline scenario included program license fees of £350 per learner (as per reduced trial fees) and program duration of 20 modules (completed program). Sensitivity analyses were included increasing the license fees to £600 per learner and also different stages of the program. In addition, subgroup analyses were conducted as follows:
Per SME costs: data were presented considering a mean aggregate level and also different SME sizes (small, medium and large).
Per learner costs: data were presented considering a mean aggregate level and also different learners' level of management (junior, middle and senior).
Qualitative data and content analysis from the Success Tracker
Data from the Success Tracker (description of success, type of success and potential commercial value of success) were treated as qualitative data as the method of recording and estimating used was unique to each individual learner. After the data were in a consistent and organized format according to intervention and time point, we disassembled the information and created meaningful groupings or coding. We identified interesting features of the data systematically across the entire data set and at multiple levels (between groups and time point). A preliminary list of coding was created a priori looking at the survey questions, the literature and data emerging form the first cohort. The main codes, or categories to which each concept is mapped, were quantified and analyzed using content analysis (Bengtsson, 2016).
Return on investment (ROI)
Return on investment (ROI) was calculated as [the average commercial value of the STAR® Manager program per learner] divided by [the average cost of delivering the STAR® Manager program to a senior manager]. The average commercial value of the STAR® Manager program per learner was calculated from the qualitative success stories completed by the learners at follow-up (as the total value of successes reported divided by 209 learners). The fully built-up cost of the STAR® Manager program for a senior manager was equal to [the STAR® Manager program fee per delegate (trial fees £350; standard fees for program £600)] + [capital usage cost of equipment to access the program per delegate (£25)] + [cost of SME administrative support for the program per delegate (£51.28)] + [cost of employee time (variable)] + [cost of overheads (variable)]. Assumptions and details on the unit costs are in supplementary material 5.
The present study: results
Participant flow
Figure 3 shows the attrition rates over the lifecycle of the trial. Thirty-two SMEs were included in the intervention, and 15 SMEs were included in the control for analysis. A total of 209 learners were included in the intervention, and 93 learners were included in the control for analysis. Analysis was done by the original assigned groups.
Baseline characteristics
The SMEs and learners participating in this trial exhibited a good spread across all of the characteristics surveyed. SMEs reported 11 to 49 employees (55%) across different industry sectors with 5 years or more since incorporation. The majority of the learners were 35–54 years old, male and white British and at middle/senior level of management. Details of the SME and learner characteristics by treatment condition can be found in supplementary material 6. F-test of joint orthogonality for the learners' covariate showed that the variables are unrelated to treatment status (F test = 2.1 p-value = 0.06). The same may apply when looking at the SME covariates (F test = 2.2 p-value = 0.07).
Primary outcome: proportion of time spent coaching
All participants were asked to estimate the proportion of time they currently spend in each of four areas (coaching, leading, managing and doing) as a percentage of a typical working day. The proportion of time across a normal working day spent coaching (and leading) was greater in the intervention (24.3% [SD: 14.7], 19.7% [SD: 10.2], respectively) than in the control (12.6% [SD: 9.9], 16.2% [SD: 9.6], respectively), and there is significant difference between groups (the average marginal effect is 13.8 [SE: 1.75; p < 0.01] and 2.76 [SE: 1.3; p < 0.04; Table 1]). The sensitivity analysis confirmed the results emerging from the base case (Table 2).
Secondary outcomes (for learners and SMEs)
Table 3 summarizes the main results emerging from the trial. For example, despite only a short period of time for analysis following the end of the trial and the fact that we had low power for this analysis, since we only had 47 observations at the SME level, positive trends can be seen in the data collected at the cluster level on the impact of the managers' changes in management behavior at an organizational (SME) level. The GAV of SMEs in both the intervention and control grew over the trial period, but this growth was at a much higher level among SMEs in the intervention. Also, in the intervention, the average number of FTEs grew between baseline and follow-up, whereas the average number of FTEs decreased over the same period in the control. Both groups reported lower levels of staff turnover at follow-up, which may be attributed to staff being nervous to lose a safe job during the business disruption caused by the COVID-19 pandemic. The levels of staff turnover reported by the intervention were lower than those in the control. Despite the average number of FTEs increasing among SMEs in the intervention, the average number of managers in the intervention SMEs fell. The opposite was true in the control. Intervention managers are more effective and can raise the wider productivity of team members, making the SME more productive. More details on the secondary outcomes are in supplementary material 7.
Intervention delivery costs
Figure 4 represents the cumulative cost of participating, per delegate, at different stages of the program (from orientation up to module 20). We assumed that the participants spent a proportion of their time working through the program on an ongoing basis and then added the cost of this time to the program delivery costs. When we consider discounted fees per delegate, the overall interventions costs per person (up to module 20) vary between £892 for junior staff and £1,364 for senior staff.
Qualitative data from the Success Tracker and ROI analysis
When recording their successes, learners were asked to estimate and record the commercial value of each success, where possible. From these 166 successes, the qualitative data analysis showed that learners have identified about £19 million of commercial value to their organizations, although a large share of the value attributed to adopting coaching behaviors is from a single respondent. Supplementary material 8 also breaks out the types of success by their value (£).
As detailed above, in addition to the price each SME was charged for each learner participating in the STAR® Manager program (£350 + VAT), we have calculated the fully built-up cost to the SME of each participating learner. We can broadly say that from the estimated commercial value of the 166 success recorded, this could equate to an average learner ROI of 74 x the initial investment made by the SMEs in the program. If the program had been purchased at the maximum program price of £600 + VAT, this would still be an ROI of 65 x the investment. Please note these are approximate figures derived from qualitative stories shared by a subgroup of learners in the intervention rather than robust cost–benefit analyses embedded in the trial outcome evaluation. Still, these findings support positive trends in increased SME productivity reported by the intervention as part of the trial evaluation at 6-month follow-up.
Discussion
The results of the RCT showed that access to the blended learning program STAR® Manager (the intervention) led to greater adoption of coaching-related management behaviors (Operational Coaching™) that drive performance and productivity increases within SMEs. By aligning the results of the trial with the different evaluation levels and measures in the Pilbeam and Corbridge (2010) model, it can be shown that impacts can be seen both quantitatively and qualitatively at each of the five levels.
In the case of this RCT, changes in managers' awareness about their impact on others and learning how to use the STAR® model to adopt an Operational Coaching™ style of management did result in positive behavioral change among intervention group learners and secured better primary outcomes. Interestingly, the process evaluation presented elsewhere showed that in terms of adoption, reach, implementation and effectiveness, no obvious variations in results correlated to particular SME or learner-related characteristics were detected. This is unlikely to be due to a data sufficiency issue, but rather because these skills have universal application in improving all aspects of management and leadership. Whilst coaching gains in popularity as a personal development intervention, Notion has previously noted that the practice of coaching has largely failed to move beyond the idea of episodic coaching “sessions” to examine the aspects that might encourage adoption of new behaviors into everyday management practices. Yet, as previously noted in research conducted by the Office of National Statistics (2018), only very small changes in a derived management score can correlate to very large increases in productivity (for each 0.1 movement in the derived score, a 9–10% improvement in productivity). The opportunity then to show leaders and managers how to use the STAR® model to develop an Operational Coaching™ style as a means of fostering higher levels of engagement, focusing their efforts on enabling others to step up and improve their performance on an ongoing basis, could pay an enormous productivity dividend. Research shows that prevailing coaching models (Barner and Higgins, 2007) trained and used for episodic coaching sessions focus on the coachee rather than on helping the manager to embrace new behaviors themselves. Notion's STAR Coaching Model® is the first Operational Coaching™ model designed to help managers develop coaching as a habitual behavior, “in the moment,” as part of their everyday work conversations, focusing on the matter at hand. Notion's STAR® model provides managers with a new mental model that enables them to adopt an “enquiry-led approach” quickly and easily in any given situation. The key to Operational Coaching™ is in the manager's ability to recognize coachable moments and then to use those opportunities to ask more powerful questions. This approach is likely to stimulate higher levels of social exchange, trust and understanding that create the conditions necessary to drive high levels of performance, productivity, innovation, engagement and retention.
It has also been noted earlier that previous research of coaching efficacy have tended to examine particular groups of people enjoying coaching sessions with a coach toward the improved accomplishment of a research objective (de Haan, 2019). Not surprisingly with this level of (close-quarter marking) 1:1 support, there is typically an improvement in the outcome of the primary objective under review.
In the case of this RCT, the changes in behavior recorded as statistically significant were achieved entirely independently of a charismatic coach, with no guarantee of participation or compliance with the trial. Changes that were generated stemmed entirely from participants' experience of following the STAR® Manager learning program, which was also accomplished in the face of the most adverse conditions in a generation. It represents a first in the field of coaching research.
The coaching industry promotes how it is helping to build coaching cultures within organizations, drawing attention to either the large number of managers that have received coaching skills training or the prevalence of trained internal coaches within businesses. As has been noted in research cited earlier, the offer of coaching skills training in its current forms (largely based around the process of conducting episodic executive coaching sessions) and the challenges with managers being able to successfully apply this approach, comfortably, ethically, within the context of the hustle and bustle of a busy day-to-day working environment, is questionable at best. As for deploying trained coaches within an organization, the research highlighted earlier has established that this can generate valued outcomes for individuals and teams. Internal coaches, though (frequently numbered only in 10s or less within organizations) are often only able to offer their services for a few hours per month, as it is not typically their day job, and the expense associated with the use of professional external executive coaches creates an economic barrier to wide-scale deployment of coaching support. In Notion's experience having worked with a broad range of organizational clients for the last 20 years, these constraints mean that in fact the number of people receiving coaching typically reaches far fewer than 5% of employees; 200 people receiving coaching in an organization employing 28,000 does not constitute a sustainable coaching culture (Notion data).
By focusing on the adoption of coaching-related behaviors and their daily application (by giving managers access to the STAR® Manager program and showing them how to adopt an Operational Coaching™ style of management through the use of the STAR® model), rather than continuing to teach managers how to conduct formulaic coaching sessions (that they are unable to find the time to hold), the number of daily coaching interactions can increase exponentially. It is only when coaching is widely practiced by all managers as a preferred style of interaction with others (Operational Coaching™), reaching the other 95% of employees, that a truly coaching orientated culture can be said to exist.
Our RCT was successful from both a methodological and practical point of view and covered a representation of SMEs across England and a range of industrial sectors. The participating SMEs were equally distributed between intervention and control groups in terms of SME size and SME industry sector. Overall reach was in line with what was anticipated and consistent with previous similar Notion experiences. Reach was comparable across industry sectors (highly operational vs. lower operational). The distribution of SME size was balanced around an average between 11 and 49 FTEs and reflected UK practice (UK Parliament, 2021). The age and gender split of the workforce was aligned with UK statistics (The CIPD and the Scottish Centre for Healthy Working Lives, 2014). All the aforementioned factors appear to be prerequisites for the generalizability of the results to the English setting. Overall, this trial project had a robust sample size made up of SMEs and learners with a well-balanced range of characteristics and strong data collection at the SME and learner level at both baseline and follow-up. The outcomes that have been achieved are consequently robust and add to the evidence base on factors that positively affect SME performance. SMEs demonstrated strong commitment by providing a consistent venue for interventions and achieving high recruitment and high intervention adherence.
There are a few limitations in our study. There is a degree of attrition at the point after SMEs were allocated to either the intervention or control group, with a higher proportion of the control than the intervention withdrawing because they did not want to pay the fee and had to wait for the conclusion of the trial before they would be able to undertake the program themselves.
From this, it is reasonable to assume that, on average, those SMEs that remained in the control were more likely to be committed to the program than those that remained in the intervention and were, consequently, more likely to be convinced of the importance of coaching. Any intervention/control difference in dropout at the allocation stage probably led to the impact of the intervention being underestimated.
The trial was not powered to capture SME impacts and reported a very short timeframe (6 months) for SME impacts to be measured. Delays in data collection and large range of data variables collected resulted in optional additional analysis not able to be carried out. We were not able to measure sustainability of impacts at the learner level over time (1+ years) due to project timeframes. Highly challenging economic circumstances, such as Brexit and COVID-19, impacted the ability to recruit larger numbers onto the RCT. The ROI analysis does not follow an appropriate protocol for an economic evaluation of a cluster RCT. ROI figures are based on a limited sample of qualitative success stories collected from learners in the intervention.
There is a significant amount of further research and analysis that can be performed following the completion of this study. We demonstrated that learners working through the STAR® Manager program adopt coaching-related behaviors (Operational Coaching™) and change their management style, resulting in benefits that impact the manager, the team, the organization and its clients. In future research, extending the analysis period for both SMEs and learners beyond the 6-month window adopted here will determine whether this behavior change and resulting outcomes can be sustained over the longer term and still deliver statistically significant outcomes at an SME level. Further research can also be performed on how the managers' behavior changes were seen by their teams and direct reports. However, we are aware that several changes to the economic environment (flexi-furlough, the tier system for locking down local economies and the change in business optimism) occurred since the COVID-19 pandemic started and it would be difficult to draw clean ongoing evidence from the original trial participants. Future analysis of the trial data could still provide more insights into the impact of the COVID-19 pandemic on the cohort of SMEs and learners enrolled in the trial and the success of the STAR® Manager program.
Conclusion
This RCT has wide-ranging implications, including the fact that the STAR® Manager program leads SME managers to greater adoption of coaching-related management behaviors (Operational Coaching™) and Operational Coaching™ can be used by SMEs as easily as it can be by large, corporate organizations to transform management culture. Given that much research cited earlier points to the correlation between management performance and organizational productivity, these important results could be indicative of the economic and productivity impacts that a change in management behavior to include an Operational Coaching™ style could have, and they warrant serious further investigation. Methods for encouraging the wider dissemination of the STAR® Manager program across the UK's business should also be explored. As a scalable solution, this RCT has proven that the STAR® Manager program is an effective means of introducing large numbers of managers in different organizations and geographic locations to the STAR® model to help them adopt Operational Coaching™ as their predominant management style. More research would be needed to test the link between adoption of coaching behaviors and business performance or productivity.
This trial was funded by the Department for Business, Energy and Industrial Strategy (BEIS) via the Business Basics funding. The funding was administered by Innovate UK on behalf of BEIS.
Figure 1
Logic model
[Figure omitted. See PDF]
Figure 2
A hierarchical approach
[Figure omitted. See PDF]
Figure 3
CONSORT flow diagram SMEs and learners
[Figure omitted. See PDF]
Figure 4
Intervention costs
[Figure omitted. See PDF]
Proportion of time (%) currently spent coaching, leading, managing and doing during a typical working day and perception of coaching
Baseline | Follow-up | Average marginal effect | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Intervention | Control | Intervention | Control | Unadjusted | Fully adjusted‡ | SE | p value* | |||||||
Total (N) | 209 | 93 | 209 | 93 | ||||||||||
Proportion of time (%) spent … | Mean | SD | Mean | SD | Mean | SD | Mean | SD | ||||||
Primary outcome | Coaching | 14.4 | 10.7 | 14.7 | 9.9 | 24.3 | 14.7 | 12.6 | 9.9 | 12.8 | 13.8 | 1.75 | <0.01 | |
Leading | 17.2 | 11.2 | 17.2 | 12 | 19.7 | 10.2 | 16.2 | 9.6 | 3.52 | 2.76 | 1.3 | 0.04 | ||
Managing | 23.5 | 15.3 | 24.1 | 12.7 | 20.9 | 11.8 | 24.2 | 14.4 | −3.34 | −2.21 | 1.63 | 0.18 | ||
Doing | 45 | 22.7 | 44 | 20.2 | 35 | 20.2 | 46.9 | 20.3 | −12.15 | −5.54 | 4.9 | 0.71 |
Note(s): SD = standard deviation
‡ We considered random-effects linear model (comparing intervention vs control) to account for clustering among learners within the same SME. In the fully adjusted model, we applied adjustment for stratification variables (SME size, SME industry sector, SME organizational experience, years since incorporation) (learner level of management, learner no. of reports, SME no. of learners, learner age, learner gender and learner ethnicity), cohort and baseline measure of outcome under consideration. SE = standard error for the marginal effect (fully adjusted model). *p value = fully adjusted model
Sensitivity analyses for the main outcomes
Baseline | Follow-up | Average marginal effect | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Intervention | Control | Intervention | Control | Unadjusted | Fully adjusted‡ | Fully adjusted without larger clusters& | SE | p value* | |||||||
Total (N) | 209 | 93 | 209 | 93 | |||||||||||
Proportion of time (%) spent … | Mean | SD | Mean | SD | Mean | SD | Mean | SD | |||||||
Primary outcome | Coaching | 14.4 | 10.7 | 14.7 | 9.9 | 24.3 | 14.7 | 12.6 | 9.9 | 12.8 | 13.8 | 14.21 | 2.03 | <0.01 | |
Leading | 17.2 | 11.2 | 17.2 | 12 | 19.7 | 10.2 | 16.2 | 9.6 | 3.52 | 2.76 | 2.64 | 1.44 | 0.05 | ||
Managing | 23.5 | 15.3 | 24.1 | 12.7 | 20.9 | 11.8 | 24.2 | 14.4 | −3.34 | −2.21 | −2.76 | 1.85 | 0.14 | ||
Doing | 45 | 22.7 | 44 | 20.2 | 35 | 20.2 | 46.9 | 20.3 | −12.15 | −5.54 | −5.73 | 2.77 | 0.80 |
Note(s): SD = standard deviation
‡We considered random-effects linear model (comparing intervention vs control) to account for clustering among learners within the same SME
& Sensitivity analysis: We re-run the analysis using random-effects linear model (see note ‡) without the 4 largest clusters (control = 21, 25; intervention = 24 and 38)
Summary results for learners and SMEs
Hierarchy level | Outcome | Measure | Summary of results |
---|---|---|---|
Level 1 | Secondary: reaction | Reactions Level Post-Program Questionnaire | Learners rated the program as ‘Good’ across 7 data points, describing usefulness of the overall program content; applicability of the program content to your job role; the structure of the program; the variety of the learning activities; the quality standard of the program; the ease of program navigation and the accessibility and availability of the program |
Levels 1 and 2 | Secondary: learning | Completion rate of program | Completion was as expected – 70% of learners were compliant with the treatment |
Levels 1 and 2 | Secondary: reaction | Short Smith Wellbeing (SWELL) questionnaire | Positive change in personality to be reported (compared with control); see appendix 4 |
Levels 1 and 2 | Secondary: reaction | Organizational development opportunities | No significant change in outcome to be reported (compared with control); see appendix 4 |
Levels 1 and 3 | Secondary: learning | Attendance at live learning stations | 55% of events were attended by SME learners, which was as anticipated according to Notion practice |
Level 2 | Secondary: learning | Knowledge assessment | 95% of learners taking the knowledge assessments passed |
Levels 2 and 3 | Secondary: learning | Perception of skill levels | The level of each skill is greater in the intervention group, although there is no significant difference between groups. For example, skill levels in giving feedback and handling difficult conversations, and demonstrating powerful communication skills was greater in the intervention group (39.7 and 43.5%) than in the control group (18.3 and 29%, respectively) at follow-up. These are core competencies of Anon™ (see appendix 5) |
Level 3 | Primary | The proportion of time spent coaching | The proportion of time spent coaching (and leading) was greater in the intervention group (24.3 and 19.7%, respectively) than in the control group (12.6 and 16.2%, respectively), and there is a significant difference between groups (see Table 1) |
Level 3 | Secondary | Perception of coaching | Perceptions of coaching have shifted among the intervention group away from more negative connotations and, their perception of the people who can be impacted by coaching has increased (see appendix 6) |
Levels 3 and 4 | Secondary: behavior | Positive results achieved from implementation of skills | 166 successes have been recorded by 62 individuals |
Level 4 | SME impact | SME data – absenteeism (unexcused absences) | No significant change in outcomes to be reported (see appendix 7) |
Level 4 | SME impact | Annual turnover and the average number of employees for the period | Staff turnover at follow-up was smaller among the intervention group than among the control group, although there is no significant difference between groups when adjusting for baseline values (see appendix 7) |
Level 4 | SME impact | Annual employee losses and the average number of employees on the payroll for the same period | Intervention group SMEs have increased their numbers of FTEs, compared with the control group, although there is no significant difference between groups when adjusting for baseline values (see appendix 7) |
Level 5 | SME impact | Cost–benefit analyses: Economic impacts: Gross Asset Value; Economic costs: cost of delivering the intervention | Intervention group SMEs have increased the Gross Asset Value (GAV), compared with the control group, although there is no significant difference between groups when adjusting for baseline values (see appendix 7) |
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
© Michela Tinelli, Dominic Ashley-Timms, Laura Ashley-Timms and Ruth Phillips. This work is published under http://creativecommons.org/licences/by/4.0/legalcode (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Abstract
Purpose
This article reports the results of a randomized field experiment that tested the effects of a new business intervention among managers of small- and medium-sized enterprises (SMEs) in England.
Design/methodology/approach
Individual managers (learners) were randomly assigned in clusters (companies) to either an intervention group (265 learners; 40 SMEs) receiving a novel virtual, blended training program designed to stimulate a change in management behavior or a no-intervention group (118 learners; 22 SMEs).
Findings
The results show that the primary objective of changing management behavior to use more of an Operational Coaching™ style of management has been achieved (to a statistically significant level), and this is against the backdrop of the devastating COVID-19 pandemic. Positive trends in SME productivity metrics were also observed in the intervention group companies.
Originality/value
These important results could be indicative of the economic and productivity impact that a change in management behavior could have, and they warrant serious further investigation.
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
Details
1 Care Policy Evaluation Centre, The London School of Economics and Political Science, London, UK
2 Notion Limited, Leamington Spa, UK