Exploring Learning Curves in Acupuncture Education Using Vision-Based Needle Tracking

Abstract

Measuring learning curves allows for the inspection of the rate of learning and competency threshold for each individual, training lesson, or training method. In this work, we investigated learning curves in acupuncture needle manipulation training with continuous performance measurement using a vision-based needle training system. We tracked the needle insertion depth of 10 students to investigate their learning curves. The results show that the group-level learning curve was fitted with the Thurstone curve, indicating that students were able to improve their needle insertion skills after repeated practice. Additionally, the analysis of individual learning curves revealed valuable insights into the learning experiences of each participant, highlighting the importance of considering individual differences in learning styles and abilities when designing training programs.

1. Introduction and Background

Acupuncture involves the insertion of one or more needles into specific locations on the surface of the human body [1,2]. Once regarded as a cultural curiosity, acupuncture has gained increasing worldwide attention in recent decades, both in terms of practice and research [3,4,5]. Many randomized clinical trials (RCTs) have been conducted to evaluate the efficacy of acupuncture, with a primary emphasis on pain management [4,6]. Other areas of investigation include arthritis, cancer, pregnancy and labor, mood disorders, stroke, nausea, sleep, and paralysis [4]. In addition to exploring the therapeutic effects of acupuncture, researchers have also studied the characteristics of acupuncture needle manipulation. This technique involves precise and sophisticated hand movements during the insertion of needles into the patient’s skin [7]. Investigating this area is crucial since improper needle manipulation can lead to pain and the bending or breaking of needles [8]. These complications can negatively impact the effectiveness of acupuncture and potentially result in a high rate of patients refusing acupuncture due to the associated discomfort [9]. Therefore, it is essential to study this area to ensure that acupuncture practitioners are properly trained in needle manipulation techniques to minimize the risk of pain and discomfort for their patients.

Proficiency in needle manipulation is crucial for novice acupuncturists, who must undergo extensive training to master essential needle insertion parameters such as insertion length, angle, speed, twisting, and rotating with minimal errors [10,11,12,13]. To ensure safety during the early stages of practice, novice acupuncturists perform acupuncture on phantom models made from materials such as silicone, gel, animal skin, apples, or cucumbers, which simulate human skin [11]. Traditionally, acupuncture educators manually measure these parameters using rulers. Recently, various sensing technologies have been applied to track the needle during insertion and automatically measure needle manipulation parameters [11,14,15]. These technologies can assist in comprehending each individual’s unique learning curve, which depicts how their performance in needle manipulation evolves over time.

Learning curves in health profession education are characterized by a slow start at the beginning, followed by acceleration as trainers become more familiar with the skill, and then a gradual slowing down after reaching the inflection point [16]. This pattern, known as the classic Thurstone curve, is depicted in Figure 1 [16,17]. In reality, the learning curve can be more complex and vary greatly among individuals and can be influenced by various factors such as the difficulty of the task, engagement level, prior knowledge and experience, motivation, and learning style. Some individuals learn a skill quickly at the start but then struggle to improve beyond a certain point, while others may learn at a slower rate initially but continue to make steady progress over time. This variation in the learning curve is reflected on the shape of the curve, the position of the inflection point, and the level of competency achieved. Studying learning curves can provide valuable insights for the development of effective training programs and strategies, as well as the personalization of learning experiences.

A detailed assessment of learning curves requires continuous performance tracking. In this context, sensing technologies are more practical for continuous needle tracking than traditional manual measurements, which rely on rulers in acupuncture training. The most widely used and portable needle tracking device is the Acusensor, which attaches force and motion sensors to the needle [7,12,18,19]. Motion tracking using advanced cameras and spherical markers attached to the needle or fingers can improve measurement accuracy [8,20,21,22,23]. Another highly accurate but expensive sensing approach is using ultrasound images for needle motion tracking [24,25,26,27]. However, attaching sensors to the needle may negatively affect the naturalness of needle manipulation. A more natural needle-tracking approach is computer vision, in which regular cameras track the needling process [15]. This approach improves the naturalness of needle manipulation with acceptable accuracy for specific acupuncture training lessons. Furthermore, it can result in portable and affordable needle-tracking systems that are suitable for large-scale deployment.

Most studies related to performance tracking in acupuncture needle manipulation have been conducted with few measurements [7,12,15,19]. The number of measurements varies from a few to around 20. Therefore, there is a need to investigate learning curves in acupuncture education using continuous performance tracking. This approach is beneficial for inspecting the rate of learning and the competency threshold for each individual, training lesson, or training method. In this work, we investigated the learning curves of novice acupuncturists during needle manipulation training by using a portable vision-based needle tracking system that was deployed in a regular acupuncture training program. The selected lesson was controlling the depth of the acupuncture needle insertion. We assessed the learning curves of the group as a whole and individual students to investigate learning patterns. We also interviewed students to obtain more insight into their learning experiences.

The paper is organized as follows. In Section 2, we present the technical system used to track the learning progress. Then, in Section 3, we describe the method used in this study, including the acupuncture training lesson, the participants, and how their progress was measured. The results of our study are presented in Section 4. Finally, we provide the discussion and conclusion in Section 5 and Section 6, respectively.

2. The Vision-Based Needle Tracking System

The needle tracking system in this work is based on computer vision with OpenCV [28]. From the prototype described in [15], we developed a portable system that can be more easily deployed in acupuncture training classes. The software was improved with a graphical user interface (GUI) allowing educators/experimenters to log the needle manipulation performance of novice acupuncturists.

2.1. Hardware

The hardware of the needle tracking system includes an artificial skin-like pad attached to an aluminum frame where novice acupuncturists practice needle insertion; see Figure 2. A camera is used to capture the needle insertion process. For ease of detecting the needle, the needle handle is painted blue and green. A dark background is placed behind the artificial skin-like pad and a white LED light is placed between the camera and the pad to increase the contrast. Images captured by the camera are processed by a Raspberry-Pi-4-embedded computer running Ubuntu 20.4. A computer screen is connected to the Raspberry Pi 4 via a Micro HDMI cable to display the needle manipulation performance on the GUI. The Raspberry Pi 4 and the computer screen can be replaced by a laptop. The specifications of the hardware components are listed in Table 1.

2.2. Software

The needle is detected using the threshold method to convert RGB to HSV. The needle handle is detected and visualized by a red bounding rectangle; see Figure 3A. The estimation of the width, height, minimum angle of rotation, and center point of the bounding rectangle is executed by Open CV functions [28]. These values of the start frame and the end frame of the insertion action are used to calculate the insertion length and angle; see Figure 3B,C. The insertion depth is the total travel distance of the needle minus the initial space between the needle and the skin surface before insertion. If the needle touches the skin surface before insertion, the initial distance is 0, and the needle insertion depth is equivalent to the needle travel distance. These parameters can be displayed on the GUI and logged by educators/experimenters to track the performance over time.

3. Method

3.1. Acupuncture Training Lesson

As a use case, we selected the lesson on controlling the depth of acupuncture needle insertion, which is one of the essential training lessons in a regular acupuncture training program at Can Tho University of Medicine and Pharmacy (CTUMP); see Figure 4. In this training, the students are required to perform needle insertion at a specific depth and angle between 80 and 90 degrees (ideally 90 degrees). The expected insertion depth error (competency threshold) was set at 0.2 cm by the educators. This training is crucial because the safe needle depth for each acupuncture point is different. Using a safe needle depth and having knowledge of the anatomical location of acupoints can ensure safe needling without puncturing internal organs, blood vessels, and nerves [29,30]. Additionally, the extent of pain relief that can be achieved may vary depending on the insertion depth in different parts of the body [31].

3.2. Participants

We recruited 10 novice acupuncture students (6 females and 4 males) with a mean age of 20.9 years (SD = 1.29), ranging in age from 19 to 22, from CTUMP. These participants had no prior experience with acupuncture. During the training, they underwent needle insertion practice for 200 trials, with the insertion depth randomly selected between 0.2 cm and 2.4 cm. This range was chosen because a depth of 0.1 cm may be too shallow to penetrate the skin surface, and a depth of 2.5 cm represents the maximum possible travel distance of the needle used in this training. This randomization ensured that the participants were exposed to a range of challenging situations, which facilitated effective skill development. The system developed in Section 2 was employed to track the needle insertion depth of all the students during the 100 min training session, enabling us to investigate the learning curves. Short rest periods of 3–5 min were scheduled after every 10 min of continuous practice to reduce the risk of fatigue and ensure the well-being of the participants. After the training session, we conducted interviews with the students to gain a deeper understanding of their learning experiences.

Informed consent was obtained from all participants prior to their inclusion in the study. A silicon-based artificial skin-like pad was used and no human patients were involved. The study was recognized by the Faculty of Traditional Medicine and conducted in accordance with the ethical guidelines provided by CTUMP. We pre-registered the study before data collection on OSF (https://osf.io/ws937, accessed on 8 June 2023).

3.3. Measurement

The performance of each trial was measured by calculating the insertion depth error between the actual needle insertion depth and the required insertion depth, which was randomly generated. A lower insertion depth error value indicates a more accurate and precise needle insertion.

3.4. Data Analysis

The collected data were stored in an Excel format and analyzed using Python using scipy, pingouin, and scikit-misc libraries.

Firstly, in the statistical analysis, we investigated the improvement by comparing the performance at the beginning and end of the training. The performance at the beginning and end of the training was calculated as the mean values of the first and last 15% of the trials, which is equivalent to 30 trials. Normality and homogeneity were checked by Shapiro–Wilk and Bartlett tests to determine the statistical tests. The significance level was set at 0.05.

Secondly, in the visual analysis, we plotted the data at a fine granularity to investigate the learning curves at both the group and individual levels. To smooth the learning curves, we implemented the LOESS method (locally estimated scatterplot smoothing) through the scikit-misc library. After that, we categorized the learning patterns based on the consensus of the authors and the educators.

4. Results

4.1. Statistical Analysis

We investigated whether there was a significant difference in learning performance between the first 30 trials and the last 30 trials of a learning task using Wilcoxon signed-rank tests. The results, as presented in Table 2, indicate a significant improvement in learning performance over time for the entire group. This was demonstrated by a significant reduction in error scores before (Mdn = 0.25, M= 0.26, SD = 0.07) and after (Mdn = 0.16, M = 0.15, SD = 0.04); W = 10.50, p < 0.001, cles = 0.91.

All participants reached or approached the competency level of 0.2 cm. However, a closer analysis of the data revealed that the improvement was not consistent across all participants. While some participants showed a significant improvement in learning performance over time (e.g., P2, P3, P4, P5, P9, and P10), others did not (e.g., P1, P6, P7, and P8). This suggests that different students have different learning patterns that can be further analyzed by examining the visual analysis of the learning curves.

4.2. Visual Analysis

4.2.1. Group-Level Learning Curve with All Individuals

When we used the Thurstone learning curve as a reference to the group-level learning curve in our study, we observed a similar pattern with the first learning part of the model; see Figure 5A. The curve was not stable at the beginning, indicating that participants initially struggled with the task. However, the curve started to stabilize at around 90–100 trials and the accuracy began to increase steadily, indicating that they were learning and improving their performance.

Around 150 trials, the group-level learning curve reached the competency level of 0.2 cm, which is the threshold for accurate performance in this task. This suggests that most participants had reached a level of proficiency that allowed them to perform the task accurately. However, it is important to note that the maximal performance of the group was not reached. This means that some participants may not have reached their personal maximum performance, indicating that there is still room for improvement.

4.2.2. Individual-Level Learning Curves

As can be seen Figure 5B–K, seven out of ten individual learning curves (i.e., P1, P3, P4, P5, P6, P9, and P10) were fitted with the Thurstone learning curve, showing a similar pattern of a low start, rapid improvement, and eventual leveling off as the participants approached their maximum performance. They all reached the competency level within the study’s timeframe. We refer to these as typical learning curves. However, the rates of learning, inflection points, and the time taken to reach the competency levels varied between individuals.

Regarding the rates of learning and the time taken to reach the competency level, P3, P6, P9, and P10 showed a faster rate of improvement compared to P1 and P4. They reached the competency level in fewer trials (less than 50) and demonstrated a more rapid decrease in insertion depth error.

However, regarding reaching their maximum performance, P4, P6, and P9 still had room for improvement, as their learning curves did not plateau by the end of the study. In contrast, P1, P3, P5, P7, and P10 had already reached their maximum levels of accuracy at around 100 trials and did not show significant improvement by the end of the study.

The learning curves of P2, P7, and P8 significantly deviated from the Thurstone model, exhibiting different patterns of learning. As a result, we classified their learning curves as atypical. Interestingly, all three participants began the task at or above the competency level. P2 and P7 exhibited similar learning patterns, starting with a good performance and then experiencing a drop in performance in the middle of the task before recovering at the end. In contrast, P8 showed a consistent performance throughout the training period but did not demonstrate any clear improvement.

Interview data revealed that P2 often experienced physical fatigue and a lack of engagement during the training sessions. These factors may have contributed to the atypical learning curve observed for this participant, with a drop in performance in the middle of the task. P7 reported being highly engaged during the training lesson. A sudden drop in performance for a short period of time was due to a pause during data collection to maintain the measurement system, and it took some time for P7 to regain the proper feeling, which may have contributed to the absence of a clear improvement in performance over time. These reasons may have contributed to the absence of a clear improvement in performance over time. No clear explanation could be drawn for the atypical learning curve observed in P8. While this participant demonstrated a consistent performance throughout the training period, there was no clear improvement in performance over time. Further research may be needed to explore the factors that may have contributed to this pattern of performance.

4.2.3. Group-Level Learning Curve for Individuals with Typical Learning Curves

When we combined the data of individuals with typical learning curves, our analysis suggests that, on average, they reached the competency level at around 50–60 trials and their maximal performance at around 100 trials; see Figure 5L. These numbers can be used as a reference when designing training programs. However, toward the end of the training period, we observed a continued increase in performance. If we track performance over a longer period of time, it may result in a complex learning curve shape. This will be discussed in more detail later.

5. Discussion

5.1. Vision-Based Needle Tracking

The use of a vision-based needle training system was essential in our study, as it allowed for continuous and accurate performance tracking and was instrumental in identifying learning patterns among participants. The vision-based system used a camera to track the needle movements during the training period, resulting in detailed and reliable performance data. This approach provided a more natural and affordable alternative to traditional sensing technologies, such as attaching force and motion sensors to the needle, which can negatively impact the naturalness of needle manipulation. With the vision-based system, we were able to effectively capture and analyze the performance data of individuals, revealing typical and atypical learning curves. This provided valuable insights into the personal characteristics of each individual during learning.

5.2. Statistical vs. Visual Analysis

The study’s results, analyzed both statistically and visually, show that students were able to improve their needle insertion skills through repeated practice. This finding is consistent with previous research, which has demonstrated that training and practice can lead to significant improvements in skill acquisition and performance. However, comparing the statistical analysis using a before–after assessment approach with a visual analysis of the learning curve through continuous performance tracking with fine granularity, the learning curve analysis can provide more insights into individual differences in learning progress and patterns.

5.3. Individual Differences and Personalized Training

The differences among individual learning curves suggest that various factors, such as physical fatigue, engagement levels, prior experience, motivation, and cognitive abilities, can affect the rate and pattern of learning. This highlights the importance of considering individual differences in learning styles and abilities when designing training programs, as a one-size-fits-all approach may not be effective for all learners.

To optimize training outcomes, it is crucial to take into account individual differences in learning styles and abilities and to tailor training programs to meet individual needs. This can be achieved through personalized training programs that account for differences in learning pace, learning style, and cognitive abilities. Additionally, providing feedback and support that are specific to individual learning needs can help to enhance motivation and engagement levels, leading to more effective and efficient skill acquisition. Overall, the findings of this study emphasize the importance of personalized training programs that are tailored to individual needs and that take into account the complex and varied nature of the learning process.

We propose several suggestions for tailoring the training program. Firstly, the duration of the training lesson can be adjusted according to the learning rate and inflection point. Secondly, feedback or notifications can be provided when an abnormal learning process is observed, such as a sudden drop in performance. Thirdly, to increase engagement, a promising approach is gamification in medical education, which uses game elements to enhance learning outcomes and motivation [32]. Gamification benefits learners by increasing their engagement, retention, skills, interaction, and personalization [32,33].

5.4. Beyond the Thurstone Learning Curve

When considering the data of individuals with typical learning curves, we observed a continued increase in performance toward the end of the training period. This suggests that the learning shape could be more complex, and further studies are needed to investigate the longer-term effects of training and the potential for a continued improvement in performance beyond the plateau phase. One possible model for further investigation is the increasing–decreasing return learning curve, which extends the Thurstone learning curve (see Figure 6) [34]. As individuals reach the plateau phase, their performance can be further increased when the skill becomes a habit and an integral part of operations (peak proficiency). Later, the skill becomes automatic, muscle memory, and unforgettable (over learning). This model can serve as a potential framework for understanding the complex shape of the learning curve and the different stages of learning, and could help to guide the development of more effective training programs.

5.5. Limitations

While this study provides valuable insights into the learning process and the effectiveness of the training program, there are several limitations that should be noted. Firstly, the present study only investigated the learning curve of novice medical students during needle insertion depth skills training. Future research could investigate the learning curve with more individual needle manipulation skills. In some learning lessons, students must perform multiple needle manipulation skills simultaneously. Therefore, future studies should also investigate this type of multi-skill practice. Secondly, although the study detected several learning patterns, it had a relatively small sample size. Future studies with larger sample sizes may be needed to confirm the results and to comprehensively examine individual differences in learning. Furthermore, expanding the sample size would enable researchers to investigate the learning patterns of more diverse populations, which could provide a more complete understanding of how different factors influence skill acquisition. Thirdly, the vision-based tracking technique based on the OpenCV method may be impacted by potential inaccuracies caused by lighting changes or non-uniform paint, as well as the calibration process and estimated bounding rectangle. Moreover, it only works with specific needle insertion parameters. Other parameters such as insertion force and twisting may require more complex sensors.

6. Conclusions

Our work highlights the potential of vision-based tracking systems as a tool for studying the learning process in acupuncture education, with the needle insertion depth training lesson serving as a use case. The findings provide insights into the learning process and emphasize the importance of personalized training programs that take into account individual differences in learning styles and abilities. By tailoring training programs to meet the specific needs of each learner, we can enhance motivation, engagement, and overall training effectiveness, leading to more efficient and effective skill acquisition.

Further research is needed to explore the long-term retention of skills and the potential for continued improvement beyond the plateau phase. Additionally, multi-skill practice should be investigated. In addition to tracking the learning curve, factors that may influence learning effectiveness, such as the level of engagement and physical fatigue, should also be studied. By addressing these questions, we can continue to refine our understanding of the learning process and develop more effective training programs that meet the needs of individual learners.

Author Contributions

Conceptualization, D.D.P., T.H.L., L.T.C.T., H.T.N.T. and H.-L.C.; methodology, D.D.P. and H.-L.C.; software, T.H.L. and L.T.C.T.; validation, D.D.P., T.H.L., H.T.N.T. and H.-L.C.; formal analysis, D.D.P. and H.-L.C.; investigation, D.D.P., T.H.L., H.T.N.T. and H.-L.C.; resources, D.D.P., T.H.L. and L.T.C.T.; data curation, H.T.N.T., T.H.L. and D.D.P.; writing—original draft preparation, D.D.P., T.H.L. and H.-L.C.; writing—review and editing, D.D.P., T.H.L. and H.-L.C.; visualization, H.-L.C.; supervision, H.-L.C.; project administration, D.D.P. and T.H.L.; funding acquisition, D.D.P. and T.H.L. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

The authors declare that this research was conducted in full accordance with the Declaration of Helsinki. The study was recognized by the Faculty of Traditional Medicine, Can Tho University of Medicine and Pharmacy (2338/QĐ-ĐHYDCT, 23 August 2022) and followed the ethical guidelines provided by the organization. The needle insertion was not applied to the human body. No human patients were involved.

Informed Consent Statement

Informed consent was obtained from all individual participants involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy issues.

Acknowledgments

The authors would like to thank Tran Minh Man, Le Hong Phuoc, Lam Xuan Truc, Le Dang Quynh Thy, Van Phuong Loan, Le Thi Xuan Hoa, and Nguyen Huynh Thao An for their support in conducting the user study.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Generic learning curve in health profession education. The curve shape, the positions of the inflection point, and the performance levels vary among measured performances and individuals.

Figure 2. The vision-based needle tracking system. (A) The hardware setup. (B) Acupuncture needle manipulation parameters. In this work, we focused on the insertion depth.

Figure 3. Calculation of the needle manipulation parameters with a focus on insertion depth based on the bounding rectangle position during the insertion process. (A) The needle is detected. (B) Start frame before insertion. (C) End frame after insertion.

Figure 4. The training of controlling the depth of acupuncture needle insertion. (A) A student performs needle insertion using the system. The graphical user interface displays needle manipulation parameters. Picture used with participant permission. (B) Four video frames demonstrate the needle insertion captured by the camera.

Figure 5. Learning curves of insertion depth error for the group and individual participants using the LOESS smoothing method, with 95% confidence intervals. (A) Group-level learning curve. (B–K) Individual learning curves with different learning patterns. The learning curves of P2, P7, and P8 are atypical. (L) Group-level learning curve for individuals with typical learning curves. The dashed blue line indicates the competency level at 0.2 cm.

Figure 5. Learning curves of insertion depth error for the group and individual participants using the LOESS smoothing method, with 95% confidence intervals. (A) Group-level learning curve. (B–K) Individual learning curves with different learning patterns. The learning curves of P2, P7, and P8 are atypical. (L) Group-level learning curve for individuals with typical learning curves. The dashed blue line indicates the competency level at 0.2 cm.

Figure 6. The increasing–decreasing return learning curve that goes beyond the Thurstone learning curve. Figure adapted from [34].

Table 1

Hardware specifications.

Components	Model/Manufacturer	Description
Artificial skin-like pad	Henan Green Medical Tech	Silicon-based
Needle	Suzhou Tianxie Acupuncture	Filiform, 50 mm length, handle $\varphi$1 mm, body $\varphi$0.3 mm
Camera	TD-WC3200	1920 × 1080
Computer	Raspberry Pi 4	Ubuntu 20.4, OpenCV
Dark background	Chochen	3 mm thickness
Light	Xiaomi	1.2 W

Table 2

Median, mean, and standard deviation of insertion depth error at the beginning and the end of the training. The data demonstrated improvement at the group level and in six out of ten individuals, suggesting that students may have different learning patterns.

Data	First 30 Trials			Last 30 Trials			W	p	CL Effect Size
	Mdn	M	SD	Mdn	M	SD
Group	0.25	0.26	0.07	0.16	0.15	0.04	10.50	***	0.91
P1	0.20	0.25	0.21	0.20	0.18	0.13	158.50	n.s	0.60
P2	0.30	0.34	0.25	0.20	0.22	0.18	104.00	*	0.65
P3	0.20	0.28	0.20	0.10	0.15	0.13	114.00	*	0.70
P4	0.35	0.39	0.29	0.20	0.22	0.18	75.50	**	0.67
P5	0.20	0.25	0.23	0.10	0.16	0.15	93.50	*	0.67
P6	0.10	0.19	0.24	0.10	0.10	0.11	99.00	n.s	0.59
P7	0.10	0.16	0.15	0.10	0.11	0.10	101.00	n.s	0.58
P8	0.10	0.16	0.15	0.20	0.17	0.11	185.00	n.s	0.46
P9	0.10	0.21	0.20	0.10	0.08	0.08	67.00	**	0.72
P10	0.20	0.32	0.31	0.10	0.16	0.17	114.00	*	0.65

Note. * p < 0.05, ** p < 0.01, *** p < 0.001, n.s. not significant.

References

1. Liu, T. Acupuncture: What underlies needle administration?. Evid.-Based Complement. Altern. Med.; 2009; 6, pp. 185-193. [DOI: https://dx.doi.org/10.1093/ecam/nen002] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18955313]

2. Nasir, L.S. Acupuncture. Prim. Care Clin. Off. Pract.; 2002; 29, pp. 393-405. [DOI: https://dx.doi.org/10.1016/S0095-4543(01)00007-0]

3. Ernst, E. Acupuncture—A critical analysis. J. Intern. Med.; 2006; 259, pp. 125-137. [DOI: https://dx.doi.org/10.1111/j.1365-2796.2005.01584.x]

4. Ma, Y.; Dong, M.; Zhou, K.; Mita, C.; Liu, J.; Wayne, P.M. Publication trends in acupuncture research: A 20-year bibliometric analysis based on PubMed. PLoS ONE; 2016; 11, e0168123. [DOI: https://dx.doi.org/10.1371/journal.pone.0168123]

5. Chau, N.V.; Hu, D. The effects of traditional Chinese medicine on Vietnamese traditional medicine–Talking about Vietnamese traditional medicinal experts, their books and the current situation. Chin. Med. Cult.; 2013; 8, pp. 53-56.

6. Chavez, L.M.; Huang, S.S.; MacDonald, I.; Lin, J.G.; Lee, Y.C.; Chen, Y.H. Mechanisms of acupuncture therapy in ischemic stroke rehabilitation: A literature review of basic studies. Int. J. Mol. Sci.; 2017; 18, 2270. [DOI: https://dx.doi.org/10.3390/ijms18112270]

7. Jung, W.M.; Lim, J.; Lee, I.S.; Park, H.J.; Wallraven, C.; Chae, Y. Sensorimotor learning of acupuncture needle manipulation using visual feedback. PLoS ONE; 2015; 10, e0139340. [DOI: https://dx.doi.org/10.1371/journal.pone.0139340]

8. Li, J.; Grierson, L.E.; Wu, M.X.; Breuer, R.; Carnahan, H. Perceptual motor features of expert acupuncture lifting-thrusting skills. Acupunct. Med.; 2013; 31, pp. 172-177. [DOI: https://dx.doi.org/10.1136/acupmed-2012-010265]

9. Li, N.; Wang, C.; Lv, J. Analysis of factors in refusal to use acupuncture in patients with lumbar disc herniation: A case study of patients in the rehabilitation and acupuncture center of Huaxi Hospital. Chinese Acupuncture & Moxibustion; Foreign Languages Press: Beijing, China, 2009; pp. 65-67.

10. Shin, S.G.; Im, I.C. Satisfaction level of clinical practice and related variables for students in the department of radiology. J. Korea Contents Assoc.; 2010; 10, pp. 276-284. [DOI: https://dx.doi.org/10.5392/JKCA.2010.10.9.276]

11. Jang, J.E.; Lee, Y.S.; Jang, W.S.; Sung, W.S.; Kim, E.J.; Lee, S.D.; Kim, K.H.; Jung, C.Y. Trends in Acupuncture Training Research: Focus on Practical Phantom Models. J. Acupunct. Res.; 2022; 39, pp. 77-88. [DOI: https://dx.doi.org/10.13045/jar.2022.00024]

12. Lee, I.S.; Lee, Y.S.; Park, H.J.; Lee, H.; Chae, Y. Evaluation of phantom-based education system for acupuncture manipulation. PLoS ONE; 2015; 10, e0117992. [DOI: https://dx.doi.org/10.1371/journal.pone.0117992] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25689598]

13. Han, Y.J.; Yi, S.Y.; Lee, Y.J.; Kim, K.H.; Kim, E.J.; Lee, S.D. Quantification of the parameters of twisting–rotating acupuncture manipulation using a needle force measurement system. Integr. Med. Res.; 2015; 4, pp. 57-65. [DOI: https://dx.doi.org/10.1016/j.imr.2015.04.003] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28664111]

14. Lyu, R.; Gao, M.; Yang, H.; Wen, Z.; Tang, W. Stimulation parameters of manual acupuncture and their measurement. Evid.-Based Complement. Altern. Med.; 2019; 2019, 1725936. [DOI: https://dx.doi.org/10.1155/2019/1725936] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31558907]

15. Luu, T.; Cao, H.; Pham, D.; Tran, L.; Verstraten, T. Development and validation of a vision-based needling training system for acupuncture on phantom model. J. Acupunct. Res.; 2023; 40, pp. 44-52. [DOI: https://dx.doi.org/10.13045/jar.2022.00325]

16. Pusic, M.V.; Boutis, K.; Hatala, R.; Cook, D.A. Learning curves in health professions education. Acad. Med.; 2015; 90, pp. 1034-1042. [DOI: https://dx.doi.org/10.1097/ACM.0000000000000681]

17. Thurstone, L.L. The learning curve equation. Psychol. Monogr.; 1919; 26, i-51. [DOI: https://dx.doi.org/10.1037/h0093187]

18. Davis, R.; Churchill, D.; Badger, G.; Dunn, J.; Langevin, H. A new method for quantifying the needling component of acupuncture treatments. BMC Complement. Altern. Med.; 2012; 12, P142. [DOI: https://dx.doi.org/10.1186/1472-6882-12-S1-P142]

19. Lim, J.W.; Jung, W.M.; Lee, I.S.; Seo, Y.J.; Ryu, H.S.; Ryu, Y.H.; Chae, Y.B. Development of acupuncture manipulation education system. J. Acupunct. Res.; 2014; 31, pp. 11-19. [DOI: https://dx.doi.org/10.13045/acupunct.2014049]

20. Tang, W.C.; Yang, H.Y.; Liu, T.Y.; Gao, M.; Xu, G. Motion Video-Based Quantitative Analysis of the ‘lifting-Thrusting—Method: A Comparison between Teachers and Students of Acupuncture. Acupunct. Med.; 2018; 36, pp. 21-28. [DOI: https://dx.doi.org/10.1136/acupmed-2016-011348]

21. Xu, L.L.; Wang, F.; Yang, H.Y.; Tang, W.C. Three-dimensional Finger Motion Tracking during Needling: A Solution for the Kinematic Analysis of Acupuncture Manipulation. J. Vis. Exp. Jove; 2021; 176, e62750.

22. Xu, L.L.; Xie, J.; Yang, H.Y.; Wang, F.; Tang, W.C. Operation Stability Analysis of Basic Acupuncture Manipulation Based on Three-Dimensional Motion Tracking Data. Wirel. Commun. Mob. Comput.; 2022; 2022, 1958984. [DOI: https://dx.doi.org/10.1155/2022/1958984]

23. Zhang, A.; Yan, X.K.; Liu, A.G. An Introduction to A Newly-developed “Acupuncture Needle Manipulation Training-evaluation System” Based on Optical Motion Capture Technique. Zhen Ci Yan Jiu = Acupunct. Res.; 2016; 41, pp. 556-559.

24. Leow, M.Q.H.; Cao, T.; Lee, S.H.E.; Cui, S.L.; Tay, S.C.; Ooi, C.C. Ultrasonography in acupuncture: Potential uses for education and research. Acupunct. Med.; 2016; 34, pp. 320-322. [DOI: https://dx.doi.org/10.1136/acupmed-2016-011182] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27371533]

25. Langevin, H.M.; Konofagou, E.E.; Badger, G.J.; Churchill, D.L.; Fox, J.R.; Ophir, J.; Garra, B.S. Tissue displacements during acupuncture using ultrasound elastography techniques. Ultrasound Med. Biol.; 2004; 30, pp. 1173-1183. [DOI: https://dx.doi.org/10.1016/j.ultrasmedbio.2004.07.010]

26. Konofagou, E.E.; Langevin, H.M. Using ultrasound to understand acupuncture. IEEE Eng. Med. Biol. Mag.; 2005; 24, pp. 41-46. [DOI: https://dx.doi.org/10.1109/MEMB.2005.1411347]

27. Chen, Y.L.; Hou, M.C.; Chang, S.C.; Chuang, K.W.; Lee, P.Y.; Huang, C.C. Development and Evaluation of Inexpensive Ultrasound Using A-Mode and M-Mode Signals to Identify Lung Depth and Avoid Risk of Pneumothorax in Acupuncture. J. Med. Biol. Eng.; 2021; 41, pp. 251-259. [DOI: https://dx.doi.org/10.1007/s40846-020-00584-x]

28. Bradski, G.; Kaehler, A. Learning OpenCV: Computer Vision with the OpenCV Library; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2008.

29. Chou, P.C.; Chu, H.Y.; Lin, J.G. Safe needling depth of acupuncture points. J. Altern. Complement. Med.; 2011; 17, pp. 199-206. [DOI: https://dx.doi.org/10.1089/acm.2010.0192]

30. Lou, X.; Yang, X.; Jiang, S.; Sun, C.; Zhang, R. Study on angle and depth of needle insertion in acupuncture at Zusanli (ST 36). Zhongguo Zhen Jiu = Chinese Acupuncture & Moxibustion; 2006; 26, pp. 483-486.

31. Itoh, K.; Minakawa, Y.; Kitakoji, H. Effect of acupuncture depth on muscle pain. Chin. Med.; 2011; 6, 24. [DOI: https://dx.doi.org/10.1186/1749-8546-6-24]

32. Krishnamurthy, K.; Selvaraj, N.; Gupta, P.; Cyriac, B.; Dhurairaj, P.; Abdullah, A.; Krishnapillai, A.; Lugova, H.; Haque, M.; Xie, S. et al. Benefits of gamification in medical education. Clin. Anat.; 2022; 35, pp. 795-807. [DOI: https://dx.doi.org/10.1002/ca.23916]

33. Falah, J.; Wedyan, M.; Alfalah, S.F.; Abu-Tarboush, M.; Al-Jakheem, A.; Al-Faraneh, M.; Abuhammad, A.; Charissis, V. Identifying the characteristics of virtual reality gamification for complex educational topics. Multimodal Technol. Interact.; 2021; 5, 53. [DOI: https://dx.doi.org/10.3390/mti5090053]

34. Valamis. Learning Curve: Theory, Meaning, Formula, Graphs [2023]. Available online: https://www.valamis.com/hub/learning-curve (accessed on 14 June 2023).