Psychometric Proprieties of a Mobile Application to Measure the Craniovertebral Angle a Validation and Reliability Study

Abstract

The aim of this study was to assess the psychometric properties of the mobile application forward head posture in terms of validity, inter- and intra-rater reliability, minimum detectable change, sensitivity, and specificity to measure craniovertebral angle. In total, 44 subjects (mean age 23.30 ± 4.44 years) were evaluated in the standing position with markers on the tragus and cutaneous prominence of seventh cervical vertebra (C7). We had two experienced and trained physiotherapists assess cervical posture using the mobile application forward head posture and photogrammetry. Intraclass correlation coefficients were used to determine validity and reliability. A contingency table was made to determine sensitivity and specificity. Intra-rater reliability of the mobile application forward head posture had an intraclass correlation coefficient of 0.88. The inter-rater reliability generated an intraclass correlation coefficient of 0.83 to 0.89. Criterion validity data were above 0.82. The minimum detectable change was 4.96° for intra-rater and 5.52° for inter-rater reliability. The smartphone application exhibited 94.4% sensitivity and 84.6% specificity. The smartphone application forward head posture is a valid and reliable tool to measure craniovertebral angle in a standing position and, therefore, could be a useful assessment tool in clinical practice.

1. Introduction

Posture can be defined as the ability to maintain a correct relationship between the different body segments within the limits of stability in relation to environment and activities [1]. An adequate posture maintains a correct body scheme with minimal effort, allowing for balance in the musculoskeletal system [2]. Appropriate craniocervical posture occurs when the head is slightly anterior to the cervical spine so that the earlobe and the shoulder can be joined by a vertical line [3]. In physiotherapy, many assessment tests are performed in a standing position because it does not require excessive effort [4].

Cervical pain is a frequent cause of medical consultation. It is estimated that more than half of the world’s population suffers from cervicalgia at some time in their life [5]. Cervical pain is related to posture and poor body alignment at that position [6,7]. Forward head posture (FHP) is one of the most common postural disorders of the cervical spine [8]. In this posture, there is extension of the upper cervical vertebrae and flexion of the lower ones [9]. This poor alignment is associated with the increased use of electronic devices, among other causes, such as long periods of study in school-aged children or sedentary office work in adults [10].

Disorders at the cervical level, such as FHP, can cause multiple problems, including neck pain and disability. Therefore, analyzing cervical posture and detecting postural disorders are important to determine their correlation with pain or dysfunction and to determine if there are changes after therapeutic interventions in craniocervical positioning [11,12].

Classically, cervical posture has been evaluated by measuring three angles: The craniovertebral angle (CV), the cervical inclination angle, and the inclination angle of the head. The CV angle is the best indicator of FHP [6]. To determine this angle, two references are defined: One line that runs from the swallow of the ear to seventh cervical vertebra (C7) and another horizontal line parallel to the ground that passes only through the spinous apophysis of C7 [6]. An angle less than 50°–53° may indicate FHP [2,6,13]. Thereby, the smaller the CV angle, the greater the disability [14].

CV angle is the most reliable and valid indicator of FHP [15]. However, there is currently no standard method for its measurement. One of the most extensive methods involves the use of a photogrammetry system and the subsequent analysis of the photograph from a sagittal plane using postural evaluation software [6]. This method has many advantages, as it is relatively fast, the image and values obtained are easily preserved, and it is more precise and reliable than visual evaluation alone [15]. Therefore, photogrammetry is considered the ‘gold standard’ for assessing head position [16,17]. A camera and even mobile application [16] can be used for registration.

Mobile applications have become a good alternative to photogrammetry systems because they simplify the process by streamlining image acquisition and data analysis and reducing costs. This study aimed to examine the psychometric properties of the FHP mobile application (FHPapp) in terms of its validity, intra- and inter-rater reliability, minimum detectable change, sensitivity, and specificity in measuring the CV angle.

2. Materials and Methods

2.1. Participants

A total of 44 voluntary subjects from the University of Alcalá (Alcalá de Henares, Spain) were recruited by the nonprobabilistic sampling of consecutive cases. Study subjects were between 18 and 65 years old with or without cervical pain, with no history of a whiplash injury or cervical spine arthrodesis, and did not suffer from tinnitus, vertigo, dizziness, or feelings of instability during the study.

Every participant provided their age, weight (body mass index (BMI) was calculated), presence and neck pain intensity (quantified using the visual analogue scale (VAS)), regular practice of exercise, and hours of exercise per week.

The data was anonymized and the ethical guidelines for clinical research in humans outlined in the Helsinki Declaration (and subsequent revisions) were followed [18]. This study was approved by the “Comité de Ética de Investigación y Experimentación Animal” (Ethical Research and Animal Experimentation Committee) of the Universidad de Alcalá (CEIM/HU/2018/33). All participants were provided with an information sheet and an informed consent sheet prior to the start of the measurements.

2.2. Procedure

FHPapp

FHPapp (Pyeongtaek, South Korea) has been available since 2016 and its objective is to determine the cervical posture by measuring the CV angle. It is an application that requires an Android version 4.0 or higher installed on a smartphone. It currently has 50,000–100,000 downloads.

This application allows users to measure the CV angle from a sideview photograph. The application also describes four phases of head positioning and provides an unvalidated questionnaire to establish the diagnosis of head anteriorization.

2.3. Measurement Procedure

In this study, the CV angle was evaluated with FHPapp on a Xiaomi^® MiA2 (Beijing, China) and with a Canon^® EDS100D camera (Tokyo, Japan). The images were subsequently analyzed using the free Kinovea^® software.

The following evaluation protocol was established: Two markers were placed on the ground separated by 1.5 m, where the camera or smartphone and the subject were located. Reference markers were placed on the spinous process of C7 and in the swallow of the subjects’ ears, which were identified through palpitation. The subject was placed on the floor mark, barefoot and standing, lateral to the evaluator. At this point, two evaluators took photographs in two sessions. The evaluators were experienced and trained physiotherapists.

In the first session, the first evaluator took a picture with a smartphone and then with a camera. Then, the second evaluator performed the same sequence. This was repeated for all participants. The measurement order of the subjects was random. The second measurement was performed one week later using the same protocol.

Once the images were obtained, the smartphone image was introduced in FHPapp to obtain the CV angle. The application provides the complement to the CV angle, thus making further calculation necessary. To determine the CV angle, it is necessary to subtract the angle provided by the application from 360°. The image obtained with the camera was uploaded onto an Asis^® laptop for analysis with the Kinovea software. This program has the advantage of directly providing the CV angle without the need for additional calculation.

Once the CV angle was obtained, all measures—the ones obtained in the analysis with Kinovea^® and the ones calculated with the FHPapp—were classified attending to the presence or not of FHP. The presence of FHP (FHP group) was determined by an angle value lower than 50°, and the absence of FHP (no-FHP group) if the angle value was greater than or equal to 50° [6,15].

2.4. Statistical Analysis

The SPSS program version 25.0 (IBM, Chicago, IL, USA) for Windows was used to perform the statistical analyses. For descriptive variables and those corresponding to the CV angle, a descriptive frequency analysis was performed. From the rest of the variables, the mean was presented with its 95% confidence interval (CI) limits and standard deviation (SD).

The Interclass Correlation Coefficient (ICC) calculation was used with a 95% CI accompanied by the standard error of the mean (SEM) SEM = SD $\sqrt{1-\mathrm{ICC}}$ and its graphical representation in the Bland and Altman diagrams to study intra-observer reliability (facing the measurements made with application by an observer in the first and second session), inter-observer reliability (facing the measures of the first and second session with application of the different observers), and the criterion validity (facing measurements made with FHPapp and photogrammetry in the different sessions) [19]. These measurements were accompanied by the minimum detectable change (MDC) with the formula MDC = 1.96 $\sqrt{2 \mathrm{SEM}}$ [20]. ICC values between 0.50 and 0.69 were considered “moderate,” from 0.70 to 0.89 “high,” and above 0.90 “excellent” [19].

For the study of the sensitivity and specificity of the application as evidence indicating the presence of FHP, a receiver operating characteristic (ROC) curve and a contingency table were used, using photogrammetry with Kinovea^® software as a gold standard [11,17]. Accuracy was assessed using the area under the curve (AUC). Values from 0.50 to 0.70 indicate low precision, 0.70 to 0.90 moderate precision, and higher than 0.90 high precision [21].

The sample size was calculated using the software “Tamaño de la muestra 1.1.” (developed by Pérez Medina et al.). Data were calculated using an intraclass correlation coefficient (ICC) of 0.83 [7], an alpha error of 0.05 and a precision level of 0.16. Using these values, the estimated sample size was 30 participants.

3. Results

A total of 44 subjects were evaluated—21 women and 23 men—with a mean age of 23.30 ± 4.44 years (mean ± SD). A descriptive analysis of the demographic data is shown in Table 1.

After performing the FHP application reliability analysis, a high value of intra-rater reliability was found with an intraclass correlation coefficient (ICC) of 0.88 for one of the observers. The corresponding Bland and Altman graph is shown in Figure 1. The inter-rater reliability was also high, with ICC values ranging from 0.83 to 0.89 and an average of 0.85. The Bland and Altman graph of inter-rater reliability with an ICC of 0.89 is shown in Figure 1. The SEM and MDC values are shown in Table 2 with the intra- and inter-rater reliability scores.

The ICC values obtained for the validity between FHPapp and photogrammetry were above 0.82, with a maximum value of 0.88 (CI: 0.82–0.91) (Table 3). These high values indicate that FHPapp is a valid tool to assess cervical posture.

The AUC was 0.93 (Figure 2). After adjusting the cutoff point to 0.333, the FHPapp demonstrated a sensitivity of 94.4% and specificity of 84.6% for the assessment of FHP (Table 4).

4. Discussion

The objective of this study was to evaluate the validity and intra- and inter-rater reliability of FHPapp. Our results indicated high values for both intra- and inter-rater reliability, and this mobile application also presented a high sensitivity and specificity with values of 94.4% and 84.6%, respectively.

FHPapp is a free mobile application that enables measurement of the CV angle for the analysis of cervical posture in the sagittal plane. In the present study, it was used to measure the CV angle. The protocol involved: (1) Preparation of the location of the camera and the subject; (2) palpation and marking anatomical reference points; (3) realization of the photographs. Our procedure was performed in a standing position based on studies from Shaghayeghfard et al. [6], which the measurement of the CV angle in a seated position was found to lead to classification of “without FHP” in subjects with alterations of said angle.

No software was used in the analysis of the anatomical references. An agreement was reached between evaluators to choose a specific point within the marker (most distal point of the marker attached to the subject’s body). According to Furlanetto et al. [15], using different reference points or evaluation software could affect the angle measurement, which would make it difficult to compare results.

The intra- and inter-rater reliability showed high ICC values in all measurements. These results are in line with other studies in the advanced head measurement. Ruivo et al. [2] found an ICC for inter-rater reliability of 0.83 to 0.88 using the postural software (PAS) and a cutoff point to determine FHP of 52°. Nam et al. [12] compared visual analysis against photogrammetry to obtain an ICC of 0.75 to 0.91 with a cutoff point of 54°. Salahzadeh et al. [15] obtained an ICC of 0.89 to 0.92 when comparing visual evaluation with the analysis of photographs using the Adobe Acrobat Software. When Guan et al. [16] studied the cervical region in different positions with a camera and analyzed the image with the ImageJ software, they obtained an ICC of 0.92 to 0.97. All studies used the same anatomical references as our study (C7 and swallow of the ear) and also waited a week between measurements.

There are many factors that can contribute to the variability in the measurements, such as the evaluators (intra-rater reliability), the patients (intra-subject reliability), and the methodology (instrument-reliability) [3]. The high reliability found in our study could be conditioned by the practice of experienced examiners, as suggested by Nam et al. [12]. However, since reliability was not evaluated among inexperienced examiners, this cannot be conclusively affirmed, as in the study by Cheung et al. [14]. However, despite the high levels of reliability, the reference point for the measurement of the CV angle represents a limitation, since it is subject-dependent and influenced by the palpation of spinous references, as indicated by Dunleavy et al. [10].

Our results related to criterion validity suggest that FHPapp represents an alternative to photogrammetry. In this study, we used an image obtained with a digital camera subsequently treated with the free Kinovea^® Software. Other authors have used three-dimensional movement systems. However, these can be expensive [22]. The main tools used in the clinic to assess CV angle are direct visual observation [3,12] and low-cost technologies such as mobile applications [7,16]. Therefore, the objective of the present study was to determine the psychometric properties of FHPapp for use in the clinical evaluation of the CV angle.

As an FHP reference point, this study used an angle less than 50°, although, according to other studies, an angle less than 53° could be considered as FHP [6,15]. The results obtained in our sample were determined by the cutoff point chosen, as indicated in the study by Bosso et al. [4]. If the reference value changed, the classification of the subjects would also change, as established in the article by Salahzadeh et al. [15].

In this study, an MDC of 4.96 was obtained for intra-rater reliability and 5.52 for inter-rater reliability. These values are similar to those obtained by Dunleavy et al. [10] in their evaluation of the reliability of a device called Optotrak by calculating the ICC. This device assesses the range of cervical movement and was compared with an inclinometer, generating an MDC that ranged between 2.5 and 6.5. Cheung et al. [14] obtained an MDC of 3.31. They evaluated the validity and reliability of the Electronic Head Posture Instrument for measuring cervical position in subjects with and without neck pain, also by calculating the ICC.

According to Gadotti et al. [3] and Nam et al. [12], visual evaluations do not confer sufficient validity or reliability. Therefore, they may not detect small differences over time or in response to an intervention. Photogrammetry and FHPapp do detect such changes, as indicated by the ability to make reproducible measurements over time.

Compared to traditional photogrammetry [16,17], FHPapp was highly sensitive and specific. A sensitivity of 94.4% and specificity of 84.6% reflected that the application is a useful tool for identifying the presence of FHP. When attempting to compare these values, the lack of literature concerning this matter became apparent. There are studies [2,12,15,16] that have analyzed the similarity between intra- and inter-rater measurements, but none have analyzed the sensitivity or specificity of the measuring instrument or software in question. Certain smartphone applications have been used to analyze cervical movement [23] and spine posture [24,25,26]. These studies also failed to provide sensitivity and specificity data on these applications. However, having an easy-to-use, free mobile application to accurately determine the CV angle in clinical practice justifies its use, even in the absence of relevant comparison.

This research presents several limitations. The measurements made in this study were subject-dependent. This can lead to measurement and selection biases if body markers or measuring devices are not placed properly or if the data are misinterpreted. In addition, these biases can affect measurements made with a camera or a phone application. The study participants were all adult-aged and, since neither children nor the elderly were included, the results cannot be extrapolated. Furthermore, the results are not applicable to people with traumatic cervical events or other exclusion criteria used in this study. The present study describes only the alignment of the static spine in a standing position. Therefore, the findings cannot be generalized to alignment during the performance of functional tasks, especially when the upper limbs are in motion or loaded.

5. Conclusions

The use of FHPapp might be suitable for clinical practice and facilitates the measurement of cervical posture due to it is simplicity to use, lack of cost, and high accuracy.

Author Contributions

Conceptualization, D.P.-M.; Formal analysis, G.G.-A.; Investigation, G.G.-A. and M.d.A.V.-C.; Methodology, E.A.-D., M.d.A.V.-C. and D.P.-M.; Project administration, G.G.-A. and D.P.-M.; Resources, T.G.-I. and M.d.A.V.-C.; Software, E.A.-D. and R.C.-d.-l.-C.; Supervision, T.G.-I. and C.C.-R.; Validation, E.A.-D.; Writing —review and editing, T.G.-I. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Acknowledgments

In this section you can acknowledge any support given which is not covered by the author contribution or funding sections. This may include administrative and technical support, or donations in kind (e.g., materials used for experiments).

Conflicts of Interest

The authors declare no conflict of interest.

Figures and Tables

Figure 1. Bland and Altman chart for intra- and inter-observer reliability.

Figure 2. Sensitivity and specificity of FHPapp (COR (Characteristic Operating Receiver) curve).

Table 1

Demographic characteristics and descriptive values.

Variable	Femalen = 21	Malen = 23	Totaln = 44
Age (Years)	22.62 (±4.37)[20.63–24.61]	23.91 (±4.52)[21.96–25.87]	23.30 (±4.44)[21.94–24.65]
Weight (kg)	66.15 (±16.02)[58.86-73.45]	75.31 (±10.76)[70.66–79.97]	70.94 (±14.14)[66.54–75.24]
Height (cm)	166.64 (±5.98)[163.91–169.36]	177.15 (±5.04)[174.97–179.33]	172.13 (±7.60)[169.82–174.44]
Body Mass Index (BMI) (kg/m2)	23.84 (±5.99)[21.11–26.57]	23.93 (±2.68)[22.77–25.09]	23.88 (±4.51)[22.51–25.26]
Cervical Pain	13 (61.90%)	9 (39.13%)	n = 22 (50%)
Visual Analog Scale (VAS) (cm)Only Cervical Pain	2.65 (±1.20)[1.92–3.38]	4.43 (±1.12)[3.57–5.30]	3.38 (±1.45)[2.73–4.02]
Physical Activity Practice	15 (71.42%)	21 (91.30%)	n = 36 (81.81%)
Physical Activity (h/week)Only Physical Activity	5.43 (±3.39)[3.55–7.31]	5.45 (±2.06)[4.51–6.39]	5.44 (±2.65)[4.54–6.34]

Table 2

Intra- and inter-observer reliability.

	Observer	Mean (SD)	ICC (95% CI)	SEM	MDC
Intra-Observer Reliability	12	52.90° (±5.52)52.79° (±5.17)	0.85 (0.74–0.91)0.88 (0.79–0.93)	2.13°1.79°	5.90°4.96°
	Session	Mean (SD)	ICC (95% CI)	SEM	MDC
Inter-Observer Reliability	12	52.87° (±5.89)52.82° (±4.74)	0.89 (0.80–0.93)0.88 (0.80–0.93)	1.95°1.64°	5.40°4.54°

SD: Standard deviation. ICC: Intraclass correlation coefficient. CI: Confidence interval. SEM: Standard error of the mean. MDC: Minimum detectable change.

Table 3

FHPapp criterion validity.

	ICC (95% CI)	SEM	MDC
FHP1 vs. KIN1	0.85 (0.78–0.90)	2.11°	5.84°
FHP2 vs. KIN2	0.88 (0.82–0.91)	1.83°	5.07°

FHP1: The first observation from both observers with FHPapp. FHP2: Addition of the second observation from both observers with FHPapp. KIN1: First observation from both observers with Kinovea. KIN2: Addition of the second observation from both observers with Kinovea. ICC: Intraclass correlation coefficient. CI: Confidence interval. SEM: Standard error of the mean. MDC: Minimum detectable change.

Table 4

Sensitivity and specificity (contingency table).

		FHP Application		Correct Percentage (%)
		No FHP	FHP
Photometry	No FHP	22	4	84.6
	FHP	1	17	94.4

FHP: Forward head posture.

References

1. Shumway-Cook, A.; Woollacott, M.H. Control Motor: Teoría y Aplicaciones Practices; 3rd ed. Lippincott Williams and Wilkins: Baltimore, MD, USA, 2007.

2. Ruivo, R.M.; Pezarat-Correia, P.; Carita, A.I. Intrarater and Interrater Reliability of Photographic Measurement of Upper-Body Standing Posture of Adolescents. J. Manip. Physiol. Ther.; 2015; 38, pp. 74-80. [DOI: https://dx.doi.org/10.1016/j.jmpt.2014.10.009]

3. Gadotti, I.C.; Biasotto-González, D.A. Sensitivity of clinical assessments of sagittal head posture. J. Eval. Clin. Pract.; 2010; 16, pp. 141-144. [DOI: https://dx.doi.org/10.1111/j.1365-2753.2009.01137.x]

4. Bosso, L.R.; Campos Golias, A.R. Rhythmic gymnastics athletes posture: Analysis through photometry. Rev. Bras. Med. Esporte.; 2012; 18, pp. 333-337. [DOI: https://dx.doi.org/10.1590/S1517-86922012000500010]

5. Hospital Da Costa Burela, España, 2005. Cervicalgias y Cervicobraquialgias. Servicio de Traumatología y Cirugía Ortopédica. España. Available online: https://sogacot.org/Documentos/Montero_Cervical.pdf (accessed on 21 July 2020).

6. ShaghayeghFard, B.; Ahmadi, A.; Maroufi, N.; Sarrafzadeh, J. Evaluation of forward head posture in sitting and standing positions. Eur. Spine J.; 2016; 25, pp. 3577-3582. [DOI: https://dx.doi.org/10.1007/s00586-015-4254-x]

7. Boland, D.M.; Neufeld, E.V.; Ruddell, J.; Dolezal, B.A.; Cooper, C.B. Inter- and intra-rater agreement of static posture analysis using a mobile application. J. Phys. Ther. Sci.; 2016; 28, pp. 3398-3402. [DOI: https://dx.doi.org/10.1589/jpts.28.3398]

8. Lee, C.H.; Lee, S.; Shin, G. Reliability of forward head posture evaluation while sitting, standing, walking and running. Hum. Mov. Sci.; 2017; 55, pp. 81-86. [DOI: https://dx.doi.org/10.1016/j.humov.2017.07.008]

9. Yong, M.; Lee, H.; Lee, M. Correlation between head posture and proprioceptive function in the cervical region. J. Phys. Ther. Sci.; 2016; 28, pp. 857-860. [DOI: https://dx.doi.org/10.1589/jpts.28.857]

10. Dunleavy, K.; Neil, J.; Tallon, A.; Adamo, D.E. Reliability and validity of cervical position measurements in individuals with and without chronic neck pain. J. Man. Manip. Ther.; 2015; 23, pp. 188-196. [DOI: https://dx.doi.org/10.1179/2042618614Y.0000000070]

11. Pinzón Ríos, I.D. Cabeza hacia adelante: Una mirada desde la biomecánica y sus implicaciones sobre el movimiento corporal humano. Rev. Univ. Ind. Santander. Salud.; 2015; 47, pp. 75-83.

12. Nam, S.H.; Son, S.M.; Kwon, J.W.; Lee, N.K. The intra- and inter-rater reliabilities of the forward head posture assessment of normal healthy subjects. J. Phys. Ther. Sci.; 2013; 25, pp. 737-739. [DOI: https://dx.doi.org/10.1589/jpts.25.737]

13. Lee, K.; Han, H.; Cheon, S.; Park, S.; Yong, M. The effect of forward head posture on muscle activity during neck protraction and retraction. J. Phys. Ther. Sci.; 2015; 27, pp. 977-979. [DOI: https://dx.doi.org/10.1589/jpts.27.977]

14. Cheung Lau, H.M.; Wing Chiu, T.T.; Lam, T. Clinical measurement of craniovertebral angle by electronic head posture instrument: A test of reliability and validity. Man. Ther.; 2009; 14, pp. 363-368. [DOI: https://dx.doi.org/10.1016/j.math.2008.05.004]

15. Salahzadeh, Z.; Maroufi, N.; Ahmadi, A.; Behtash, H.; Razmjoo, A.; Gohari, M.; Parnianpour, M. Assessment of forward head posture in females: Observational and photogrammetry methods. J. Back Musculoeskelet Rehabil.; 2014; 27, pp. 131-139.

16. Furlanetto, T.S.; Sedrez, J.A.; Candotti, C.T.; Loss, J.F. Photogrammetry as a tool for the postural evaluation of the spine: A systematic review. World J. Orthop.; 2015; 7, pp. 136-148. [DOI: https://dx.doi.org/10.5312/wjo.v7.i2.136] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26925386]

17. Guan, X.; Fan, G.; Wu, X.; Zeng, Y.; Su, H.; Gu, G.; Zhou, Q.; Gu, X.; Zhang, H.; He, S. Photographic measurement of head and cervical posture when viewing mobile phone: A pilot study. Eur. Spine. J.; 2015; 24, pp. 2892-2898. [DOI: https://dx.doi.org/10.1007/s00586-015-4143-3] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26206292]

18. Lau, K.T.; Cheung, K.Y.; Chan, K.B.; Chan, M.H.; Lo, K.Y.; Chiu, T.T. Relationships between sagittal postures of thoracic and cervical spine, presence of neck pain, neck pain severity and disability. Man. Ther.; 2010; 15, pp. 457-462. [DOI: https://dx.doi.org/10.1016/j.math.2010.03.009] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/20430685]

19. Pita-Fernández, S.; Pértega-Díaz, S.; Maseda, E.R. La fiabilidad de las mediciones clínicas: El análisis de concordancia para variables numéricas. Cuad. Atención Primaria; 2003; 10, pp. 290-296.

20. Weir, J.P. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J. Strength Cond. Res.; 2005; 19, pp. 231-240.

21. Haley, S.M.; Fragala-Pinkham, M.A. Interpreting change scores of tests and measures used in physical therapy. Phys. Ther.; 2006; 86, pp. 735-743. [DOI: https://dx.doi.org/10.1093/ptj/86.5.735]

22. Cerda, J.; Cifuentes, L. Using ROC curves in clinical investigation: Theoretical and practical issues. Rev. Chilena. Infectol.; 2012; 29, pp. 138-141. [DOI: https://dx.doi.org/10.4067/S0716-10182012000200003]

23. Bethoux, F.; Bennett, S. Evaluating walking in patients with multiple sclerosis: Which assessment tools are useful in clinical practice?. Int. J. MS Care; 2011; 13, pp. 4-14. [DOI: https://dx.doi.org/10.7224/1537-2073-13.1.4]

24. Szucs, K.A.; Donoso Brown, E.V. Rater reliability and construct validity of a mobile application for posture analysis. J. Phys. Ther. Sci.; 2018; 30, pp. 30-36. [DOI: https://dx.doi.org/10.1589/jpts.30.31]

25. Quek, J.; Brauer, S.G.; Treleaven, J.; Pua, Y.H.; Mentiplay, B.; Clark, R.A. Validity and intra-rater reliability of an Android phone application to measure cervical range-of-motion. J. Neuroeng. Rehab.; 2014; 11, pp. 65-71. [DOI: https://dx.doi.org/10.1186/1743-0003-11-65] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/24742001]

26. Youssef, A.R.; Gumaa, M. Validity and reliability of smartphone applications for clinical assessment of the neuromusculoskeletal system. Expert. Rev. Med. Devices; 2017; 14, pp. 481-493. [DOI: https://dx.doi.org/10.1080/17434440.2017.1325319] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28462674]