Measurement Instruments for the Social Sciences

Abstract

[...]multiple measures for the same construct exist, rendering comparisons—for instance, across disciplines—difficult. [...]researchers’ conclusions do not always live up to scrutiny. Journal development When we at GESIS-Leibniz-Institute for the Social Sciences—pondered the launch of a new open access journal—we discovered that the scientific community was open to the idea of a new journal specifically devoted to Measurement Instruments in the Social Sciences (MISS). Six types of articles are acceptable for publication, which may further evolve in the future (for details, see the journal submission guidelines: https://measurementinstrumentssocialscience.biomedcentral.com/submission-guidelines): 1) New measurement instruments present an instrument that may either be completely new or offer a better version of existing measurement approaches. 2) International adaptations of measurement instruments present a tool (e.g., a questionnaire) in different languages to foster the international harmonization of measurement instruments. 3) Validation of measurement instruments presents validation studies on established questionnaires or tests, enhancing what has been known and tested about their psychometric properties. 4) Advances in methodology portray best practice in social-scientific measurement or describe recommended changes in testing and analytical procedures on the basis of scientific evidence. 5) Test reviews focus on a standardized evaluation of a measurement instrument’s characteristics and its materials as well as current evidence of psychometric quality in line with established test criteria and guidelines (e.g., international test commissions).

Editorial

At a first glance, the emergence of a new scientific journal in the ever-growing market of scientific journals may raise eyebrows. Would a new journal rather aid a solution or contribute to the “replicability crisis” recently acknowledged in some social scientific disciplines? Are there not already more scientific outlets than anyone can ever follow? And do not we all have enough items on our reading lists already? At a second and closer look, it becomes clear that if, in the past, effort had been devoted to developing high-quality measurement instruments, it often went unnoticed. Documenting such effort is not a typical goal of journals focusing on substantive research, because measurement is just perceived to be a vehicle to answer substantive questions. In addition, even if an instrument was published, it was most often only recognized and reused in its discipline of origin. A transfer of knowledge among disciplines, even though they aimed to study the same constructs, only seldom happens. A slight imbalance then needs to be addressed; substantive questions can only be meaningfully answered if measurement as such is sound. We reason that a new interdisciplinary journal specifically devoted to disseminating open measurement instruments freely, across cultural and language barriers, while supporting open access, open data, and open methodology can aid researchers around the globe and across many disciplines.

Relevance of sound measurement

Measurement instruments are the central tools to acquire sound and scientifically based knowledge and to make theoretical progress in a variety of social scientific disciplines. Across these disciplines, empirical researchers rely on information that has to be collected in an objective manner, in many cases, to be quantified and statistically analyzed afterwards. Irrespective of different underpinnings in test-theoretical terms, this requires at least a standardized approach of collecting information and integrating (often numerical) survey responses or other participant data, before making inferences at the construct level and quantifying individual differences. This process applies to various disciplines, regardless whether the goal is to describe political systems and societies, implement societal change, prepare individual interventions, or predict economic decision-making in the lab or in the field, to name a few examples.

Regardless whether political surveys, personality assessment, educational testing, changes in societal values, or health screenings are concerned, the adequacy of interpretations of measurement outcomes as well as the correct tracking of trends in societies, the tools we use are fundamental for obtaining meaningful measures. They need to be objective and—ideally—transparent for independent inspection and evaluation. Different research purposes may prolong the parallel existence of instruments measuring the same construct, yet after some time, so one might hope, scientific evolution will have helped some of the “best” measures to shine, whereas others that appear to be weaker will have paled in comparison.

Current challenges

According to our reading, the current state of measurement in the social sciences is suboptimal in some regards: (a) there is multiplicity of instruments in a world full of diverse disciplines; (b) in some cases, the instruments themselves, or existing translations, are not available to the research community; and (c) despite the need for brief measures or cross-cultural adaptations, there is a lack of recognition for this kind of work (e.g., resulting in desk rejections due to the lack of originality), leading to lower effort in this regard.

To elaborate, scientific disciplines exist next to each other and continue to diversify, so a number of measurement approaches compete and the same—or highly related—constructs are being assessed by economists, psychologists, or sociologists, albeit for different purposes. Consequently, multiple measures for the same construct exist, rendering comparisons—for instance, across disciplines—difficult.

Furthermore, there is an unknown number of measurement instruments hidden in file drawers, which is probably true for each discipline. This refers to newly developed instruments as well as to abbreviated measures (Rammstedt & Beierlein, 2014). Whereas long item batteries are often not feasible in large-scale assessments or general population surveys, selecting items in an ad hoc manner is not wise either, yet some existing short scales are hardly visible (Ziegler, Kemper, & Kruyen, 2014). Also, questionnaires that have been translated to a foreign language, or adapted for a different population, context, and culture, rarely find the attention and respect they deserve. The efforts in providing and testing them do not only contribute to replicable science but to the only way that science can shed light on human universals.

Unless a journal is devoted to assessment or measurement instruments, the typical description of measurement instruments is rather succinct. Often crucial psychometric evidence on an instrument is missing (if only for space constraints). Sometimes, even a sound definition of the construct is lacking, not to mention the intended uses, target population, and users of a measurement tool (Ziegler, 2014). Sometimes relevant aspects of a questionnaire are not disclosed (e.g., full item set, item translations/adaptations, instructions, response options, aggregation rules, measurement model). Not surprisingly then, data and/or crucial statistical code used for analyses might be missing, too. This current practice can undermine a thorough investigation of the quality of instruments. It definitely hampers attempts at replicating the original research and conducting further research on the instrument.

Yet even if all the information one seeks were available, the comparability of many tools across cultures and languages, or across subgroups in the same population, might appear rather doubtful. We would like to rest assured that bias does not distort the comparability of measurement outcomes across groups. Some researchers might devote attention to developing cross-culturally valid items from the beginning (e.g., by cognitive pretesting); others might test (or not) for measurement equivalence at least in retrospect (Thalmeyer & Saucier, 2014). Sadly, the current practice is that measurement equivalence is not considered often enough, even in cross-cultural research where this topic should be ranking high on the agenda (Boer, Hanke, & He, 2018). Accordingly, researchers’ conclusions do not always live up to scrutiny.

Journal development

When we at GESIS-Leibniz-Institute for the Social Sciences—pondered the launch of a new open access journal—we discovered that the scientific community was open to the idea of a new journal specifically devoted to Measurement Instruments in the Social Sciences (MISS). We were encouraged in our view that there is a fundamental need in the scientific community addressing the aforementioned challenges and that GESIS is ideally suited for setting up an interdisciplinary journal, for gathering an initial editorial board, and for helping the fields to develop an integrated view on measurement. It goes without saying that our newest addendum to the journal list is not the only outlet to further the cause of measurement in science. Each discipline has established its own flagship.

Nonetheless, we have tried to achieve a combination of features that uniquely characterize the new journal. The open access journal functions as a platform to be shared by various disciplines (e.g., sociology, psychology, education, political science, economics), and the journal disseminates public measurement instruments intended for scientific use across multiple languages and settings. Though focusing predominantly on social surveys for the general population, the instruments may also be relevant for the study of individual differences and useful for specific groups or in specific diagnostic contexts. Submitting authors have to subscribe to high scientific, open-science, and ethical standards (more information can be found at the journal homepage: https://measurementinstrumentssocialscience.biomedcentral.com).

All submissions undergo double-blind peer review. Six types of articles are acceptable for publication, which may further evolve in the future (for details, see the journal submission guidelines: https://measurementinstrumentssocialscience.biomedcentral.com/submission-guidelines):

1) New measurement instruments present an instrument that may either be completely new or offer a better version of existing measurement approaches.

2) International adaptations of measurement instruments present a tool (e.g., a questionnaire) in different languages to foster the international harmonization of measurement instruments.

3) Validation of measurement instruments presents validation studies on established questionnaires or tests, enhancing what has been known and tested about their psychometric properties.

4) Advances in methodology portray best practice in social-scientific measurement or describe recommended changes in testing and analytical procedures on the basis of scientific evidence.

5) Test reviews focus on a standardized evaluation of a measurement instrument’s characteristics and its materials as well as current evidence of psychometric quality in line with established test criteria and guidelines (e.g., international test commissions).

6) Systematic reviews with or without meta-analysis may occasionally be published. These critical assessments of literature and data sources focus on a topic that concerns the use and analysis of measurement instruments in the social sciences. They should emphasize quality indicators such as objectivity of tools, reliability estimates, construct validity, factorial validity/measurement model, predictive validity, or fairness/comparability/equivalence across groups.

7) Meeting reports summarize the major themes of a meeting, symposium, or workshop, focusing on the key developments as well as new discoveries relevant to measurement instruments, changes in best practices regarding measurement and use of measurement instruments, and the application of recently gained knowledge in the social sciences.

Recommendations for authors

On the basis of the requests that our interdisciplinary and international editorial board members have received so far, the first three article types are about to become the most frequent publication types appearing in Measurement Instruments for the Social Sciences for a while. If you feel that your approach does not fit one of the recommended article types and structures, please be in touch with the editors.

It should be evident that any progress in measurement-related issues across the social sciences will always be gradual and incremental. We appreciate you contributing to this worthwhile endeavor by submitting a manuscript or helping out with peer review. Starting from 2018 through 2020, GESIS is sponsoring a number of articles through a full waiver (further details at https://measurementinstrumentssocialscience.biomedcentral.com/submission-guidelines/fees-and-funding). Most of all, we hope to stimulate discussion across fields. Next to submitting your work as a paper to MISS, you can actively contribute to this enterprise by suggesting future special issues or thematic series (also see journal homepage).

References

Boer, D., Hanke, K., & He, J. (2018). On detecting systematic measurement error in cross-cultural research: a review and critical reflection on equivalence and invariance tests. Journal of Cross-Cultural Psychology, 49, 713–734.

Rammstedt, B., & Beierlein, C. (2014). Can’t we make it any shorter? The limits of personality assessment and ways to overcome them. Journal of Individual Differences, 35, 212–220.

Thalmayer, A. G., & Saucier, G. (2014). The questionnaire big six (QB6) in 26 nations: developing cross-culturally applicable big six, big five, and big two inventories. European Journal of Personality, 28, 482–496.

Ziegler, M. (2014). Stop and state your intentions! Let’s not forget the ABC of test construction. European Journal of Psychological Assessment, 30, 239–242.

Ziegler, M., Kemper, C. J., & Kruyen, P. (2014). Short scales: five misunderstandings and ways to overcome them. Journal of Individual Differences, 35, 185–189.