Applying the anchoring vignette method in educational research: Literature review and methodological recommendations
Vol.26,No.3(2016)
Questionnaire surveys with (self-)assessment items have been frequently employed in educational research. Respondents can, however, assess the same situations differently, i.e. choose different scale categories. The anchoring vignette method provides a solution to this problem. It helps adjust (self-)assessment for heterogeneity in scale usage and make respondents’ answers comparable. This paper has two aims. First, to provide a review of works in the area of educational research in which the method has been used. Second, to summarize methodological recommendations concerning the development and correct use of anchoring vignettes. The method has been successfully used for measuring students’ dishonest behavior, ICT knowledge and skills, parents’ satisfaction with a school, teacher’s classroom management and teacher’s support. The results suggest that respondents with different characteristics indeed use the scale differently and that the anchoring vignette method can help explain some paradoxical findings concerning relationships between variables. Our methodological recommendations for the use of the method focus on the support of its key assumptions, number and discriminatory power of vignettes, relevancy of their content, and some linguistic aspects of their formulation. Summarizing current applications and contributions of the method to educational research, the authors discuss the possibilities of its future utilization for the measurement of educational concepts.
anchoring vignette method; response consistency; vignette equivalence; discriminatory power; number of vignettes; vignette formulation
Angelini, V., Cavapozzi, D., Corazzini, L., & Paccagnella, O. (2012). Age, health and life satisfaction among older Europeans. Social Indicators Research, 105(2), 293–308. https://doi.org/10.1007/s11205-011-9882-x
Angelini, V., Cavapozzi, D., & Paccagnella, O. (2012). Cross-country differentials in work disability reporting among older Europeans. Social Indicators Research, 105(2), 211–226. https://doi.org/10.1007/s11205-011-9878-6
Au, N., & Lorgelly, P. K. (2014). Anchoring vignettes for health comparisons: An analysis of response consistency. Quality of Life Research, 23(6), 1721–1731. https://doi.org/10.1007/s11136-013-0615-2
Bago d’Uva, T., Lindeboom, M., O’Donnell, O., & van Doorslaer, E. (2011). Slipping anchor? Testing the vignettes approach to identification and correction of reporting heterogeneity. Journal of Human Resources, 46(4), 872–903.
Bago d’Uva, T., O’Donnell, O., & van Doorslaer, E. (2008). Differential health reporting by education level and its impact on the measurement of health inequalities among older Europeans. International Journal of Epidemiology, 37(6), 1375–1383. https://doi.org/10.1093/ije/dyn146
Bonsang, E., & van Soest, A. (2012). Satisfaction with social contacts of older Europeans. Social Indicators Research, https://doi.org/10.(2), 273–292. https://doi.org/10.1007/s11205-011-9886-6
Buckley, J. (2008). Survey context effects in anchoring vignettes. Dostupné z http://www.polmeth.wustl.edu/media/Paper/surveyartifacts.pdf
Buckley, J., & Schneider, M. (2007). Charter schools: Hope or hype? Princeton: Princeton University Press.
Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20(2), 303–315. https://doi.org/10.1086/209351
He, J., & Van de Vijver, F. J. R. (2016). The motivation-achievement paradox in international educational achievement tests: Toward a better understanding. In R. B. King & A. B. I. Bernardo (Eds.), The psychology of Asian learners: A festschrift in honor of David Watkins (s. 253–268). Singapore: Springer Science.
Kapteyn, A., Smith, J. P., & van Soest, A. (2007). Vignettes and self-reports of work disability in the US and the Netherlands. American Economic Review, 97(1), 461–473. https://doi.org/10.1257/aer.97.1.461
Kapteyn, A., Smith, J. P., van Soest, A., & Voňková, H. (2011). Anchoring vignettes and response consistency (RAND Working Paper No. WR-840). Dostupné z http://www.rand.org/content/dam/rand/pubs/working_papers/2011/RAND_WR840.pdf
King, G. (2009). Anchoring vignettes: Frequently asked questions. Dostupné z http://gking.harvard.edu/files/gking/files/vfaq.pdf
King, G., Murray, C., Salomon, J., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 567–583. https://doi.org/10.1017/S000305540400108X
King, G., & Wand, J. (2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis, 15, 46–66. https://doi.org/10.1093/pan/mpl011
Kok, R., Avendano, M., Bago d’Uva, T., & Mackenbach, J. (2012). Can reporting heterogeneity explain differences in depressive symptoms across Europe? Social Indicators Research, (2), 191–210.
Kyllonen, P. C., & Bertling, J. (2013). Innovative questionnaire assessment methods to increase cross-country comparability. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), A handbook of international large-scale assessment data analysis: Background, technical issues, and methods of data analysis (s. 277–286). London: Chapman Hall/CRC Press.
Organisation for Economic Cooperation and Development (OECD). (2013). PISA 2012 results: What makes a school successful? Resources, policies and practices (Volume IV). Annex A6: Anchoring vignettes in PISA 2012 student questionnaire. Dostupné z https://www.oecd.org/pisa/keyfindings/PISA-2012-results-Annex%20A6-VolIII-VolIV.pdf
Peracchi F., & Rossetti C. (2012). Heterogeneity in health responses and anchoring vignettes. Empirical Economics, 42(2), 513–538. https://doi.org/10.1007/s00181-011-0530-8
Voňková, H. (2012). Metoda ukotvujících vinět a možnosti využití v pedagogice. Orbis scholae, 6(1), 27–40. https://doi.org/10.14712/23363177.2015.47
Voňková, H. (2013). Subjektivní hodnocení problémů s pohybem: užití parametrického modelu metody ukotvujících vinět. Orbis scholae, 7(1), 49–66. https://doi.org/10.14712/23363177.2015.25
Voňková, H., Bendl, S., & Papajoanu, O. (2016). How students report dishonest behavior in school: Self-assessment and anchoring vignettes. The Journal of Experimental Education (advance online publication).
Voňková, H., & Hrabák, J. (2015a). Řešení nesrovnatelnosti hodnocení ICT znalostí a dovedností žáků skrze ukotvující viněty. Pedagogika, 65(3), 274–291.
Voňková, H., & Hrabák, J. (2015b). The (in) comparability of ICT knowledge and skill self-assessments among upper secondary school students: The use of the anchoring vignette method. Computers & Education, 85, 191–202. https://doi.org/10.1016/j.compedu.2015.03.003
Voňková, H., & Hullegie, P. (2011). Is the anchoring vignettes method sensitive to the domain and choice of the vignette? Journal of the Royal Statistical Society: Series A, 174(3), 597–620. https://doi.org/10.1111/j.1467-985X.2011.00704.x
Voňková, H., Zamarro, G., DeBerg, V., & Hitt, C. (2015). Comparisons of student perceptions of teacherʼs performance in the classroom: Using parametric anchoring vignette methods for improving comparability (EDRE Working Paper No. 2015-01). Dostupné z http://www.uaedreform.org/downloads/2015/05/comparisons-of-student-perceptions-of-teachers-performance-in-the-classroom-using-parametric-anchoring-vignette-methods-for-improving-comparability.pdf
Wand, J., King, G., & Lau, O. (2011). Anchors: Software for anchoring vignette data. Journal of Statistical Software, 42(3), 1–25. https://doi.org/10.18637/jss.v042.i03
Wei, H. S., & Huang, C. K. (2005). Reviewing school bullying research: Empirical findings and methodical considerations. NTTU Educational Research Journal, 16(1), 69–112.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright © 2016 Hana Voňková, Ondřej Papajoanu, Stanislav Bendl