An Examination of Different Methodological Approaches in Student School Behavior Research: The Issue of the Incomparability of Student Self-assessments
Vol.28,No.4(2018)
ENGLISH ISSUE
The issue of school educational outcomes measurement is of great concern to both researchers and practitioners. We can distinguish two main types of outcomes: outcomes in the domain of academic achievement (e.g., mathematics, information and communication technologies, and history) and outcomes in the behavioral domain (school discipline). Both types of outcomes are assessed and graded in schools. However, if we were to let different teachers assess the same students’ knowledge and skills, their assessment would frequently differ and the same applies to their assessments of student behavior. The following question arises: How accurately do we measure school educational outcomes? In our study we aim to address the following issues: (a) describe the most commonly used methodological approaches to student school behavior measurement and discuss their strengths and weaknesses. Specifically, we focus on school documentation analysis, interviews, observations, and questionnaire surveys. The section about school documentation analysis focuses on empirical analysis of the selected school documents of particular Czech schools; (b) propose an innovative approach to student school behavior measurement combining student self-reports and peer-reports with the anchoring vignette method to enhance data comparability.
school discipline; academic achievement; self-assessment; bias; anchoring vignette method
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213">https://doi.org/10.1037/0033-2909.101.2.213
Bagley, C., & Pritchard, C. (1998). The reduction of problem behaviours and school exclusion in at-risk youth: an experimental study of school social work with cost-benefit analyses. Child and Family Social Work, 3(4), 219–226. https://doi.org/10.1046/j.1365-2206.1998.00101.x">https://doi.org/10.1046/j.1365-2206.1998.00101.x
Bago d’Uva T., O’Donnell, O., & van Doorslaer, E. (2008). Differential Health Reporting by Education Level and its Impact on the Measurement of Health Inequalities among Older Europeans. International Journal of Epidemiology, 37(6), 1375–1383. https://doi.org/10.1093/ije/dyn146">https://doi.org/10.1093/ije/dyn146
Bago d’Uva, T., van Doorslaer, E., Lindeboom, M., & O’Donnell, O. (2008). Does reporting heterogeneity bias the measurement of health disparities? Health Economics, 17(3), 351–375. https://doi.org/10.1002/hec.1269">https://doi.org/10.1002/hec.1269
Bago d’Uva T., Lindeboom, M., O’Donnell, O., & van Doorslaer, E. (2011). Education-related Inequity in Health Care with Heterogeneous Reporting of Health. Journal of the Royal Statistical Association Series A, 174(3), 639–664. https://doi.org/10.1111/j.1467-985X.2011.00706.x">https://doi.org/10.1111/j.1467-985X.2011.00706.x
Baly, M. W., Cornell, D. G., & Lovegrove, P. (2014). A longitudinal investigation of self- and peer reports of bullying victimization across middle school. Psychology in the Schools, 51(3), 217–240. https://doi.org/10.1002/pits.21747">https://doi.org/10.1002/pits.21747
Barrish, H. H., Saunders, M., & Wolf, M. M. (1969). Good behavior game: Effects of individual contingencies for group consequences on disruptive behavior in a classroom. Journal of Applied Behavior Analysis, 2(2), 119–124. https://doi.org/10.1901/jaba.1969.2-119">https://doi.org/10.1901/jaba.1969.2-119
Baumgartner, H., & Steenkamp, J. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840">https://doi.org/10.1509/jmkr.38.2.143.18840
Bendl, S. (1987). Prověřování, hodnocení a klasifikace v českém jazyce na 1. stupni základní školy (Unpublished master’s thesis). Prague: Charles university in Prague.
Bendl, S. (2000). Kázeň na sídlištních školách. Pedagogická orientace, 10(3), 54–64.
Bendl, S. (2002). Které charakteristiky učitele podporují kázeň žáků. Pedagogika, 52(3), 346–363.
Bendl, S. (2011). Kázeňské problémy ve škole. Praha: Triton.
Bohanon, H., Fenning, P., Carney, K. L., Minnis-Kim, M. J., Anderson-Harriss, S., Moroz, K. B., ... Pigott, T. D. (2006). Schoolwide application of positive behavior support in an urban high school. Journal of Positive Behavior Interventions, 8(3), 131–145. https://doi.org/10.1177/10983007060080030201">https://doi.org/10.1177/10983007060080030201
Branson, C. E., & Cornell, D. G. (2009). A comparison of self and peer reports in the assessment of middle school bullying. Journal of Applied School Psychology, 25(1), 5–27. https://doi.org/10.1080/15377900802484133">https://doi.org/10.1080/15377900802484133
Brimble, M., & Stevenson-Clarke, P. (2005). Perceptions of the prevalence and seriousness of academic dishonesty in Australian universities. The Australian Educational Researcher, 32(3), 19–44. https://doi.org/10.1007/BF03216825">https://doi.org/10.1007/BF03216825
Broeckelman-Post, M. A., Tacconelli, A., Guzmán, J., Rios, M., Calero, B., & Latif, F. (2016). Teacher misbehavior and its effects on student interest and engagement. Communication Education, 65(2), 204–212. https://doi.org/10.1080/03634523.2015.1058962">https://doi.org/10.1080/03634523.2015.1058962
Buckley, J. (2009). Cross-national response styles in international educational assessments: Evidence from PISA 2006. New York University: Department of Humanities and Social Sciences in the Professions. Retrieved from: https://edsurveys.rti.org/PISA/documents/Buckley_PISAresponsestyle.pdf
Campbell, A., & Anderson, C. M. (2011). Check‐in/check‐out: A systematic evaluation and component analysis. Journal of applied behavior analysis, 44(2), 315–326. https://doi.org/10.1901/jaba.2011.44-315">https://doi.org/10.1901/jaba.2011.44-315
Casey-Cannon, S., Hayward, C., & Gowen, K. (2001). Middle-school girls' reports of peer victimization: Concerns, consequences, and implications. Professional School Counseling, 5(2), 138–148.
Chen, Ch., Lee, S., & Stevenson, H. W. (1995). Response style and cross-cultural comparisons of rating scales among East Asian and North American students. Psychological Science, 6(3), 170–175. https://doi.org/10.1111/j.1467-9280.1995.tb00327.x">https://doi.org/10.1111/j.1467-9280.1995.tb00327.x
Crothers, L. M., & Levinson, E. M. (2004). Assessment of bullying: A review of methods and instruments. Journal of Counseling & Development, 82(4), 496–503. https://doi.org/10.1002/j.1556-6678.2004.tb00338.x">https://doi.org/10.1002/j.1556-6678.2004.tb00338.x
Czech School Inspectorate. (2016). Výroční zpráva České školní inspekce za školní rok 2015/2016. Retrieved from http://www.csicr.cz/getattachment/cz/Dokumenty/Vyrocni-zpravy/Vyrocni-zprava-Ceske-skolni-inspekce-za-skolni-(2)/Vyrocni_zprava_CSI_2015-2016.pdf
Czech School Inspectorate. (2017). Kvalita a efektivita vzdělávání a vzdělávací soustavy ve školním roce 2016/2017: Výroční zpráva České školní inspekce. Retrieved from http://www.csicr.cz/getattachment/cz/Dokumenty/Vyrocni-zpravy/Kvalita-a-efektivita-vzdelavani-a-vzdelavaci-soust/VZ_CSI_2017_web_new.pdf
Davis, S. F., Drinan, P. F., & Gallant, T. B. (2009). Cheating in school: What we know and what we can do. John Wiley & Sons.
Ding, M., Li, Y., Li, X., & Kulm, G. (2008). Chinese teachers' perceptions of students' classroom misbehaviour. Educational Psychology, 28(3), 305–324. https://doi.org/10.1080/01443410701537866">https://doi.org/10.1080/01443410701537866
Ding, M., Li, Y., Li, X., & Kulm, G. (2010). Chinese teachers' attributions and coping strategies for student classroom misbehaviour. Asia Pacific Journal of Education, 30(3), 321–337.
Epkins, C. C., & Meyers, A. W. (1994). Assessment of childhood depression, anxiety, and aggression: Convergent and discriminant validity of self-, parent-, teacher-, and peer-report measures. Journal of Personality Assessment, 62(2), 364–381. https://doi.org/10.1207/s15327752jpa6202_16">https://doi.org/10.1207/s15327752jpa6202_16
Flower, A., McKenna, J., Muething, C. S., Pedrotty Bryant, D., & Bryant, B. R. (2014). Effects of the good behavior game on classwide off-task behavior in a high school basic algebra resource classroom. Behavior Modification, 38(1), 45–68. https://doi.org/10.1177/0145445513507574">https://doi.org/10.1177/0145445513507574
Fox, C. L., & Boulton, M. J. (2005). The social skills problems of victims of bullying: Self, peer and teacher perceptions. British Journal of Educational Psychology, 75(2), 313–328. https://doi.org/10.1348/000709905X25517">https://doi.org/10.1348/000709905X25517
Hintze, J. M., Volpe, R. J., & Shapiro, E. S. (2002). Best practices in systematic direct observation of student behavior. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp. 993–1006). Bethesda, MD: National Association of School Psychologists.
Hoy, A. W., & Weinstein, C. S. (2006). Student and teacher perspectives on classroom management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice and contemporary issues (pp. 181–222). Mahwah, NJ: Lawrence Erlbaum.
iDnes.cz. (2018). Zastaralé důtky děti ignorují, trestejme i rodiče, navrhují někteří učitelé. Retrieved from https://zpravy.idnes.cz/reditelska-dutka-kazenske-opatreni-zakladni-skola-fwm-/domaci.aspx?c=A180504_161959_domaci_nub
Institute for Information on Education. (2007). Rychlá šetření 1/2007. Copy in possession of authors.
Institute for Information on Education. (2008). Rychlá šetření 2/2008. Copy in possession of authors.
Irvin, L. K., Horner, R. H., Ingram, K., Todd, A. W., Sugai, G., Sampson, N. K., & Boland, J. B. (2006). Using office discipline referral data for decision making about student behavior in elementary and middle schools: An empirical evaluation of validity. Journal of Positive Behavior Interventions, 8(1), 10–23. https://doi.org/10.1177/10983007060080010301">https://doi.org/10.1177/10983007060080010301
Kapteyn, A., Smith, J. P., & van Soest, A. (2007). Vignettes and self-reports of work disability in the US and the Netherlands. American Economic Review, 97(1), 461–473. https://doi.org/10.1257/aer.97.1.461">https://doi.org/10.1257/aer.97.1.461
Kearney, P., Plax, T. G., Hays, E. R., & Ivey, M. J. (1991). College teacher misbehaviors: What students don't like about what teachers say and do. Communication quarterly, 39(4), 309–324. https://doi.org/10.1080/01463379109369808">https://doi.org/10.1080/01463379109369808
Kim, Y. S., Koh, Y. J., & Leventhal, B. L. (2004). Prevalence of school bullying in Korean middle school students. Archives of pediatrics & adolescent medicine, 158(8), 737–741. https://doi.org/10.1001/archpedi.158.8.737">https://doi.org/10.1001/archpedi.158.8.737
King, G., Murray, C., Salomon, J., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 567–583. https://doi.org/10.1017/S000305540400108X">https://doi.org/10.1017/S000305540400108X
Klimusova, H., Buresova, I., & Cermak, I. (2014). Cross-informant agreement and teacher nomination technique in the assessment of children behavior problems. In C. Pracana (Ed.), Psychology Applications & Developments: Advances in psychology and psychological trends series (pp. 174–182). Lisboa: inScience Press.
Koutrouba, K. (2013). Student misbehaviour in secondary education: Greek teachers' views and attitudes. Educational Review, 65(1), 1–19. https://doi.org/10.1080/00131911.2011.628122">https://doi.org/10.1080/00131911.2011.628122
Lannie, A. L., & McCurdy, B. L. (2007). Preventing disruptive behavior in the urban classroom: Effects of the Good Behavior Game on student and teacher behavior. Education and Treatment of Children, 30(1), 85–98. https://doi.org/10.1353/etc.2007.0002">https://doi.org/10.1353/etc.2007.0002
Leadbeater, B., Hoglund, W., & Woods, T. (2003). Changing contexts? The effects of a primary prevention program on classroom levels of peer relational and physical victimization. Journal of Community Psychology, 31(4), 397–418. https://doi.org/10.1002/jcop.10057">https://doi.org/10.1002/jcop.10057
Lindsey, B. C. (2008). Looking at positive behavior interventions and supports through the lens of innovations diffusion. The Innovation Journal: The Public Sector Innovation Journal, 13(2), 1–18.
Lindstrom, W. A., Lease, A. M., & Kamphaus, R. W. (2007). Peer- and self-rated correlates of a teacher-rated typology of child adjustment. Psychology in Schools, 44(6), 1–21. https://doi.org/10.1002/pits.20249">https://doi.org/10.1002/pits.20249
Loeber, R., Green, S. M., & Lahey, B. B. (1990). Mental health professionals’ perception of the utility of children, mothers, and teachers as informants on childhood psychopathology. Journal of Clinical Child Psychology, 19(2), 136–143. https://doi.org/10.1207/s15374424jccp1902_5">https://doi.org/10.1207/s15374424jccp1902_5
Luiselli, J. K., Putnam, R. F., Handler, M. W., & Feinberg, A. B. (2005). Whole-school positive behavior support: Effects on student discipline problems and academic performance. Educational Psychology, 25(2–3), 183–198. https://doi.org/10.1080/0144341042000301265">https://doi.org/10.1080/0144341042000301265
McCabe, D. L., Butterfield, K. D., & Trevino, L. K. (2006). Academic dishonesty in graduate business programs: Prevalence, causes, and proposed action. Academy of Management Learning & Education, 5(3), 294–305. https://doi.org/10.5465/amle.2006.22697018">https://doi.org/10.5465/amle.2006.22697018
McCurdy, B. L., Mannella, M. C., & Eldridge, N. (2003). Positive behavior support in urban schools: Can we prevent the escalation of antisocial behavior? Journal of Positive Behavior Interventions, 5(3), 158–170. https://doi.org/10.1177/10983007030050030501">https://doi.org/10.1177/10983007030050030501
McMahon, S. D., & Washburn, J. J. (2003). Violence prevention: An evaluation of program effects with urban African-American students. The Journal of Primary Prevention, 24(1), 43–62. https://doi.org/10.1023/A:1025075617356">https://doi.org/10.1023/A:1025075617356
Mitchell, R. R., Tingstrom, D. H., Dufrene, B. A., Ford. W. B., & Sterling, H. E. (2015). The effects of the good behavior game with general-education high school students. School Psychology Review, 44(2), 191–207. https://doi.org/10.17105/spr-14-0063.1">https://doi.org/10.17105/spr-14-0063.1
Morrison, G. M., & Skiba, R. (2001). Predicting violence from school misbehavior: Promises and perils. Psychology in the Schools, 38(2), 173–184. https://doi.org/10.1002/pits.1008">https://doi.org/10.1002/pits.1008
Nastasi, B. K., & Schensul, S. L. (2005). Contributions of qualitative research to the validity of intervention research. Journal of School Psychology, 43(3), 177–195. https://doi.org/10.1016/j.jsp.2005.04.003">https://doi.org/10.1016/j.jsp.2005.04.003
Nolan, J. D., Filter, K. J., & Houlihan, D. (2014). Preliminary report: An application of the Good Behavior Game in the developing nation of Belize. School Psychology International, 35(4), 421–428.
Paulhus, D. L., Harms, P. D., Bruce, M. N., & Lysy, D. C. (2003). The over-claiming technique: Measuring self-enhancement independent of ability. Journal of Personality and Social Psychology, 84(4), 890–904. https://doi.org/10.1037/0022-3514.84.4.890">https://doi.org/10.1037/0022-3514.84.4.890
Pellegrini, A. D., & Bartini, M. (2000). An empirical comparison of methods of sampling aggression and victimization in school settings. Journal of Educational Psychology, 92(2), 360–366. https://doi.org/10.1037/0022-0663.92.2.360">https://doi.org/10.1037/0022-0663.92.2.360
Průcha, J. (2005). Česko-anglický pedagogický slovník. Praha: ARSCI.
Smith, R. G., & Gross, A. M. (2006). Bullying: Prevalence and the effect of age and gender. Child & Family Behavior Therapy, 28(4), 13–37. https://doi.org/10.1300/J019v28n04_02">https://doi.org/10.1300/J019v28n04_02
Sugai, G., Sprague, J. R., Horner, R. H., & Walker, H. M. (2000). Preventing school violence: The use of office discipline referrals to assess and monitor school-wide discipline interventions. Journal of Emotional and Behavioral Disorders, 8(2), 94–101. https://doi.org/10.1177/106342660000800205">https://doi.org/10.1177/106342660000800205
Todd, A. W., & Horner, R. H. (2006). Referral form examples. Retrieved from http://www.pbis.org/common/cms/files/NewTeam/Data/ReferralFormExamples_v4.0%5B1%5D.doc
Tomášek, J. (2008). Učitel jako oběť násilí – poznatky z viktimologického šetření na českých středních školách. Pedagogika, 58(4), 379–391.
Tsouloupas, C. N., Carson, R. L., Matthews, R., Grawitch, M. J., & Barber, L. K. (2010). Exploring the association between teachers’ perceived student misbehaviour and emotional exhaustion: The importance of teacher efficacy beliefs and emotion regulation. Educational Psychology, 30(2), 173–189. https://doi.org/10.1080/01443410903494460">https://doi.org/10.1080/01443410903494460
Van de Vijver, F. J. R., & Tanzer, N. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. Revue Européenne de Psychologie Appliquée, 54(2), 119–135. https://doi.org/10.1016/j.erap.2003.12.004">https://doi.org/10.1016/j.erap.2003.12.004
Vonkova, H., & Hullegie, P. (2011). Is the anchoring vignettes method sensitive to the domain and choice of the vignette? Journal of the Royal Statistical Society: Series A, 174(3), 597–620. https://doi.org/10.1111/j.1467-985X.2011.00704.x">https://doi.org/10.1111/j.1467-985X.2011.00704.x
Vonkova, H., Bendl, S., & Papajoanu, O. (2017). How students report dishonest behavior in school: Self-assessment and anchoring vignettes. The Journal of Experimental Education, 85(1), 36–53. https://doi.org/10.1080/00220973.2015.1094438">https://doi.org/10.1080/00220973.2015.1094438
Vonkova, H., Papajoanu, O., & Bendl, S. (2016). Aplikace metody ukotvujících vinět v pedagogickém výzkumu: přehled literatury a metodologická doporučení. Pedagogická Orientace, 26(3), 537–559. https://doi.org/10.5817/PedOr2016-3-537">https://doi.org/10.5817/PedOr2016-3-537
Vonkova, H., Zamarro, G., & Hitt, C. (2018). Cross-country heterogeneity in students’ reporting behavior: The use of the anchoring vignette method. Journal of Educational Measurement, 55(1), 3–31. https://doi.org/10.1111/jedm.12161">https://doi.org/10.1111/jedm.12161
Vrbová, J., & Stuchlíková, I. (2012). Školní podvádění starších žáků – pilotní studie. Pedagogika, 62(3), 317–331.
Wei, H. S., & Huang, C. K. (2005). Reviewing school bullying research: empirical findings and methodical considerations. NTTU Educational Research Journal, 16(1), 69–112.
Weiss, B., Harris, W., & Catron, T. (2002). Development and initial validization of the peer-report measure of internalizing and externalizing behavior. Journal of Abnormal Child Psychology, 30(3), 285–294. https://doi.org/10.1023/A:1015158930705">https://doi.org/10.1023/A:1015158930705
Wright, R. A., & McCurdy, B. L. (2011). Classwide positive behavior support and group contingencies: Examining a positive variation of the Good Behavior Game. Journal of Positive Behavior Interventions, 14(3), 173–180. https://doi.org/10.1177/1098300711421008">https://doi.org/10.1177/1098300711421008
Základní škola a mateřská škola Bílá / ZŠ a MŠ Bílá. (2017). Školní řád. Retrieved from http://www.zsbila.cz/images/stories/o_skole/2017-2018_Skolni_rad_od_1_9_2017.pdf
Základní škola a mateřská škola Ostrava – Bělský Les / ZŠ a MŠ Ostrava – Bělský Les. (2017). Vnitřní klasifikační řád – metodický pokyn. Retrieved from http://www.zsdvorskeho.eu/index.php?option=com_content&view=article&id=208&Itemid=152
Základní škola Jana Wericha / ZŠ Jana Wericha. (2011). Příloha č. 1 Školního řádu ZŠ Jana Wericha – Pravidla pro hodnocení výsledků vzdělávání žáků. Retrieved from http://www.zs-jana-wericha.cz/res/archive/067/007335.pdf?seek=1330878920
Základní škola Jana Wericha / ZŠ Jana Wericha. (2017). Příloha č. 2 ke Školnímu řádu ZŠ Jana Wericha – Bodový systém. Retrieved from http://www.zs-jana-wericha.cz/res/archive/182/019499.pdf?seek=1506052796
Základní škola Petřiny – sever / ZŠ Petřiny – sever. (2015). Školní řád. Retrieved from http://www.zspetriny.cz/soubor-sr-230-.pdf
Základní škola Sázavská / ZŠ Sázavská. (2013). Klasifikační řád. Retrieved from http://www.zssazavska.cz/data/dokumenty/598_klasifikacni_rad_novy.doc
Základní škola Želenice / ZŠ Želenice. (2012). Školní řád. Retrieved from https://www.zszelenice.cz/skola12/skolnirad14.pdf
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright © 2018 Stanislav Bendl, Hana Voňková, Ondřej Papajoanu, Eva Vaňkátová