An Examination of Different Methodological Approaches in Student School Behavior Research: The Issue of the Incomparability of Student Self-assessments

Vol.28,No.4(2018)
ENGLISH ISSUE

Abstract

The issue of school educational outcomes measurement is of great concern to both researchers and practitioners. We can distinguish two main types of outcomes: outcomes in the domain of academic achievement (e.g., mathematics, information and communication technologies, and history) and outcomes in the behavioral domain (school discipline). Both types of outcomes are assessed and graded in schools. However, if we were to let different teachers assess the same students’ knowledge and skills, their assessment would frequently differ and the same applies to their assessments of student behavior. The following question arises: How accurately do we measure school educational outcomes? In our study we aim to address the following issues: (a) describe the most commonly used methodological approaches to student school behavior measurement and discuss their strengths and weaknesses. Specifically, we focus on school documentation analysis, interviews, observations, and questionnaire surveys. The section about school documentation analysis focuses on empirical analysis of the selected school documents of particular Czech schools; (b) propose an innovative approach to student school behavior measurement combining student self-reports and peer-reports with the anchoring vignette method to enhance data comparability.


Keywords:
school discipline; academic achievement; self-assessment; bias; anchoring vignette method
References

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213">https://doi.org/10.1037/0033-2909.101.2.213

Bagley, C., & Pritchard, C. (1998). The reduction of problem behaviours and school exclusion in at-risk youth: an experimental study of school social work with cost-benefit analyses. Child and Family Social Work, 3(4), 219–226. https://doi.org/10.1046/j.1365-2206.1998.00101.x">https://doi.org/10.1046/j.1365-2206.1998.00101.x

Bago d’Uva T., O’Donnell, O., & van Doorslaer, E. (2008). Differential Health Reporting by Education Level and its Impact on the Measurement of Health Inequalities among Older Europeans. International Journal of Epidemiology, 37(6), 1375–1383. https://doi.org/10.1093/ije/dyn146">https://doi.org/10.1093/ije/dyn146

Bago d’Uva, T., van Doorslaer, E., Lindeboom, M., & O’Donnell, O. (2008). Does reporting heterogeneity bias the measurement of health disparities? Health Economics, 17(3), 351–375. https://doi.org/10.1002/hec.1269">https://doi.org/10.1002/hec.1269

Bago d’Uva T., Lindeboom, M., O’Donnell, O., & van Doorslaer, E. (2011). Education-related Inequity in Health Care with Heterogeneous Reporting of Health. Journal of the Royal Statistical Association Series A, 174(3), 639–664. https://doi.org/10.1111/j.1467-985X.2011.00706.x">https://doi.org/10.1111/j.1467-985X.2011.00706.x

Baly, M. W., Cornell, D. G., & Lovegrove, P. (2014). A longitudinal investigation of self- and peer reports of bullying victimization across middle school. Psychology in the Schools, 51(3), 217–240. https://doi.org/10.1002/pits.21747">https://doi.org/10.1002/pits.21747

Barrish, H. H., Saunders, M., & Wolf, M. M. (1969). Good behavior game: Effects of individual contingencies for group consequences on disruptive behavior in a classroom. Journal of Applied Behavior Analysis, 2(2), 119–124. https://doi.org/10.1901/jaba.1969.2-119">https://doi.org/10.1901/jaba.1969.2-119

Baumgartner, H., & Steenkamp, J. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840">https://doi.org/10.1509/jmkr.38.2.143.18840

Bendl, S. (1987). Prověřování, hodnocení a klasifikace v českém jazyce na 1. stupni základní školy (Unpublished master’s thesis). Prague: Charles university in Prague.

Bendl, S. (2000). Kázeň na sídlištních školách. Pedagogická orientace, 10(3), 54–64.

Bendl, S. (2002). Které charakteristiky učitele podporují kázeň žáků. Pedagogika, 52(3), 346–363.

Bendl, S. (2011). Kázeňské problémy ve škole. Praha: Triton.

Bohanon, H., Fenning, P., Carney, K. L., Minnis-Kim, M. J., Anderson-Harriss, S., Moroz, K. B., ... Pigott, T. D. (2006). Schoolwide application of positive behavior support in an urban high school. Journal of Positive Behavior Interventions, 8(3), 131–145. https://doi.org/10.1177/10983007060080030201">https://doi.org/10.1177/10983007060080030201

Branson, C. E., & Cornell, D. G. (2009). A comparison of self and peer reports in the assessment of middle school bullying. Journal of Applied School Psychology, 25(1), 5–27. https://doi.org/10.1080/15377900802484133">https://doi.org/10.1080/15377900802484133

Brimble, M., & Stevenson-Clarke, P. (2005). Perceptions of the prevalence and seriousness of academic dishonesty in Australian universities. The Australian Educational Researcher, 32(3), 19–44. https://doi.org/10.1007/BF03216825">https://doi.org/10.1007/BF03216825

Broeckelman-Post, M. A., Tacconelli, A., Guzmán, J., Rios, M., Calero, B., & Latif, F. (2016). Teacher misbehavior and its effects on student interest and engagement. Communication Education, 65(2), 204–212. https://doi.org/10.1080/03634523.2015.1058962">https://doi.org/10.1080/03634523.2015.1058962

Buckley, J. (2009). Cross-national response styles in international educational assessments: Evidence from PISA 2006. New York University: Department of Humanities and Social Sciences in the Professions. Retrieved from: https://edsurveys.rti.org/PISA/documents/Buckley_PISAresponsestyle.pdf

Campbell, A., & Anderson, C. M. (2011). Check‐in/check‐out: A systematic evaluation and component analysis. Journal of applied behavior analysis, 44(2), 315–326. https://doi.org/10.1901/jaba.2011.44-315">https://doi.org/10.1901/jaba.2011.44-315

Casey-Cannon, S., Hayward, C., & Gowen, K. (2001). Middle-school girls' reports of peer victimization: Concerns, consequences, and implications. Professional School Counseling, 5(2), 138–148.

Chen, Ch., Lee, S., & Stevenson, H. W. (1995). Response style and cross-cultural comparisons of rating scales among East Asian and North American students. Psychological Science, 6(3), 170–175. https://doi.org/10.1111/j.1467-9280.1995.tb00327.x">https://doi.org/10.1111/j.1467-9280.1995.tb00327.x

Crothers, L. M., & Levinson, E. M. (2004). Assessment of bullying: A review of methods and instruments. Journal of Counseling & Development, 82(4), 496–503. https://doi.org/10.1002/j.1556-6678.2004.tb00338.x">https://doi.org/10.1002/j.1556-6678.2004.tb00338.x

Czech School Inspectorate. (2016). Výroční zpráva České školní inspekce za školní rok 2015/2016. Retrieved from http://www.csicr.cz/getattachment/cz/Dokumenty/Vyrocni-zpravy/Vyrocni-zprava-Ceske-skolni-inspekce-za-skolni-(2)/Vyrocni_zprava_CSI_2015-2016.pdf

Czech School Inspectorate. (2017). Kvalita a efektivita vzdělávání a vzdělávací soustavy ve školním roce 2016/2017: Výroční zpráva České školní inspekce. Retrieved from http://www.csicr.cz/getattachment/cz/Dokumenty/Vyrocni-zpravy/Kvalita-a-efektivita-vzdelavani-a-vzdelavaci-soust/VZ_CSI_2017_web_new.pdf

Davis, S. F., Drinan, P. F., & Gallant, T. B. (2009). Cheating in school: What we know and what we can do. John Wiley & Sons.

Ding, M., Li, Y., Li, X., & Kulm, G. (2008). Chinese teachers' perceptions of students' classroom misbehaviour. Educational Psychology, 28(3), 305–324. https://doi.org/10.1080/01443410701537866">https://doi.org/10.1080/01443410701537866

Ding, M., Li, Y., Li, X., & Kulm, G. (2010). Chinese teachers' attributions and coping strategies for student classroom misbehaviour. Asia Pacific Journal of Education, 30(3), 321–337.

Epkins, C. C., & Meyers, A. W. (1994). Assessment of childhood depression, anxiety, and aggression: Convergent and discriminant validity of self-, parent-, teacher-, and peer-report measures. Journal of Personality Assessment, 62(2), 364–381. https://doi.org/10.1207/s15327752jpa6202_16">https://doi.org/10.1207/s15327752jpa6202_16

Flower, A., McKenna, J., Muething, C. S., Pedrotty Bryant, D., & Bryant, B. R. (2014). Effects of the good behavior game on classwide off-task behavior in a high school basic algebra resource classroom. Behavior Modification, 38(1), 45–68. https://doi.org/10.1177/0145445513507574">https://doi.org/10.1177/0145445513507574

Fox, C. L., & Boulton, M. J. (2005). The social skills problems of victims of bullying: Self, peer and teacher perceptions. British Journal of Educational Psychology, 75(2), 313–328. https://doi.org/10.1348/000709905X25517">https://doi.org/10.1348/000709905X25517

Hintze, J. M., Volpe, R. J., & Shapiro, E. S. (2002). Best practices in systematic direct observation of student behavior. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp. 993–1006). Bethesda, MD: National Association of School Psychologists.

Hoy, A. W., & Weinstein, C. S. (2006). Student and teacher perspectives on classroom management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice and contemporary issues (pp. 181–222). Mahwah, NJ: Lawrence Erlbaum.

iDnes.cz. (2018). Zastaralé důtky děti ignorují, trestejme i rodiče, navrhují někteří učitelé. Retrieved from https://zpravy.idnes.cz/reditelska-dutka-kazenske-opatreni-zakladni-skola-fwm-/domaci.aspx?c=A180504_161959_domaci_nub

Institute for Information on Education. (2007). Rychlá šetření 1/2007. Copy in possession of authors.

Institute for Information on Education. (2008). Rychlá šetření 2/2008. Copy in possession of authors.

Irvin, L. K., Horner, R. H., Ingram, K., Todd, A. W., Sugai, G., Sampson, N. K., & Boland, J. B. (2006). Using office discipline referral data for decision making about student behavior in elementary and middle schools: An empirical evaluation of validity. Journal of Positive Behavior Interventions, 8(1), 10–23. https://doi.org/10.1177/10983007060080010301">https://doi.org/10.1177/10983007060080010301

Kapteyn, A., Smith, J. P., & van Soest, A. (2007). Vignettes and self-reports of work disability in the US and the Netherlands. American Economic Review, 97(1), 461–473. https://doi.org/10.1257/aer.97.1.461">https://doi.org/10.1257/aer.97.1.461

Kearney, P., Plax, T. G., Hays, E. R., & Ivey, M. J. (1991). College teacher misbehaviors: What students don't like about what teachers say and do. Communication quarterly, 39(4), 309–324. https://doi.org/10.1080/01463379109369808">https://doi.org/10.1080/01463379109369808

Kim, Y. S., Koh, Y. J., & Leventhal, B. L. (2004). Prevalence of school bullying in Korean middle school students. Archives of pediatrics & adolescent medicine, 158(8), 737–741. https://doi.org/10.1001/archpedi.158.8.737">https://doi.org/10.1001/archpedi.158.8.737

King, G., Murray, C., Salomon, J., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 567–583. https://doi.org/10.1017/S000305540400108X">https://doi.org/10.1017/S000305540400108X

Klimusova, H., Buresova, I., & Cermak, I. (2014). Cross-informant agreement and teacher nomination technique in the assessment of children behavior problems. In C. Pracana (Ed.), Psychology Applications & Developments: Advances in psychology and psychological trends series (pp. 174–182). Lisboa: inScience Press.

Koutrouba, K. (2013). Student misbehaviour in secondary education: Greek teachers' views and attitudes. Educational Review, 65(1), 1–19. https://doi.org/10.1080/00131911.2011.628122">https://doi.org/10.1080/00131911.2011.628122

Lannie, A. L., & McCurdy, B. L. (2007). Preventing disruptive behavior in the urban classroom: Effects of the Good Behavior Game on student and teacher behavior. Education and Treatment of Children, 30(1), 85–98. https://doi.org/10.1353/etc.2007.0002">https://doi.org/10.1353/etc.2007.0002

Leadbeater, B., Hoglund, W., & Woods, T. (2003). Changing contexts? The effects of a primary prevention program on classroom levels of peer relational and physical victimization. Journal of Community Psychology, 31(4), 397–418. https://doi.org/10.1002/jcop.10057">https://doi.org/10.1002/jcop.10057

Lindsey, B. C. (2008). Looking at positive behavior interventions and supports through the lens of innovations diffusion. The Innovation Journal: The Public Sector Innovation Journal, 13(2), 1–18.

Lindstrom, W. A., Lease, A. M., & Kamphaus, R. W. (2007). Peer- and self-rated correlates of a teacher-rated typology of child adjustment. Psychology in Schools, 44(6), 1–21. https://doi.org/10.1002/pits.20249">https://doi.org/10.1002/pits.20249

Loeber, R., Green, S. M., & Lahey, B. B. (1990). Mental health professionals’ perception of the utility of children, mothers, and teachers as informants on childhood psychopathology. Journal of Clinical Child Psychology, 19(2), 136–143. https://doi.org/10.1207/s15374424jccp1902_5">https://doi.org/10.1207/s15374424jccp1902_5

Luiselli, J. K., Putnam, R. F., Handler, M. W., & Feinberg, A. B. (2005). Whole-school positive behavior support: Effects on student discipline problems and academic performance. Educational Psychology, 25(2–3), 183–198. https://doi.org/10.1080/0144341042000301265">https://doi.org/10.1080/0144341042000301265

McCabe, D. L., Butterfield, K. D., & Trevino, L. K. (2006). Academic dishonesty in graduate business programs: Prevalence, causes, and proposed action. Academy of Management Learning & Education, 5(3), 294–305. https://doi.org/10.5465/amle.2006.22697018">https://doi.org/10.5465/amle.2006.22697018

McCurdy, B. L., Mannella, M. C., & Eldridge, N. (2003). Positive behavior support in urban schools: Can we prevent the escalation of antisocial behavior? Journal of Positive Behavior Interventions, 5(3), 158–170. https://doi.org/10.1177/10983007030050030501">https://doi.org/10.1177/10983007030050030501

McMahon, S. D., & Washburn, J. J. (2003). Violence prevention: An evaluation of program effects with urban African-American students. The Journal of Primary Prevention, 24(1), 43–62. https://doi.org/10.1023/A:1025075617356">https://doi.org/10.1023/A:1025075617356

Mitchell, R. R., Tingstrom, D. H., Dufrene, B. A., Ford. W. B., & Sterling, H. E. (2015). The effects of the good behavior game with general-education high school students. School Psychology Review, 44(2), 191–207. https://doi.org/10.17105/spr-14-0063.1">https://doi.org/10.17105/spr-14-0063.1

Morrison, G. M., & Skiba, R. (2001). Predicting violence from school misbehavior: Promises and perils. Psychology in the Schools, 38(2), 173–184. https://doi.org/10.1002/pits.1008">https://doi.org/10.1002/pits.1008

Nastasi, B. K., & Schensul, S. L. (2005). Contributions of qualitative research to the validity of intervention research. Journal of School Psychology, 43(3), 177–195. https://doi.org/10.1016/j.jsp.2005.04.003">https://doi.org/10.1016/j.jsp.2005.04.003

Nolan, J. D., Filter, K. J., & Houlihan, D. (2014). Preliminary report: An application of the Good Behavior Game in the developing nation of Belize. School Psychology International, 35(4), 421–428.

Paulhus, D. L., Harms, P. D., Bruce, M. N., & Lysy, D. C. (2003). The over-claiming technique: Measuring self-enhancement independent of ability. Journal of Personality and Social Psychology, 84(4), 890–904. https://doi.org/10.1037/0022-3514.84.4.890">https://doi.org/10.1037/0022-3514.84.4.890

Pellegrini, A. D., & Bartini, M. (2000). An empirical comparison of methods of sampling aggression and victimization in school settings. Journal of Educational Psychology, 92(2), 360–366. https://doi.org/10.1037/0022-0663.92.2.360">https://doi.org/10.1037/0022-0663.92.2.360

Průcha, J. (2005). Česko-anglický pedagogický slovník. Praha: ARSCI.

Smith, R. G., & Gross, A. M. (2006). Bullying: Prevalence and the effect of age and gender. Child & Family Behavior Therapy, 28(4), 13–37. https://doi.org/10.1300/J019v28n04_02">https://doi.org/10.1300/J019v28n04_02

Sugai, G., Sprague, J. R., Horner, R. H., & Walker, H. M. (2000). Preventing school violence: The use of office discipline referrals to assess and monitor school-wide discipline interventions. Journal of Emotional and Behavioral Disorders, 8(2), 94–101. https://doi.org/10.1177/106342660000800205">https://doi.org/10.1177/106342660000800205

Todd, A. W., & Horner, R. H. (2006). Referral form examples. Retrieved from http://www.pbis.org/common/cms/files/NewTeam/Data/ReferralFormExamples_v4.0%5B1%5D.doc

Tomášek, J. (2008). Učitel jako oběť násilí – poznatky z viktimologického šetření na českých středních školách. Pedagogika, 58(4), 379–391.

Tsouloupas, C. N., Carson, R. L., Matthews, R., Grawitch, M. J., & Barber, L. K. (2010). Exploring the association between teachers’ perceived student misbehaviour and emotional exhaustion: The importance of teacher efficacy beliefs and emotion regulation. Educational Psychology, 30(2), 173–189. https://doi.org/10.1080/01443410903494460">https://doi.org/10.1080/01443410903494460

Van de Vijver, F. J. R., & Tanzer, N. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. Revue Européenne de Psychologie Appliquée, 54(2), 119–135. https://doi.org/10.1016/j.erap.2003.12.004">https://doi.org/10.1016/j.erap.2003.12.004

Vonkova, H., & Hullegie, P. (2011). Is the anchoring vignettes method sensitive to the domain and choice of the vignette? Journal of the Royal Statistical Society: Series A, 174(3), 597–620. https://doi.org/10.1111/j.1467-985X.2011.00704.x">https://doi.org/10.1111/j.1467-985X.2011.00704.x

Vonkova, H., Bendl, S., & Papajoanu, O. (2017). How students report dishonest behavior in school: Self-assessment and anchoring vignettes. The Journal of Experimental Education, 85(1), 36–53. https://doi.org/10.1080/00220973.2015.1094438">https://doi.org/10.1080/00220973.2015.1094438

Vonkova, H., Papajoanu, O., & Bendl, S. (2016). Aplikace metody ukotvujících vinět v pedagogickém výzkumu: přehled literatury a metodologická doporučení. Pedagogická Orientace, 26(3), 537–559. https://doi.org/10.5817/PedOr2016-3-537">https://doi.org/10.5817/PedOr2016-3-537

Vonkova, H., Zamarro, G., & Hitt, C. (2018). Cross-country heterogeneity in students’ reporting behavior: The use of the anchoring vignette method. Journal of Educational Measurement, 55(1), 3–31. https://doi.org/10.1111/jedm.12161">https://doi.org/10.1111/jedm.12161

Vrbová, J., & Stuchlíková, I. (2012). Školní podvádění starších žáků – pilotní studie. Pedagogika, 62(3), 317–331.

Wei, H. S., & Huang, C. K. (2005). Reviewing school bullying research: empirical findings and methodical considerations. NTTU Educational Research Journal, 16(1), 69–112.

Weiss, B., Harris, W., & Catron, T. (2002). Development and initial validization of the peer-report measure of internalizing and externalizing behavior. Journal of Abnormal Child Psychology, 30(3), 285–294. https://doi.org/10.1023/A:1015158930705">https://doi.org/10.1023/A:1015158930705

Wright, R. A., & McCurdy, B. L. (2011). Classwide positive behavior support and group contingencies: Examining a positive variation of the Good Behavior Game. Journal of Positive Behavior Interventions, 14(3), 173–180. https://doi.org/10.1177/1098300711421008">https://doi.org/10.1177/1098300711421008

Základní škola a mateřská škola Bílá / ZŠ a MŠ Bílá. (2017). Školní řád. Retrieved from http://www.zsbila.cz/images/stories/o_skole/2017-2018_Skolni_rad_od_1_9_2017.pdf

Základní škola a mateřská škola Ostrava – Bělský Les / ZŠ a MŠ Ostrava – Bělský Les. (2017). Vnitřní klasifikační řád – metodický pokyn. Retrieved from http://www.zsdvorskeho.eu/index.php?option=com_content&view=article&id=208&Itemid=152

Základní škola Jana Wericha / ZŠ Jana Wericha. (2011). Příloha č. 1 Školního řádu ZŠ Jana Wericha – Pravidla pro hodnocení výsledků vzdělávání žáků. Retrieved from http://www.zs-jana-wericha.cz/res/archive/067/007335.pdf?seek=1330878920

Základní škola Jana Wericha / ZŠ Jana Wericha. (2017). Příloha č. 2 ke Školnímu řádu ZŠ Jana Wericha – Bodový systém. Retrieved from http://www.zs-jana-wericha.cz/res/archive/182/019499.pdf?seek=1506052796

Základní škola Petřiny – sever / ZŠ Petřiny – sever. (2015). Školní řád. Retrieved from http://www.zspetriny.cz/soubor-sr-230-.pdf

Základní škola Sázavská / ZŠ Sázavská. (2013). Klasifikační řád. Retrieved from http://www.zssazavska.cz/data/dokumenty/598_klasifikacni_rad_novy.doc

Základní škola Želenice / ZŠ Želenice. (2012). Školní řád. Retrieved from https://www.zszelenice.cz/skola12/skolnirad14.pdf

Metrics

0


752

Views

427

PDF views