Uncovering Students' Problem-Solving Gaps in Basic Chemistry Through Rasch Analysis

Sri Rejeki Dwi Astuti(1), Pramudya Dwi Aristya Putra(2), Nur Ahmad(3), Firdha Yusmar(4), Jadnika Dwi Rakhmawan Amrullah(5), Ika Nur Fitriani(6),

DOI: https://doi.org/10.46843/jiecr.v7i3.3001


(1) Department of Science Education, Faculty of Education, Universitas Jember, Indonesia
(2) Department of Science Education, Faculty of Education, Universitas Jember, Indonesia
(3) Department of Science Education, Faculty of Education, Universitas Jember, Indonesia
(4) Department of Science Education, Faculty of Education, Universitas Jember, Indonesia
(5) Department of Science Education, Faculty of Education, Universitas Jember, Indonesia
(6) School of Chemistry, The University of Sydney, Australia

Abstract

Problem-solving skills are fundamental competencies in science education, particularly in basic chemistry, as they enable students to implement conceptual knowledge to real-life phenomena. This study aims to examine the psychometric properties of the problem-based chemistry test and to describe students' problem-solving skills. The test consisted of five essay-based problem-solving items administered to 123 first-year biology education students enrolled in a general chemistry course. Data were analyzed using EFA with SPSS 25 and Rasch Model with Winsteps Software version 3.73. The results indicated that all items fit the Rasch Model and demonstrated good item reliability, with a range of difficulty levels. However, person reliability was low, suggesting that the test had limited sensitivity in differentiating students' ability levels. Analysis of students' abilities revealed that most students were classified as having low to very low problem-solving skills, while a smaller proportion demonstrated moderate to high ability. A substantial number of persons misfit were identified, indicating irregular response patterns that may be influenced by test-taking behavior or the open-ended nature of the items. These findings suggest that the test is suitable for formative and diagnostic assessment of problem-solving skills in basic chemistry; however, further refinement and additional items are suggested to improve measurement precision.

Keywords

assessment; basic chemistry; problem-solving skills; rasch analysis

Full Text:

PDF

References

Abdellatif, H. (2023). Test Results With and Without Blue Printing: Psychometric Analysis Using the Rasch Model. Educación Médica, 24(3), 100802.

Abo-Khalil, A. G. (2024). Integrating Sustainability into Higher Education Challenges and Opportunities for Universities Worldwide. Heliyon, 10(9), e29946.

Adedoyin, O. O., & Mokobi, T. (2013). Using IRT Psychometric Analysis in Examining the Quality of Junior Certificate Mathematics Multiple Choice Examination Test Items. International Journal of Asian Social Science, 3(4), 992–1011.

Alturki, U., & Aldraiweesh, A. (2023). The Factors Influencing 21st Century Skills and Problem-Solving Skills: The Acceptance of Blackboard as Sustainable Education. Sustainability, 15(17), 12845.

Amanda, F. F., Sumitro, S. B., Lestari, S. R., & Ibrohim. (2024). Enhancing Critical Thinking and Problem Solving Skills by Complexity Science-Problem Based Learning Model. Multidisciplinary Journal of Educational Research, 14(1), 96–114.

Artama, K. K., Budasi, I. G., & Ratminingsih, N. M. (2023). Promoting the 21st Century Skills Using Project-Based Learning. Language Circle: Journal of Language and Literature, 17(2), 325–332.

Asyhari, A., & Sifa’i, M. (2021). Problem-Based Learning to Improve Problem Solving Skill: Is It Effective Enough? Indonesian Journal of Science and Mathematics Education, 4(1), 78–88.

Baker, F. B. (2001). The Basics of Item Response Theory (2nd ed.). ERIC Clearinghouse on Assessment and Evaluation.

Balogun, A. G., Balogun, S. K., & Onyencho, C. V. (2017). Test Anxiety and Academic Performance Among Undergraduates: The Moderating Role of Achievement Motivation. The Spanish Journal of Psychology, 20(14), 1–8.

Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical Considerations for Using Exploratory Factor Analysis in Educational Research. Practical Assessment, Research & Evaluation, 18(6), 1–13.

Biancardi, A., Colasante, A., D’Adamo, I., Daraio, C., Gastaldi, M., & Uricchio, A. F. (2023). Strategies for Developing Sustainable Communities in Higher Education Institutions. Scientific Reports, 13, 20596.

Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research. Frontiers in Public Health, 6, 149.

Bolbolian, M., Asgari, S., Sefidi, F., & Zadeh, A. S. (2021). The Relationship Between Test Anxiety and Academic Procrastination Among the Dental Students. Journal of Education and Health Promotion, 10, 1–6.

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Routledge.

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch Analysis in the Human Sciences. Springer.

Boran, M., & Karakus, F. (2022). The Mediator Role of Critical Thinking Disposition in the Relationship Between Perceived Problem-Solving Skills and Metacognitive Awareness of Gifted and Talent Students. Participatory Educational Research, 9(1), 61–72.

Braeken, J., & van Laar, S. (2025). Reducing Calibration Bias for Person Fit Assessment by Mixture Model Expansion. Educational and Psychological Measurement, 85(1), 1–24.

Bueno, P. M. (2014). Assessment of Achievement in Problem-Solving Skills in a General Chemistry. Journal of Technology and Science Education, 4(4), 260–269.

Centoni, M., & Maruotti, A. (2021). Students' Evaluation of Academic Courses: An Exploratory Analysis to an Italian Case Study. Studies in Educational Evaluation, 70, 101054.

Chi, L. T., & Ninh, T. T. (2018). Assessment of Problem Solving Ability and Creativity in Chemistry Teaching at Secondary Schools in Binh Dinh, Vietnam. American Journal of Educational Research, 6(6), 757–762.

Contrino, M. F., Reyes-Millán, M., Vázquez-Villegas, P., & Membrillo-Hernández, J. (2024). Using an Adaptive Learning Tool to Improve Student Performance and Satisfaction in Online and Face-to-Face Education for a More Personalized Approach. Smart Learning Environments, 11, 6.

Costales, J. A., Catulay, J. J., Bermudez, N. P., & Costales, J. A. (2022). Kaiser-Meyer-Olkin Factor Analysis: A Quantitative Approach on Mobile Gaming Addiction Using Random Forest Classifier. Proceedings of the 6th International Conference on Information System and Data Mining (pp. 18–24). Association for Computing Machinery.

Creswell, J. W. (2009). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (3rd ed.). Sage Publications.

Curtis, D. D. (2001). Misfits: People and Their Problems. What Might It All Mean?. International Education Journal, 2(4), 91–99.

Davis, D. R., & Boone, W. (2021). Using Rasch Analysis to Evaluate the Psychometric Functioning of the Other-Directed, Lighthearted, Intellectual, and Whimsical (OLIW) Adult Playfulness Scale. International Journal of Educational Research Open, 2, 100054.

Demitra, & Sarjoko. (2018). Effects of Handep Cooperative Learning Based on Indigenous Knowledge on Mathematical Problem Solving Skill. International Journal of Instruction, 11(2), 103–114.

Dikmen, M. (2023). Test Anxiety in Online Exams: Scale Development and Validity. Current Psychology, 42, 30210–30222.

Doleck, T., Bazelais, P., Lemay, D. J., Saxena, A., & Basne, R. B. (2017). Algorithmic Thinking, Cooperativity, Creativity, Critical Thinking, and Problem Solving: Exploring the Relationship Between Computational Thinking Skills and Academic Performance. Journal of Computers in Education, 4(4), 355–369.

Dünya, B. A. (2024). Adapting the Person Fit Analysis: Ideas on Detecting Person Misfit in Computerized Adaptive Testing. Journal of Measurement and Evaluation in Education and Psychology, 15(1), 1–4.

Effatpanah, F., Baghaei, P., Ravand, H., & Kunina-Habenicht, O. (2024). Fitting the Mixed Rasch Model to the Listening Comprehension Section of the IELTS: Identifying Latent Class Differential Item Functioning. International Journal of Testing, 25(1), 50–89.

Eidesen, P. B., Bjune, A. E., & Lang, S. I. (2023). "Show Me How to Use a Microscope" – The Development and Evaluation of Certification as Direct Assessment of Practical Lab Skills. Ecology and Evolution, 13(10), e10592.

Embretson, S. E., & Reise, S. P. (2013). Item Response Theory for Psychologists. Psychology Press.

Fährmann, K., Köhler, C., Hartig, J., & Heine, J. H. (2022). Practical Significance of Item Misfit and Its Manifestations in Constructs Assessed in Large-Scale Studies. Large-Scale Assessments in Education, 10(7).

Fatmawati, B., Marzuki, M., Roshayanti, F., & Suprapto, P. K. (2024). Fostering Students' Problem-Solving Skills Through Biology Learning Model Integrated With Kurikulum Merdeka. Jurnal Pendidikan Biologi Indonesia, 10(2), 392–403.

Fauzie, M., Pada, A. U., & Supriatno. (2021). Analysis of the Difficulty Index of Item Bank According to Cognitive Aspects During the COVID-19 Pandemic. Jurnal Penelitian dan Evaluasi Pendidikan, 25(2), 162–169.

Fischer, L., Rohm, T., Carstensen, C. H., & Gnambs, T. (2021). Linking of Rasch-Scaled Tests: Consequences of Limited Item Pools and Model Misfit. Frontiers in Psychology, 12, 633896.

Fox, J. P., & Marianti, S. (2016). Joint Modeling of Ability and Differential Speed Using Responses and Response Times. Multivariate Behavioral Research, 51(4), 540–553.

Fr'echette-Simard, C., Plante, I., Duchesne, S., & Chaffee, K. E. (2023). A Latent Growth Analysis of Individual Factors Predicting Test Anxiety During the Transition From Elementary to Secondary School. Journal of Early Adolescence, 43(3), 265–293.

Gebremichael, M. W., Baraki, B., Mehari, A. B., & Assalfew, B. (2025). Item Analysis of Multiple Choice Questions From Assessment of Health Sciences Students, Tigray, Ethiopia. BMC Medical Education, 25.

Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless Responding in Questionnaire Measures: Detection, Impact, and Remedies. The Leadership Quarterly, 31(4), 101384.

Hambleton, R. K., & Swaminathan, H. (1985). Item Response Theory: Principles and Applications. Kluwer Academic Publishers.

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. Sage Publications.

Hattie, J. (1985). Methodology Review: Assessing Unidimensionality of Tests and Items. Applied Psychological Measurement, 9(2), 139–164.

Hmelo-Silver, C. E. (2004). Problem-Based Learning: What and How Do Students Learn?. Educational Psychology Review, 16, 235–266.

Hrnji?i?, A., & Alihodži?, A. (2024). Measuring Students' Conceptual Understanding of Real Functions: A Rasch Model Analysis. International Electronic Journal of Mathematics Education, 19(1), em0761.

Hsu, J. L., Sung, R. J., Swarat, S. L., Gore, A. J., Kim, S., & Lo, S. M. (2024). Variations in Student Approaches to Problem Solving in Undergraduate Biology Education. CBE—Life Sciences Education, 23(2), 1–19.

Jamshidi, S., Rezaei, S., Hassanzadeh, M., & Dehqan, M. (2019). Development and Validation of an Authorial Identity Model and Questionnaire: A Factor Analytic Approach. Issues in Language Teaching, 8(2), 243–273.

Karabatsos, G. (2003). Comparing the Aberrant Response Detection Performance of Thirty-Six Person-Fit Statistics. Applied Measurement in Education, 16(4), 277–298.

Karan, E., & Brown, L. (2022). Enhancing Student's Problem-Solving Skills Through Project-Based Learning. Journal of Problem Based Learning in Higher Education, 10(1), 74–87.

Küçükayd?n, M. A., & Ayaz, E. (2025). Validation of the Scientific Reasoning Competencies Instrument: Relationships With Epistemological Beliefs. Instrument: Relationships with Epistemological Beliefs, 23.

Kusuma, I. Y., Triwibowo, D. N., Pratiwi, A. D., & Pitaloka, D. A. (2022). Rasch Modelling to Assess Psychometric Validation of the Knowledge About Tuberculosis Questionnaire (KATUB-Q) for the General Population in Indonesia. International Journal of Environmental Research and Public Health, 19(24), 16753.

Lahza, H., Smith, T. G., & Khosravi, H. (2022). Beyond Item Analysis: Connecting Student Behaviour and Performance Using E-Assessment Logs. British Journal of Educational Technology, 54, 355–354.

Lee, H. Y., Chen, Y. L., & Weng, L. J. (2023). Brief Research Report: Effects of Sampling Error and Categorization on Estimation of Measure of Sampling Adequacy. The Journal of Experimental Education, 92(2), 398–410.

Lestari, S., & Yusuf, F. N. (2025). Aligning Assessment Practices With Learning Objectives: A Case of EFL Classes in Indonesia. Indonesian Journal of English Language Teaching and Applied Linguistics, 10(1).

Linacre, J. M. (1995). The Effect of Misfit on Measurement. The Annual Meeting of the International Objective Measurement Workshop (pp. 1–6).

Linacre, J. M. (2004). Rasch Model Estimation: Further Topics. Journal of Applied Measurement, 5(1).

Linacre, J. M. (2022). Optimizing Rating Scale Category Effectiveness. Journal of Applied Measurement, 3(1).

Linacre, J. M. (2023). User Guide to Winsteps Ministep: Rasch-Model Computer Program. Winsteps.

Martin, T. J., Serrano-Estrada, L., & Esteve-Faubel, J. M. (2020). Critical Thinking: A Base for Urban Sustainable Development. Higher Education Research & Development, 40(2), 309–324.

Meijer, R. R. (1996). The Influence of the Presence of Deviant Item Score Patterns on the Power of a Person-Fit Statistic. Applied Psychological Measurement, 20, 141–154.

Mendez, L., Rivera, A. T., Vasquez, I., Aguilar, A. G., Owens, M. T., & Owens, C. L. (2024). How Students Taking Introductory Biology Experience the Chemistry Content. Journal of Microbiology & Biology Education, 25(3).

Mousavi, A., & Cui, Y. (2020). The Effect of Person Misfit on Item Parameter Estimation and Classification Accuracy: A Simulation Study. Education Sciences, 10, 0324.

Müller, M. (2020). Item Fit Statistics for Rasch Analysis: Can We Trust Them?. Journal of Statistical Distribution and Applications, 7(5).

Nansen, C. (2024). Active Learning, Living Laboratories, Student Empowerment, and Urban Sustainability. Sustainability, 16(10), 3902.

Nkansah, B. K. (2018). On the Kaiser-Meyer-Olkin's Measure of Sampling Adequacy. Mathematical Theory and Modeling, 8(7), 52–76.

Nold, H. (2017). Using Critical Thinking Teaching Methods to Increase Student Success: An Action Research Project. International Journal of Teaching and Learning in Higher Education, 29(1), 17–32.

Ntumi, S. (2025). Using Hierarchical Multiple Regression to Model the Impact of AI-Powered Adaptive Testing on Student Academic Fortunes and Test Anxiety in the Ghanaian Context. Discover Education, 141, 4.

Patton, J. M., Cheng, Y., Hong, M., & Diao, Q. (2019). Detection and Treatment of Careless Responses to Improve Item Parameter Estimation. Journal of Educational and Behavioral Statistics, 44(3), 309–341.

Patyka, N., Gryschenko, O., Kucher, A., He?dak, M., & Raszka, B. (2021). Assessment of the Degree of Factors Impact on Employment in Ukraine's Agriculture. Sustainability, 13(2), 564.

Pitaloka, D. A., Kusuma, I. Y., Pratiwi, H., & Pradipta, I. S. (2023). Development and Validation of Assessment Instrument for the Perception and Attitude Toward Tuberculosis Among the General Population in Indonesia: A Rasch Analysis of Psychometric Properties. Frontiers in Public Health, 11, 1143120.

Planinic, M., Boone, W. J., Susac, A., & Ivanjek, L. (2019). Rasch Analysis in Physics Education Research: Why Measurement Matters. Physical Review Physics Education Research, 15, 020111.

Polat, M. (2020). Analysis of Multiple-Choice Versus Open-Ended Questions in Language Tests According to Different Cognitive Domain Levels. Novitas-ROYAL (Research on Youth and Language), 14(2), 76–96.

Post, M. W., Fellinghauer, C. S., Charlifue, S., New, P. W., Forchheimer, M. B., & Tate, D. G. (2022). Rasch Analysis of the International Quality of Life Basic Data Set Version 2.0. Archives of Physical Medicine and Rehabilitation, 103, 2120–2130.

Price, E., Nicholson, D. T., Dunk, R., Lawler, C., Carney, M., ... & Mottram, S. (2024). Enabling Change Agents for Sustainable Development: A Whole-Institution Approach to Embedding the UN Sustainable Development Goals in Higher Education. International Journal of Sustainability in Higher Education, 25(7), 1333–1350.

Rad, H. S. (2025). Reinforcing L2 Reading Comprehension Through Artificial Intelligence Intervention: Refining Engagement to Foster Self-Regulated Learning. Smart Learning Environments, 12, 23.

Reckase, M. D. (1979). Unifactor Latent Trait Models Applied to Multifactor Tests: Results and Implications. Journal of Educational Statistics, 4(3), 207–230.

Rezigalla, A. A., Eleragi, A. M., Elhussein, A. B., Alfaifi, J., ALGhamdi, M. A., ... & Adam, M. I. (2024). Item Analysis: The Impact of Distractor Efficiency on the Difficulty Index and Discrimination Power of Multiple-Choice Items. BMC Medical Education, 24, 445.

Riaz, S., Kousar, S., Saddique, R., & Rafiq, M. (2023). An Analysis of Laboratory Skills Among University Science Students. iRASD Journal of Educational Research, 4(1), 10–17.

Riyadi., Syarifah, T. J., & Nikmaturrohmah, P. (2021). Profile of Students' Problem-Solving Skills Viewed From Polya's Four Steps Approach and Elementary School Students. European Journal of Educational Research, 10(4), 1625–1638.

Rodríguez-Zurita, D., Jaya-Montalvo, M., Moreira-Arboleda, J., Díez, E., & Carrión-Mero, P. (2025). Sustainable Development Through Service Learning and Community Engagement in Higher Education: A Systematic Literature Review. International Journal of Sustainability in Higher Education, 25(1), 158–201.

Rosak-Szyrocka, J., & Knop, K. (2024). Sustainable Development Universities-Opportunities & Challenges on the Road to Society 5.0 From the Key Stakeholder Perspective. PLoS One, 19(8), e0308929.

Sa-ngiemjit, M., Vázquez-Alonso, A., & Mas, M. A. (2024). Problem-Solving Skills of High School Students in Chemistry. International Journal of Evaluation and Research in Education, 13(3), 1825–1831.

Shadle, S. E., Brown, E. C., Towns, M. H., & Warner, D. L. (2012). A Rubric for Assessing Students' Experimental Problem-Solving Ability. Journal of Chemical Education, 89(3), 319–325.

Shana, Z., & Abulibdeh, E. S. (2020). Science Practical Work and Its Impact on Students' Science Achievement. Journal of Technology and Science Education, 10(2), 199–215.

Sideridis, G., & Jaffari, F. (2023). Identifying Person Misfit Using the Person Backward Stepwise Reliability Curve (PBRC). Frontiers in Psychology, 14, 1273582.

Sigudla, J., & Maritz, J. E. (2023). Exploratory Factor Analysis of Constructs Used for Investigating Research Uptake for Public Healthcare Practice and Policy in a Resource-Limited Setting, South Africa. BMC Health Services Research, 23, 1423.

Smits, N., Cuijpers, P., & van Straten, A. (2011). Applying Computerized Adaptive Testing to the CES-D Scale: A Simulation Study. Psychiatry Research, 188, 147–155.

Soeharto, S., & Csapó, B. (2022). Exploring Indonesian Student Misconceptions in Science Concepts. Heliyon, 8, e10720.

Souza, M. A., Coster, W. J., Mancini, M. C., Dutra, F. C., Kramer, J., & Sampaio, R. F. (2017). Rasch Analysis of the Participation Scale (P-Scale): Usefulness of the P-Scale to a Rehabilitation Services Network. BMC Public Health, 17, 1–9.

Stehle, S. M., & Peters-Burton, E. E. (2019). Developing Student 21st Century Skills in Selected Exemplary Inclusive STEM High Schools. International Journal of STEM Education, 6(39).

Sudjana, N. (2017). Penilaian Hasil Proses Belajar Mengajar. Remaja Rosdakarya.

Sumintono, B., & Widhiarso, W. (2015). Aplikasi Pemodelan Rasch pada Assessment Pendidikan. Trim Komunikata.

Tan, J. P., Choo, S. S., Kang, T., & Liem, G. A. (2017). Educating for Twenty-First Century Competencies and Future-Ready Learners: Research Perspectives From Singapore. Asia Pacific Journal of Education, 37(4), 425–436.

Tasgin, A., & Dilek, C. (2023). The Mediating Role of Critical Thinking Dispositions Between Secondary School Student's Self-Efficacy and Problem-Solving Skills. Thinking Skills and Creativity, 50, 101400.

Tavakol, M., & Dennick, R. (2011). Making Sense of Cronbach's Alpha. International Journal of Medical Education, 2, 53–55.

Tesio, L., Caronni, A., Simone, A., Kumbhare, D., & Scarano, S. (2024). Interpreting Results From Rasch Analysis 2: Advanced Model Applications and the Data-Model Fit Assessment. Disability and Rehabilitation, 46(3), 04–617.

Thornhill-Miller, B., Camarda, A., Mercier, M., Burkhardt, J. M., Morisseau, T., ... & Lubart, T. (2023). Creativity, Critical Thinking, Communication, and Collaboration: Assessment, Certification, and Promotion of 21st Century Skills for the Future of Work and Education. Journal of Intelligence, 11(3), 54.

Turner, K. T., & Engelhard, G. (2022). Using Functional Clustering to Diagnose Person Misfit. The Journal of Experimental Education, 92(2), 377–397.

Underwood, S. M., Posey, L. A., Herrington, D. G., Carmel, J. H., & Cooper, M. M. (2018). Adapting Assessment Tasks to Support Three-Dimensional Learning. Journal of Chemical Education, 95, 207?217.

Watkins, M. W. (2018). Exploratory Factor Analysis: A Guide to Best Practice. Journal of Black Psychology, 44(3), 219–246.

Wegner, C., Bruneman, R. L., & Steffen, A. (2019). The Thing About Chemistry: Investigating the Fundamental Chemistry Skills of Biology Education Students at Bielefeld University. Science Education International, 30(4), 334–341.

Wright, B. D., & Stone, M. H. (1979). Best Test Design: Rasch Measurement. Mesa Press.

Wu, Q., Zhang, Z., Song, Y., Zhang, Y., Zhang, F., & Li, R. (2013). The Development of Mathematical Test Based on Item Response Theory. International Journal of Advancements in Computing Technology, 5(10), 209–216.

Yang, F. M., & Kao, S. T. (2014). Item Response Theory for Measurement Validity. Shanghai Archives of Psychiatry, 26(3), 171–177.

Yasin, Y. M., Kehyayan, V., Khraim, F., & Al-Lenjawi, B. (2023). Psychometric Evaluation of the Acute Care Nurses' Job Satisfaction Scale-Revised. Nursing Open, 10, 488–497.

Yerushalmi, E., & Eylon, B. (2015). Problem Solving in Science Learning. In R. Gunstone (Ed.), Encyclopedia of Science Education (pp. 786–790). Springer.

Ying, Y., & Tiemann, R. (2024). Development of an Assessment Tool for Collaborative Problem-Solving Skills in Chemistry. Disciplinary and Interdisciplinary Science Education Research, 6, 25.

Ziegler, M., & Hagemann, D. (2015). Testing the Unidimensionality of Items. European Journal of Psychological Assessment, 31(4), 231–237.

Zubairi, A. M., & Kassim, N. L. (2006). Classical and Rasch Analyses of Dichotomously Scored Reading Comprehension Test Items. Malaysian Journal of ELT Research, 2, 1–20.

Refbacks

  • There are currently no refbacks.


View My Stats