El uso del modelo GPT de OpenAI para el análisis de textos abiertos en investigación educativa

  1. Héctor González-Mayorga
  2. Agustín Rodríguez-Esteban
  3. Javier Vidal
Revista:
Pixel-Bit: Revista de medios y educación

ISSN: 1133-8482

Ano de publicación: 2024

Número: 69

Páxinas: 227-253

Tipo: Artigo

DIALNET GOOGLE SCHOLAR lock_openAcceso aberto editor

Outras publicacións en: Pixel-Bit: Revista de medios y educación

Resumo

Assigning meaning to segments of information through analysis of open texts in qualitative research requires considerable investment of time. Natural Language Processing tools can be a valuable resource for qualitative researchers, as their algorithms allow for faster, qualitative interpretation of texts. However, this requires testing these tools’ levels of verbal comprehension beforehand. The introduction of OpenAI's GPT-3 model has marked a qualitative leap forward compared to previous Natural Language Processing models. The study objective was to analyse this tool’s verbal comprehension ability. The tests from the verbal comprehension index of the WAIS-IV IQ battery were applied. The results of the reliability tests were satisfactory. The responses put GPT-3 higher than the 99th percentile of human standards of verbal comprehension. These results demonstrate that it is possible to use this model as a tool to analyse open texts, opening up enormous possibilities for qualitative research, although its use must be based on precise, specific utilization for each analysis process.

Referencias bibliográficas

  • Aspers, P., & Corte, U. (2019). What is qualitative in qualitative research. Qualitative Sociology, 42(2), 139–160. https://doi.org/10.1007/s11133-019-9413-7
  • Bail, C. (2014). The cultural environment: Measuring culture with big data. Theory and Society, 43(3), 465–482. https://doi.org/10.1007/s11186-014-9216-5
  • Bajaj, D., Goel, A., Gupta, S.C., & Batra, H. (2022). MUCE: A multilingual use case model extractor using GPT-3. International Journal of Information Technology, 14, 1543–1554. https://doi.org/10.1007/s41870-022-00884-2
  • Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. PNAS. Proceedings of the National Academy of Sciences, 120(6), e2218523120. https://doi.org/10.1073/pnas.2218523120
  • Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In H. Larochelle, M.A. Ranzato, R. Hadsell, M.F. Balcan, & H.T. Lin (Eds.), Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://bit.ly/3ZmRPQG
  • Carriere, J., Shafi, H., Brehon, K., Pohar, M.K., Churchill, K., Ho, C., & Tavakoli, M. (2021). Case report: Utilizing AI and NLP to assist with healthcare and rehabilitation during the COVID-19 pandemic. Frontiers in Artificial Intelligence, 4, 613637. https://doi.org/10.3389/frai.2021.613637
  • Chang, T., DeJonckheere, M., Vydiswaran, V.G.V., Li, J., Buis, L.R., & Guetterman, T.C. (2021). Accelerating mixed methods research with natural language processing of big text data. Journal of Mixed Methods Research, 15(3), 398–412. https://doi.org/10.1177/15586898211021196
  • Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education. Routledge. https://doi.org/10.4324/9780203224342
  • Cope, B., & Kalantzis, M. (2015). Sources of evidence-of-learning: Learning and assessment in the era of big data. Open Review of Educational Research, 2(1), 194–217. https://doi.org/10.1080/23265507.2015.1074869
  • Cypress, B.S. (2019). Data analysis software in qualitative research. Preconceptions, expectations, and adoption. Dimensions of Critical Care Nursing, 38(4), 213-220. https://doi.org/10.1097/DCC.0000000000000363
  • Detterman, D. (2011). A challenge to Watson. Intelligence, 39(2-3), 77-78. https://doi.org/10.1016/j.intell.2011.02.006
  • Dobrev, D. (2019). The IQ of artificial intelligence. Serdica Journal of Computing, 13(1-2), 41-70. https://doi.org/10.55630/sjc.2019.13.41-70
  • Dowe, D.L., & Hernández-Orallo, J. (2012). IQ tests are not for machines, yet. Intelligence, 40(2), 77–81. https://doi.org/10.1016/j.intell.2011.12.001
  • Flick, U. (2018). Designing qualitative research. SAGE.
  • García-Peñalvo, F.J. (2023). La percepción de la inteligencia artificial en contextos educativos tras el lanzamiento de ChatGPT: Discrupción o pánico. Education in the Knowledge Society, 24, e31279. https://doi.org/10.14201/eks.31279
  • García Peñalvo, F.J., Llorens-Largo, F., & Vidal, J. (2023). La nueva realidad de la educación ante los avances de la inteligencia artificial generativa. RIED-Revista Iberoamericana de Educación a Distancia, 27(1). https://doi.org/10.5944/ried.27.1.37716
  • Golder, S.A., & Macy, M.W. (2011). Diurnal and seasonal mood vary with work, sleep, and day length across diverse cultures. Science, 333(6051), 1878–1881. https://doi.org/10.1126/science.1202775
  • Grodal, S., Anteby, M., & Holm, A.L. (2021). Achieving rigor in qualitative analysis: The role of active categorization in theory building. Academy of Management Review, 46(3), 591-612. https://doi.org/10.5465/amr.2018.0482
  • Guetterman, T.C., Chang, T., DeJonckheere, M., Basu, T., Scruggs, E., & Vydiswaran, V. (2018). Augmenting qualitative text analysis with natural language processing: Methodological study. Journal of Medical Internet Research, 20(6), e231. https://doi.org/10.2196/jmir.9702
  • Hammersley, M. (2013). What is qualitative research? Bloomsbury Academic.
  • Hernández-Orallo, J., & Minaya-Collado, N. (1998). A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In Proceedings of the international symposium of engineering of intelligent systems (EIS’98) (pp. 146–163). ICSC Press. https://bit.ly/3JdQTbK
  • Longo, L. (2020). Empowering qualitative research methods in education with artificial intelligence. In A.P. Costa, L.P. Reis, & A. Moreira (Eds.), Computer supported qualitative research. New trends on qualitative research (WCQR2019). (pp. 1-21). Springer. https://doi.org/10.1007/978-3-030-31787-4
  • Mills, K. A. (2019). Big data for qualitative research. Routledge Focus. https://doi.org/10.4324/9780429056413
  • Oussalah, M. (2022). AI explainability. A bridge between machine vision and natural language processing. In A. Del Bimbo, R. Cucchiara, S. Sclaroff, G.M. Farinella, T. Mei, M. Bertini, H.J. Escalante, & R. Vezzani (Eds.), Pattern recognition. ICPR International workshops and challenges (pp. 257-273). Springer. https://doi.org/10.1007/978-3-030-68790-8
  • Rezaeenour, J., Ahmadi, M., Jelodar, H., & Shahrooei, R. (2022). Systematic review of content analysis algorithms based on deep neural networks. Multimedia Tools & Applications. https://doi.org/10.1007/s11042-022-14043-z
  • Shlomo, N., & Goldstein, H. (2015). Big data in social research. Journal of the Royal Statistical Society, 178(4), 787–790. https://doi.org/10.1111/rssa.12144
  • Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., & Wang, L. (2022). Prompting GPT-3 to be reliable. arXiv. https://doi.org/10.48550/arXiv.2210.09150
  • Silver, C., & Lewins, A. (2014). Using software in qualitative research: A step-by-step guide. SAGE.
  • Tack, A. & Piech, C. (2022). The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. arXiv. https://doi.org/10.48550/arXiv.2205.07540
  • Towler, L., Bondaronek, P., Papakonstantinou, T., Amlôt, R., Chadborn, T., Ainsworth, B., & Yardley, L. (2022). Applying machine-learning to rapidly analyse large qualitative text datasets to inform the COVID-19 pandemic response: Comparing human and machine-assisted topic analysis techniques. MedRxiv. https://doi.org/10.1101/2022.05.12.22274993
  • Turing, A.M.I. (1950). Computing machinery and intelligence. Mind, 236(LIX), 433–460. https://doi.org/10.1093/mind/LIX.236.433
  • Wang, H., Tian, F., Gao, B., Bian, J., & Liu, T.Y. (2016). Solving verbal comprehension questions in IQ test by knowledge-powered word embedding. arXiv. https://doi.org/10.48550/arXiv.1505.07909
  • Wechsler, D. (2012a). WAIS-IV. Escala de inteligencia de Wechsler para adultos-IV. Manual de aplicación y corrección. Pearson.
  • Wechsler, D. (2012b). WAIS-IV. Escala de inteligencia de Wechsler para adultos-IV. Manual técnico y de interpretación. Pearson.
  • Williams, E.N., & Morrow, S.L. (2009). Achieving trustworthiness in qualitative research: A pan-paradigmatic perspective. Psychotherapy Research, 19(4-5), 576-582. https://doi.org/10.1080/10503300802702113
  • Williams, M., & Moser, T. (2019). The art of coding and thematic exploration in qualitative research. International Management Review, 15(1), 45-55. https://bit.ly/3mt4BPh
  • Xie, Y., Vosoughi, S., & Hassanpour, S. (2022). Interpretation quality score for measuring the quality of interpretability methods. arXiv. https://doi.org/10.48550/arXiv.2205.12254
Fonte de datos: Dialnet