Data and schools

In this post I talk about educational research and how it’s used in school policy. I sort of ramble about a bit, but quite liked the process of writing it whilst the exploring sources at the same time!

In recent weeks I’ve been enjoying being part of a teacher researcher group that we have setup at school. In the group we’ve talked about what we’re interested in and thought about the theme of autonomy in a bit of detail. More generally, we’ve discussed what constitutes good evidence in educational research with a common thread being that it’s important to look critically at studies that appeal to big data. This is prevalent in the educational research that informs practice in schools and arguably with good reason, as drawing from large meta-analyses can be seen as a ‘best bet’ when schools have limited means to conduct their own research and draw their own conclusions. But in this post, I’m going to first make the case that we need to be cautious about accepting large-scale quantitative studies as truths, and then put forward my suggestion for conducting focused, school-level research that schools can then use to interpret data through the lens of their own context.

Good research can be used to revolutionise the experience of young people in school.  To my mind, by ‘good research’ I mean investigating a question using rigorous and transparent methods to generate data that is used to make valid, contextually grounded conclusions. This is important because recently I feel as though I’ve been cringing a bit more regularly when I’ve heard people in schools slot the phrase “research shows” into what they’re saying in a way that suggests that their statement is fact. The issue of course is that it’s possible to find research that supports almost any statement. As an example, if I wanted to dive into supporting or opposing the use of iPads in schools then I might make the claim that ‘the research shows that using iPads improves learning’ and stick a reference such as the following in brackets down (Grigoryan, 2020), or indeed that ‘the research shows that using iPads does not improve learning’ (Boon, Boon & Bartle, 2021). Both these articles are valid, and having given them a bit of a read they’re both in my opinion pieces of work that the authors should be proud of to have out in the public domain – you can find their full references below. They both appear to make suitable conclusions and if someone wants to increase their understanding regarding iPads in schools then they’d do well to have a read – they’re good research! So, what then is the issue? If I was putting a case to my colleagues at school then surely backing up what I’ve said with research is a good thing?

Whilst writing the last paragraph I did an internet search for papers that could be deemed as in support or in opposition to iPad learning. Both papers add to the available evidence about the use of iPads in schools but neither of them are conclusive in showing enough that I could claim to have found the truth about iPad use having skimmed them. The issue therefore was in the way I drew my conclusions from the research, and what I said about the research, rather than the research itself. These two studies are very different despite having a similar field that they’re investigating. Firstly, Grigoryan (2020) is a mixed methods study so it has some quantitative data (from test scores and surveys in this example), as well as some qualitative data (from reflective journaling). Researchers might use a mixed methods approach to try to triangulate their data, or understand their data more deeply. The context is important here if I wanted to use this to support my argument about iPads in UK schools, because it turns out that this study was carried out at a higher learning institution in the UAE, and all the participants (there were 80 of them selected in the end) were women aged between 17 and 25. Furthermore, the researcher is only investigating the influence of the iPad on English language learning, and only amongst participants who are beginner learners. At this point a teacher might feel that they should turn away from this study because it’s too far removed from what they’re interested in, but we’ll come back to it in a bit.

The second article (Boon, Boon and Bartle, 2021) is a type of study called a systematic review. That means that the researchers have collected together lots of different studies that have been carried out on the topic of iPad learning (221 studies in this example), and then evaluated whether they are appropriate for answering their question, ‘does iPad use support learning in students aged 9–14 years?’. Now this sounds like a great source (and it is!)  to draw conclusions from: 221 different studies! Systematic reviews are often seen as the gold standard in informing evidence-based practice in schools. But there are big problems wrapped up in schools using systematic reviews as their source of fact and so I’m going to drill down a bit more into this particular study. Before I do that though I want to be clear that it was mid-sentence a few paragraphs back that I decided I’d use iPads as an example to work with, and I’m researching this as I write, so who knows what I’ll find. 

The evaluation process in this paper disregarded studies that didn’t fit their question so reducing the researchers down to 43 studies to draw from (p529), and this is the number that they’ve put in their abstract. Of these 43, only 21 were studies that looked exclusively at 9- to 14-year-olds (p531), so they acknowledge there’s some data creeping in that isn’t specific to their study. They’re also clear that whilst some of these are studies with lots of participants, some are very small: the smallest (Bruhn et al., 2016) actually just had two participants with quite specific needs. In that study they claim in their research that a specific app was beneficial for the two participants over the period of the research, and from reading the results, that could be the case. There’s a disclaimer at the bottom that says “it should be noted that the intervention app being examined was cocreated by the first author” (Bruhn et al., 2016, p74), which probably raises a few questions about the validity of the data here. 

Although the systematic review (Boon, Boon and Bartle, 2021) is for the Australian Educational Researcher, looking through the sources it turns out they’re from all over the world including Australia, a lot from the USA (Ditzler et al, 2016), some from India (Sankardas & Rajanahally, 2017) amongst other places. The authors rightly acknowledge that their data search is only going to let them access English language journals, and only those available on the four platforms that they’ve drawn from. About half of the chosen studies are focused on supporting students with disabilities, and although the study is about supporting learning, they’ve included a broad field from the core subjects, through to studies about student motivation, developing social skills, improving behaviour and creativity, amongst other areas.

In isolation the studies that contribute to the Boon, Boon and Bartle paper (2021) will probably constitute good research given that they'll have been peer reviewed in order to be published. Those I checked are focused in tight context and use rigourous, repeatable procedures, so just to be really clear again: there is loads of good research here, right through to the study of just two participants ((Bruhn et al., 2016). My point though is that whilst these individual studies are useful, clumping them together is less so. The findings of Boon, Boon and Bartle (2021) therefore reflect the varied nature of the studies and overall the strongest conclusion that they form is that iPads are used differently in different places, and not always effectively. Here, the sum is the parts (systematic review) is less than the whole.

So back to thinking about both of the articles that I suggested were for and against iPads for learning. I would argue that whilst the systematic review has lots of data, there’s a huge amount to be drawn from the first, seemingly abstract article. Not in terms of conclusions necessarily, but in terms of methods. If a school really wants to find out about the impact of iPads on learning then they could be drawing from methods that they see as valid and suitable for their context, and then reproducing this research themselves in their educational context. Schools can focused their research in this way, they can engage students by seeking student voice, and most importantly it is teachers who I believe are best placed to interpret data drawn from their educational landscape, rather than policy makers removed from the classroom who at best can use generalisation to forecast or backcast trends.

I believe that owning your own context should be valued by schools, and used as protection against potentially damaging voices from outside their schools that are based solely on big data from systematic reviews and meta-analyses. When schools take their own data then they can understand it in context, and they’re able to draw their own conclusions based on their professional judgement. There is, I would argue, a growing danger that because there is so much more data readily available these days then schools either have to accept it because it comes to them from policy makers, or that they take it on as that ‘best bet’ given that they believe they’re not equipped to run their own research. This is from Zeide (2017):

By relocating the site of pedagogical functions, datadriven education technologies make it more difficult for students, parents, and communities to exercise agency and demand accountability. (p168)

It is my hope that a rising tide of teacher researchers can help schools reclaim their contexts, where their working-understanding of pedagogy, and deep knowledge of their pupils can be used to add the storylines and lived experiences to their data that make it real. I’m looking forward to finding out more about teacher research, and I’m really excited about using what I now know about researching gender patterns to help schools take their own data and better understand what it is to be a pupil at their school, by using careful research alongside their wealth of experience. 

 

References
Boon, H.J., Boon, L. & Bartle, T. Does iPad use support learning in students aged 9–14 years? A systematic review. Aust. Educ. Res. 48, 525–541 (2021). https://doi.org/10.1007/s13384-020-00400-0
Bruhn, A. L., Vogelgesang, K., Fernando, J., & Lugo, W. (2016). Using Data to Individualize a Multicomponent, Technology-Based Self-Monitoring Intervention. Journal of Special Education Technology, 31(2), 64-76. https://doi.org/10.1177/0162643416650024 (Original work published 2016)
Ditzler, C., Hong, E., & Strudler, N. (2016). How tablets are utilized in the classroom. Journal of Research on Technology in Education, 48(3), 181–193. https://doi.org/10.1080/15391523.2016.1172444.
Grigoryan, T. (2020). Investigating the effectiveness of iPad based language learning in the UAE context. Open Learning: The Journal of Open, Distance and e-Learning, 37(2), 146–168. https://doi.org/10.1080/02680513.2020.1718488     
Sankardas, S. A., & Rajanahally, J. (2017). iPad: efficacy of electronic devices to help children with autism spectrum disorder to communicate in the classroom. Support for Learning, 32(2), 144–157. https://doi.org/10.1111/1467-9604.12160.
Zeide, E. (2017). The structural consequences of big data-driven education. Big Data, 5(2), 164-172.