Making the most of research evidence and school data – an integrated approach by @DrGaryJones

If you are school-research lead then this post is for you.  In particular, if you are interested in using evidence – be it research, school data, stakeholder views and practitioner expertise – within your school, then this post will help you understand the different stages of the evidence-informed inquiry process.  To help with this,  I will be drawing upon the work of Chris Brown, (UCL Institute of Education), Kim Schildkamp and Mireille D. Hubers (both of the University of Twente) and their presentation COMBINING THE BEST OF TWO WORLDS: INTEGRATING DATA-USE WITH RESEARCH INFORMED TEACHING FOR SCHOOL IMPROVEMENT which they made at the recent International Congress for School Effectiveness and Improvement conference held earlier this year in Glasgow.

The rest of this post will: first, summarise the thesis underpinning the development of Brown et al’s model; second look in more detail at the proposed cycle of evidence-based inquiry; third, identify a number of potential limitations/design flaws within the proposed cycle; fourth, put forward an alternative model which is based upon evidence-based practice.

This is a re-blog post originally posted by Gary Jones and published with kind permission.

The original post can be found here. Read more from Gary by clicking here.

Do you have a blog post which you are proud of? Submit your blog post for reblogging on by clicking here.

The underpinning thesis 

The main ‘thesis’ underpinning Brown et al’s presentation was a call to integrate data based decision-making (DBDM) and research informed teaching practice (RITP) into a comprehensive professional learning based approach that is designed to enhance teaching quality, and a result increased student achievement.  As such, the model is attempting to use the strengths of each approach to offset the weaknesses of the other.  Brown et al summarise the offsetting benefits of each approach as follows:

  • RITP is not based on a real need in the field vs DBDM starts with the vision and goals of a specific school, focusing on a context specific problem. 
  • DBDM data can inform educators about problems in their school, but what causes this problem vs  RITP educators can draw upon a variety of effective approaches to school improvement 
  • RITP is the ‘one size does not fit all’ vs DBDM based on data, schools develop a context specific solution 
  • DBDM is that data can be used to pinpoint possible causes of a problem, but educators may still not know the best available course for school improvement vs RITP picking a promising solution, based on an existing evidence base (slide 23). 
An eight stage cycle of evidence-based inquiry
Brown et al then go onto integrate DBDM and RITP into the following eight stage cycle.

On initial reflection this model provides an incredibly useful way of thinking about the relationship between data and research evidence within the inquiry cycle, with with the local data setting the scene for the necessary research evidence.   Indeed, one of the things I have been thinking about lately is where ‘local data-collection’ fits within  the models of evidence-based practice and evidence-based medicine which I have previously described and advocated.  And this model, clearly places local data prior to seeking the research evidence.

Potential limitations

However, for me, the model two limitations to be taken into account: first,  the determination of  causes coming before the collection of data; second, the potential of for increasing the risk of confirmation bias.    So lets now examine each of these limitations in more detail.

Causes before data or vice versa

The eight stage model presented places the stage of determining possible causes of the problem before collecting data about causes,  which to me seems to be out of sequence and in reverse to order to what they should be . Indeed, there are other models of data led inquiry where data collection comes before diagnosis – and this illustrated in the four stage model of inquiry popularised by Roger Fisher and William Ury  in the their classic book Getting to Yes

In this model the data stage focuses on identifying what’s wrong, what are the current symptoms, and then moves to the identifying of possible causes in the diagnosis phase, with the next two phases being direction setting and action planning.  For me, and I know when you use these cycles you move back and forth between the stages – it seems far more sensible to place the data-collection before the diagnosis phase.   In addition, and I thank Rob Briner for this observation, the proposed process seems to separate use of the of research evidence from the diagnosis phase.  Yet, why would you not use research evidence to help you identify possible causes of the symptoms being experienced.

The potential for cognitive bias

By placing both vision and goal setting next to determining causes and the drawing of conclusions prior to the search for the solution in the research evidence, this may lead to confirmation bias i.e the tendency to selectively search or interpret information in a way that confirms your perceptions or hypotheses.  So determining the vision and goals may lead to certain problems to be highlighted as they are consistent with the already established, whereas other problems or data inconsistent with the ‘vision’ may be ignored.  Furthermore, given that conclusions are drawn before searching for research evidence – this may lead to research evidence being sought which is consistent with the conclusions that have been already been drawn.

A possible alternative

Barends, Rousseau and Briner (2014) provide an extremely useful definition of evidence-based practice, which identifies a six-stage process in making decisions which are based upon evidence.  In this model the search for and acquiring of evidence happens at stage two, with research evidence, organisational data and stakeholder views being accessed.  This leads to the subsequent appraisal and aggregation of the evidence resulting in the incorporation of the evidence into the decision-making and action planning process.

However, this process does not fully incorporate all the elements of included in the proposed eight-stage cycle of evidence-informed inquiry i.e vision and goal setting.  This ‘omission’ can be rectified if so required, by the inclusion of another ‘A’ representing : Ambition, aims and goal setting and which relates to the vision, mission and values of the school, and which gives us a 7 stage cycle of evidence-based inquiry.

A seven stage cycle of evidence-informed inquiry

Some final words

Brown et al make a powerful case for combining DBDM and RITP and have developed a potentially useful initial synthesis of the two processes.

However, the proposed model has some inherent limitations, be it diagnosis taking place before data collection and increased possibilities of confirmation bias taking place.  Finally, an alternative model – the 7 A’s of evidence-informed inquiry – is put forward as way to think about the process of linking DBDM and RITP.

About @Chilledu 2308 Articles
The Editorial Account of UKEdChat. “Mastery is an unattainable illusion”

Be the first to comment