Tag Archives: Education

Brain Training: Placebo effects, publication bias, small sample sizes… and what we do next?

Over the past decade the young field of cognitive training – sometimes referred to as ‘brain training’ – has expanded rapidly. In our lab we have been extremely interested in brain training (Astle et al. 2015; Barnes et al. 2016). It has the potential to tell us a lot about the brain and how it can dynamically respond to changes in our experience.

The basic approach is to give someone lots of practice on a set of cognitive exercises (e.g. memory games), see whether they get better at other things too, and in some cases see whether there are significant brain changes following the training. The appeal is obvious: the potential to slow age-related cognitive decline (e.g. Anguera et al. 2013), remediate cognitive deficits following brain injury (e.g. Westerberg et al. 2007), boost learning (e.g. Nevo and Breznitz 2014) and reduce symptoms associated with neurodevelopmental disorders (e.g. Klingberg et al. 2005). But these strong claims require compelling evidence and the findings in this area have been notoriously inconsistent.

menuscreens

(Commercial brain training programmes are available to both academics and the general public)

I have been working on a review paper for a special issue, and having trawled through the various papers, I think that some consensus is emerging. Higher-order cognitive processes like attention and memory can be trained. These gains will transfer to similarly structured but untrained tasks, and are mirrored by enhanced activity and connectivity within the brain systems responsible for these cognitive functions. However, the scope of these gains is currently very narrow. To give an extreme example, learning to remember very long lists of letters does not necessarily transfer to learning long lists of words, even though those two tasks are so similar – the training can be very content specific (Harrison et al. (2013); see also Ericcson et al. (1980)). But other studies seem to buck that trend, and show substantial wide transfer effects – i.e. people get better not just at what they trained on, but even very different tasks. Why this inconsistency? Well I think there are a few important differences in how the studies are designed, here are two of the most important:

  1. Control groups: Some studies don’t have control groups at all, and many that do don’t have active control groups (i.e. the controls don’t actually do anything, so it is pretty obvious that they are controls). This means that these studies can’t properly control for the placebo effect (https://en.wikipedia.org/wiki/Placebo). If a study doesn’t have an active control group then it is more likely to show a wide transfer effect.
  2. Sample size: The smaller the study (i.e. the fewer the participants) the more likely it is to show wider transfer effects. If studies include lots of participants then it is far more likely to accurately estimate the true size of the transfer effect, which is very small.

When you consider these two factors and only look at the best designed studies, the effect size for wider transfer effects is about d=0.25 – if you are not familiar with this statistic, this is small (Melby-Lervag et al., in press). Furthermore, when considering the effect sizes in this field it is important to remember that this literature almost certainly suffers from a publication bias – it is difficult to publish null effects, and easier to publish positive results. Meaning that there are probably quite a few studies showing no training effects sat in researchers’ drawers, unpublished. As a result, even this small effect size is likely an overestimate of the genuine underlying effect. The true effect is probably even closer to zero.

So claims that training on some cognitive games can produce improvements that spread to symptoms associated with particular disorders – like ADHD – are particularly incredible. Just looking at the best designed studies, the effect size is small, again about d=0.25 (Sonuga-Barke et al., 2013). The publication bias caveat applies here too – even this small effect size is likely an overestimate of the true effect. Some studies do show substantially larger effects, but these are usually not double blind. That is, the person rating those symptoms knows whether or not the individual (usually a child) received the training. This will result in a substantial placebo effect, and this likely explains these supposed enhanced benefits.

Where do we go from here? As a field we need to ensure that future studies have active control groups, double blinding and that we include enough participants to show the effects we are looking for. I think we also need theory. A typical approach is to deliver a training programme, alongside a long list of assessments, and then explore which assessments show transfer. There is little work that explicitly generates and then tests a theory, but I think this is necessary for future progress. Where research is theoretically grounded it is far easier for a field to make meaningful progress, because it gives a collective focus, creates a shared set of critical questions, and provides a framework that can be tested, falsified and revised.

Author information:

Dr. Duncan Astle, Medical Research Council Cognition and Brain Science Unit, Cambridge.

https://www.mrc-cbu.cam.ac.uk/people/duncan.astle/

Reference:

Anguera JA, Boccanfuso J, Rintoul JL, Al-Hashimi O, Faraji F, Janowich J, Kong E, Larraburo Y, Rolle C, Johnston E, Gazzaley A (2013) Video game training enhances cognitive control in older adults. Nature 501:97-101.

Astle DE, Barnes JJ, Baker K, Colclough GL, Woolrich MW (2015) Cognitive training enhances intrinsic brain connectivity in childhood. J Neurosci 35:6277-6283.

Barnes JJ, Nobre AC, Woolrich MW, Baker K, Astle DE (2016) Training Working Memory in Childhood Enhances Coupling between Frontoparietal Control Network and Task-Related Regions. J Neurosci 36:9001-9011.

Ericcson KA, Chase WG, Faloon S (1980) Acquisition of a memory skill. Science 208:1181-1182.

Harrison TL, Shipstead Z, Hicks KL, Hambrick DZ, Redick TS, Engle RW (2013) Working memory training may increase working memory capacity but not fluid intelligence. Psychological science 24:2409-2419.

Klingberg T, Fernell E, Olesen PJ, Johnson M, Gustafsson P, Dahlstrom K, Gillberg CG, Forssberg H, Westerberg H (2005) Computerized training of working memory in children with ADHD–a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry 44:177-186.

Melby-Lervag M, Redick TS, Hulme C (in press) Working memory training does not improve performance on measures of intelligence or other measures of “Far Transfer”: Evidence from a meta-analytic review. Perspectives on Psychological Science.

Nevo E, Breznitz Z (2014) Effects of working memory and reading acceleration training on improving working memory abilities and reading skills among third graders. Child neuropsychology : a journal on normal and abnormal development in childhood and adolescence 20:752-765.

Sonuga-Barke EJ, Brandeis D, Cortese S, Daley D, Ferrin M, Holtmann M, Stevenson J, Danckaerts M, van der Oord S, Dopfner M, Dittmann RW, Simonoff E, Zuddas A, Banaschewski T, Buitelaar J, Coghill D, Hollis C, Konofal E, Lecendreux M, Wong IC, Sergeant J (2013) Nonpharmacological interventions for ADHD: systematic review and meta-analyses of randomized controlled trials of dietary and psychological treatments. The American journal of psychiatry 170:275-289.

Westerberg H, Jacobaeus H, Hirvikoski T, Clevberger P, Ostensson ML, Bartfai A, Klingberg T (2007) Computerized working memory training after stroke–a pilot study. Brain injury 21:21-29.

1.4 percent: Behind the numbers of ADHD prevalence

A question closely tied to the issue of prescribing ADHD medication is the actual growth of the disorder. Media reports have cited substantial increases in Ritalin use in the UK, and from looking at prescribing trends in the US one possibility is that the diagnosis of ADHD increases in parallel with growing Ritalin prescriptions. It is particularly important to understand how the increased prescribing of ADHD medication in the UK relates to the actual prevalence of the disorder. However, a key issue in the ADHD debate is whether the diagnosis represents a core developmental disorder. The growth in Ritalin prescriptions and the disproportionately high rates of ADHD in the US in particular have often been used to suggest that ADHD diagnosis is socially motivated.

When we look more closely at the numbers around ADHD, however, three important observations emerge.  First, we can see that although there has been a consistent growth in ADHD medication prescriptions in the UK, there is no evidence speaking to whether this also corresponds to an increase in ADHD diagnoses. Second, the common perception of the US being disproportionately affected by ADHD is surprisingly unsupported. Rather, we can see that different methods and approaches to ADHD diagnosis are what best explain its relative prevalence.  Third, thinking about ADHD as a label for a highly heterogeneous group can perhaps more readily account for the differential rates of ADHD based on the diagnostic criteria used.

Has the prevalence of ADHD in the UK increased in line with Ritalin prescriptions?

As we saw in our previous blog post, the number of Ritalin prescriptions dispensed in England increased by 156% from 2004 to 2014. However, this does not mean that the prevalence of ADHD has increased by a similar proportion. Why? Factors that can influence the number of prescriptions for ADHD medication written include GPs being more likely to prescribe such medication due to more positive peer-reviewed articles about its effects, changes in prescription protocols (such as more frequent repeat prescriptions being provided), and more awareness amongst parents and the public about the possible benefit of ADHD medication meaning a higher proportion of people ask for medication1. Further, although NICE guidelines published in 2013 recommended treatment for ADHD with medication as part of a comprehensive treatment programme (including psychological, behavioural and educational support), reductions in funding available for such support may contribute to a higher prescription of medication by GPs as an alternative. All of these factors can lead to a higher number of prescriptions being written, and this means the number of prescriptions does not necessarily align with the number of individuals diagnosed with ADHD.

So we need to take a different approach to estimating the prevalence of ADHD diagnoses, by looking at the proportion of actual ADHD diagnoses in the population. However, there are currently no available public health records about ADHD prevalence in the UK. Instead, one useful source of data come from population-based studies, which take a large sample of children (often around 8000-14,000) and estimate the prevalence of ADHD through the percentage of children with parent reports of symptoms and/or ADHD diagnoses. These studies have the benefit of estimating the prevalence of ADHD through a non-referred sample, avoiding the biases arising from estimates reached by only counting the number of children brought to medical attention.

One such study conducted by Green and colleagues in 20042 estimated the UK prevalence of ADHD in 5-15 year-olds at approximately 1.5%. (A similar report from 2003 estimated the prevalence in this age group at 2.2%3). A more recent report by Russell et al. (2014)4 based on the Millenium Cohort Study estimated ADHD prevalence in the UK at 1.4%. This estimate came from data collected in 2008-2009 when children were 6-8 years old, and parents reported whether ADHD had been identified by a doctor or other health professional.

At face value, these figures are a striking contrast to the prescription data: the parent-report data suggest no increase in ADHD diagnoses (from 2004-2009, at least), in contrast to the yearly increase in Ritalin prescriptions over the same time period. However, it is important to consider that the more recent Millennium Cohort Study figures may slightly underestimate the prevalence of children affected by ADHD: this is because the Millennium Cohort sample children were younger, and more children are likely to be diagnosed when they are older.

So whilst it is possible that ADHD diagnoses in the UK have increased, we do not have the data to accurately evaluate this change. As we saw in our previous blog post, data from the US would suggest that growing use of ADHD medication does align with increasing diagnoses, but it is difficult to evaluate this relationship for the UK because we do not have statistics from comparable samples of children at different time points. It is particularly important to verify if the actual number of individuals diagnosed with ADHD has increased to understand whether such a growth in prescriptions stems from more diagnoses, a growing trend to provide medication in lieu of other treatment options, or a combination of the two.

But the current wave of Millennium Cohort Study data were recently collected when children were 11 years old, and using these data researchers can assess if more children in the sample have now been diagnosed with ADHD as they have gotten older – so we might have an answer soon.

Is the United States disproportionately affected by cases of ADHD, and does this tell us anything about causality?

The growth of ADHD medication use and the concurrent prevalence of the disorder is closely related to the debate around whether it reflects an increase in a genuine developmental disorder, or whether such growth is part of a cultural trend towards diagnosis. The geographical vagaries of ADHD in the US particularly have often been invoked as an argument against its legitimacy as a disorder, with some suggestion that the US peak in ADHD rates indicates its diagnosis can be culturally motivated. Sir Ken Robinson, an author, speaker and education adviser, gave a popular TED talk in 2010 in which he showed a map of ADHD prescription rates across the USA, indicating that they increased as one moved east across the country – or as put by Sir Ken, “people start losing interest in Oklahoma, they can hardly think straight in Arkansas, and by the time they get to Washington they’ve lost it completely.”

This brings us to our next question: is it really the case that ADHD is over diagnosed in the US relative to the rest of the world, as media discussion often suggests? The geographical distribution of ADHD is important for two reasons. First, consistent differences in the geographical spread of ADHD may offer clues as to its aetiology: possible environmental causation in areas of high rates, and preventative factors in areas of low rates. Second, the worldwide distribution of ADHD informs the debate on whether it stems from cultural factors which make it an ‘American’ disorder – whatever this may mean – in which case we would expect it to be more prevalent in the US than anywhere else.

When we look at just the US and UK, it appears that US does have higher diagnosis rate of 6.3% (as reported by Russell et al., 2014). This is the closest estimate using the same sampling methods as the UK figure of 1.4% from the Millennium Cohort Study. However, as pointed out in the Millennium Cohort report by Russell et al., there is an explanation for this: a difference in the measurement of ADHD between the US and UK. When the data were collected the DSM-IV was used in the US, but researchers in the UK from whom the data were collected mainly used the ICD-10. The ICD-10 is more stringent, in which a child must show symptoms in all three dimensions of inattention, hyperactivity, and impulsivity to be diagnosed with ADHD, with no comorbid disorders. The DSM-IV is less stringent in the classification of ADHD, whereby children can be diagnosed with symptoms in one dimension only (such as inattention) alongside comorbid disorders. This difference in relative preference of diagnostic criteria can at least partly explain why the DSM-IV-using US has an epidemic compared to the relative calm of the ICD-10-preferring UK5.

When we look at the geographical spread of ADHD across the rest of the world, the figures are particularly striking. A meta-analysis by Polanczyk et al. in 20076 combined 102 studies, comprising data from 171,756 children, to estimate the average prevalence rates reported from samples worldwide in the previous 25 years. This figure was 5.3%. Second, they evaluated whether the US prevalence rates really did exceed everywhere else in the world, relative to Africa, Asia, Europe (excluding the UK), the Middle East, Oceania, and South America. The US rate of 6.2% did not significantly differ from the European rate (4.6%), or the highest rate in South America (11.8%). And importantly, there was no overall association between geographic location and ADHD prevalence rates when the methodology of the studies was controlled for. The figure below shows the prevalence estimates for each region (the bars represent the variability in this estimate).Polanczyk et al. (2007) Worldwide Figure

Finally, Polanczyk and colleagues verified that the different ADHD prevalence estimates between the US and Europe in particular were indeed explained by the different diagnostic criteria of the DSM and ICD. They did this by coding for all the methodological features in each study, and found that US diagnoses were more frequently made on DSM criteria, and European diagnoses more so on ICD-10 criteria. Another meta-analysis by Willcutt (2012)7 also found that differences in ADHD prevalence across countries were not significant after diagnostic approaches were controlled for. So these data suggest that ADHD is not disproportionately prevalent in the US; rather, its relative prevalence seems to depend more on the diagnostic tool used.

But on an individual level, any parent of a child with ADHD, or any individual with ADHD, is unlikely to attest that their difficulties with learning, concentration and communication are because they are living in Bolivia rather than Finland. The critical ongoing issue is therefore what constitutes our diagnosis of ADHD. In practice, a child is likely to be identified with ADHD because of how their various cognitive difficulties are expressed, and how this expression is interpreted by clinicians. Part of this interpretation may be due to less quantifiable environmental, social and cultural factors: in particular, Conrad and Bergey (2014)8 recognise that the historical approach to ADHD and medication use in a country can greatly influence whether clinicians classify children’s difficulties under the ADHD diagnostic label. But the different diagnostic criteria of the DSM and ICD do play a role. As noted by Russell et al. (2014), children with hyperactive behaviour and social difficulties may be more likely to be diagnosed with autism spectrum disorder in the UK (due to ICD-10 criteria ruling out a combined diagnosis), and ADHD in the US (due to the DSM-IV allowing ADHD to be diagnosed alongside other developmental disorders). It therefore seems that ADHD is not disproportionately prevalent in the US when we take diagnosis methods into account, but that fact that different tools can yield different prevalence rates makes it important to verify what the ADHD diagnostic label is mapping onto.

A dimensional approach to ADHD

At first glance, quantifying the prevalence of ADHD would suggest counting all children and adults struggling with attention, hyperactivity, and impulsivity and suggesting those individuals reflect the relative rate of ADHD in a particular population. In practise, however, there is no natural threshold between affected and unaffected individuals with ADHD. ADHD can frequently co-occur with other developmental disorders, such as autism spectrum disorder (ASD); in the Millennium Cohort dataset, 24% of the children with ADHD were also diagnosed with ASD. This individual variation supports a dimensional approach to classifying developmental disorders, which considers the extent to which various cognitive difficulties are present rather than a clear-cut diagnostic label.

This is critical in deciding how we give children help in the classroom. For example, we have seen previously that children with poor working memory and children with ADHD have highly similar working memory and executive function deficits, indicating that these shared difficulties are not limited to children with ADHD. Further, although an estimated 25-40% of children with ADHD also have dyslexia9, the treatment approaches for ADHD do not usually address reading difficulties, even though these can be a main constraint on a child’s progress at school. The diagnostic label of ADHD thus represents a highly heterogeneous group of children, and what best helps one child with ADHD may not be what best helps another.

But one might argue that neuroimaging data suggesting reduced prefrontal cortex activity in ADHD is indicative of a clear group of children who suffer from a specific neurobiological disorder. However, even such neurobiological ‘markers’ of ADHD are present to a varying degree in individuals without ADHD. Shaw et al. (2007)10 reported that the age of reaching peak cortical thickness – a measure of brain maturation, before cortical thinning proceeds throughout late childhood and adolescence – was delayed in children with ADHD relative to typically developing children. This finding suggested that ADHD may reflect a delay in typical brain development, rather than a qualitative difference between children with and without an ADHD diagnosis. And Shaw et al. (2011)11 found that this rate of cortical thinning was related to the degree of hyperactive and impulsive symptoms in typically developing children without an ADHD diagnosis. Children with higher ratings of hyperactive/impulsive symptoms had a slower rate of cortical thinning, similar to what was observed in children with ADHD. These data align with the idea that ADHD may be a disorder that is better understood dimensionally, with those who are affected being at the end of a continuum of inattention and/or hyperactivity symptoms, rather than there being a distinct cut-off between ADHD and ‘typical’ development.

How does this inform the question on whether ADHD exists as a developmental disorder? From the evidence above it seems that the label of ADHD does not identify a separate category of children who are inattentive and hyperactive. Rather, what we think of as ADHD is a complex cluster of cognitive and behavioural difficulties which fall on a scale for all children, from unaffected to severe. It is how these difficulties are expressed by children and interpreted by different diagnostic approaches, in different cultural contexts, which determine the cut-off at which a child is diagnosed. Whilst difficult to unpick, these factors may best capture how ADHD is identified and why prescriptions have steadily grown; however, we critically need further data to accurately evaluate ADHD growth in the UK. And on an individual level, the symptoms of ADHD have a substantial impact on the day-to-day lives of those who are affected. It is important we ensure that diagnosis appropriately reflects individual differences between children, and think about how to best to commission education and health services to help affected children and adults.

References

  1. Prescribing in primary care: Understanding what shapes GPs’ prescribing choices and how might these be changed, RAND Technical Report. Available at http://www.nao.org.uk/wp-content/uploads/2007/05/TR443_3C.pdf
  2. Green, H., McGinnity, A., Meltzer, H., & Ford, T. G., R. (2004). Mental health of children and young people in Great Britain, 2004. Report commissioned by the Office for National Statistics.
  3. Reported in the NICE Guidelines pp. 26, available here: https://www.nice.org.uk/guidance/cg72/resources/cg72-attention-deficit-hyperactivity-disorder-adhd-full-guideline-2
  4. Russell, G., Rodgers, L. R., Ukoumunne, O. C., & Ford, T. (2014). Prevalence of parent-reported ASD and ADHD in the UK: findings from the Millennium Cohort Study. J Autism Dev Disord, 44(1), 31-40. doi: 10.1007/s10803-013-1849-0
  5. For interested readers, there is also an excellent timeline here depicting ADHD rates in the US as diagnostic tools have changed from 1970 to the present day.
  6. Polanczyk, G., de Lima, M. S., Horta, B. L., & Biederman. J. (2007). The Worldwide Prevalance of ADHD: A Systematic Review and Metaregression Analysis. American Journal of Psychiatry, 164(6). 942-948.
  7. Willcutt, E. G. (2012). The prevalence of DSM-IV attention-deficit/hyperactivity disorder: a meta-analytic review. Neurotherapeutics, 9(3), 490-499. doi: 10.1007/s13311-012-0135-8
  8. Conrad, P., & Bergey, M. R. (2014). The impending globalization of ADHD: notes on the expansion and growth of a medicalized disorder. Soc Sci Med, 122, 31-43. doi: 10.1016/j.socscimed.2014.10.019
  9. Reported in Eden, G. F., & Vaidya, C. J. (2008). ADHD and developmental dyslexia: two pathways leading to impaired learning. Ann N Y Acad Sci, 1145, 316-327. doi: 10.1196/annals.1416.022
  10. Shaw, P., Eckstrand, K., Sharp, W., Blumenthal, J., Lerch, J. P., Greenstein, D., . . . Rapoport, J. L. (2007). Attention-deficit/hyperactivity disorder is characterized by a delay in cortical maturation. Proc Natl Acad Sci U S A, 104(49), 19649-19654. doi: 10.1073/pnas.0707741104
  11. Shaw, P., Gilliam, M., Liverpool, M., Weddle, C., Malek, M., Sharp, W., . . . Giedd, J. (2011). Cortical development in typically developing children with symptoms of hyperactivity and impulsivity: support for a dimensional view of attention deficit hyperactivity disorder. Am J Psychiatry, 168(2), 143-151. doi: 10.1176/appi.ajp.2010.10030385

Do children with stronger functional connectivity at rest have better working memory capacity?

You are driving to the house of a friend, and you are getting directions from a GPS. “Drive to the end of the road, turn right, drive straight ahead and turn down the third road on your left, and the house is at the end of the road.”  As you are driving you need to remember each instruction, and hold that information in mind whilst completing each step. This is an example of a task that requires working memory. Working memory is the ability to hold relevant information in mind for a short period of time, whilst processing or using other material to complete a task. Working memory is critically important for learning at school, and failures of working memory can be a key predictor of children showing poor academic performance.

One way in which we can start to think about why some children seem to do far better on working memory tasks than others is by exploring the underlying neurophysiological processes that support working memory. With a technique called magnetoencephalography (MEG) we are able to look at how and when different brain areas are communicating, by seeing whether activity between different brain regions co-occurs in time. Importantly, it is possible to observe the strength of communication – functional connectivity – between different brain areas at rest.

In a new paper, we found that a child’s spatial working memory capacity can be predicted on the basis of particular patterns of brain activity at rest. We measured the resting-state functional connectivity of 8 to 11-year-olds, and assessed whether the strength of functional connectivity could be predicted by the child’s capacity. To measure working memory, we used well-validated behavioural measures of spatial and verbal working memory, taken outside the MEG scanner. These measures are routinely used in educational settings and are closely related to children’s literacy and numeracy levels. To examine functional connectivity, we measured the extent to which the activity of different brain regions was coordinated whilst the children were at rest inside the scanner. We then selected specific networks of coordinated regions that we considered ought to relate to working memory, based on adult networks responsible for cognitive control, because this is thought to be a key component of working memory. We reasoned that better these control areas can become coordinated with other brain systems, the better the child ought to do at control-demanding working memory exercises.

And this was what we found. We identified that the strength of connectivity between areas of the frontal and parietal lobe, and a portion of the temporal lobe,  was related to children’s spatial working memory capacity. The stronger the coordination between these areas, the higher the child’s capacity. Importantly, this relationship could not be explained simply by differences in motivation or strategy during the spatial working memory tasks, as children were at rest when the MEG was measured. Interestingly, we did not find a relationship between verbal working memory scores and any functional networks at rest – this may have been because the brain networks involved in verbal working memory tasks are less discrete or easily detectible with MEG.

Why is this important? Working memory is a key predictor of educational outcomes, and being able to relate individual differences in working memory to the functional coordination of different brain regions helps us understand the mechanisms supporting successful (and less successful) performance. The specific network related to children’s spatial working memory was the bilateral superior parietal cortex and middle frontal gyri, areas often associated with spatial attentional control, and lower-level processing areas. This suggests that children with better spatial working memory may be those with stronger connections between these frontal control regions and lower-level sensory areas. We can use this knowledge to begin to ask questions such as whether training can strengthen this connectivity (yes, as we have seen in a previous blog post), and to identify and redress the ways in which it can impact on learning.

References

Barnes, J. K., Woolrich, M. W., Baker, K., Colclough, G. L., & Astle, D. E. (2015). Electrophysiological measures of resting state functional connectivity and their relationship with working memory capacity in childhood. Developmental Science, doi: 10.1111/desc.12297.

Does working memory training change neurophysiology in childhood?

The short answer to that question is ‘yes’.

We have known for some time that training particular cognitive skills, like working memory, can produce improvements in cognition. These improvements transfer to other untrained tasks, provided that they are similarly structured. However, we know very little about how these kinds of intensive cognitive training programmes change children’s performance.

The study has been under embargo whilst it awaits publication in the Journal of Neuroscience, but the embargo has just been lifted, and we can now tell you all about it (the paper itself should be published very soon – open access, naturally). We used magnetoencephalography – a technique for measuring electrical brain activity – to explore patterns of brain activity as children rested in the scanner with their eyes closed. We repeated this procedure before and after the children underwent working memory training. Importantly, only half of the children underwent an intensive version of the training, with the other half doing a low intensity version. This latter group acted as a control, and the children were randomly allocated to the high or low intensity conditions.

After the training, children’s working memory performance improved. We used standardised assessments of working memory to show this. Importantly, these improvements were specific to the group of children who had undertaken the intensive training programme, with the control group showing little or no improvement. This pattern could not have resulted from us expecting that the children in the intensive group ought to show bigger improvements than the controls, because the researcher doing the assessments did not know which group the children were assigned to.

The magnetoencephalography data allowed us to explore whether there were any significant changes in children’s brains, and whether these changes mirrored the improvements in performance that we observed. We used the spontaneous electrical activity in the children’s brains to explore network connectivity – that is, how different brain areas are coordinated. After the training, connectivity within networks involved in attentional control were significantly enhanced. Furthermore, the bigger the change in connectivity, the bigger the improvement in the child’s working memory.

We have a lot more data on these children, which we are slowly crunching our way through. So there will be more to come!

ADHD and Low Working Memory: A comparison

I was recently directed to a paper titled ‘Children with low working memory and children with ADHD: same or different?’ From the title alone this paper piqued my curiosity. As scientists we often focus on collecting large amounts of data, and in some cases forget the importance of forming new theories and ideas. For me, this paper casts the profiles of an ADHD diagnosis and a deficit in working memory, a common childhood issue, in a new light. What does it mean (if anything) to have a clinical diagnosis? And what is the best way of characterising children with problems of attention and memory?

Whilst I will summarise the main findings below, for those that want to read the paper for themselves it can be accessed here:

http://journal.frontiersin.org/Journal/10.3389/fnhum.2014.00976/full

This paper provides a novel comparison of cognitive skills, executive function, educational attainment and behaviour between those with a low WM and those with a diagnosis of ADHD. It utilised three groups of children (8-11yrs): one group identified by routine screening as having a low WM, one group with diagnoses of ADHD and one group of typically developing children.

Age was controlled for across the groups and for each group the following measures were taken: Working memory (Automated working memory assessment); Executive function (Delis-Kaplan Executive Function System, The K-test of the Continuous Performance Test and Walk/Don’t Walk from the Test of Everyday Attention for Children); Academic ability (Wechsler reading, spelling, reading comprehension, mathematical reasoning and number operations and the four WASI subtests); And classroom behaviour problems (Connors teacher rating and the Behaviour rating inventory of executive function). Those children taking medication for their ADHD stopped taking it at least 24hrs prior to the session to ensure the effects of the drugs were eliminated by the time of testing.

The theory that makes up the basis for the research, includes the fact that both ADHD and low WM in childhood are significantly associated with poor educational attainment. It has also been established in the literature that there are associations between poor WM and inattentive behaviour and poor WM and executive function deficits. These elements combined suggest a high degree of overlap between the profiles of those with a deficit in WM and those with an ADHD diagnosis. In ADHD, we see executive function problems (deficits in response inhibition, attentional switching, planning and sustained attention) and also excessively high levels of motor activity and impulsive behaviour.

Some have argued that the executive function deficits in ADHD are a result of an underlying low WM, whilst others have proposed that ADHD involves two distinct neurodevelopmental systems:

1) a ‘cool/cognitive system’ that affects executive functions and WM

and

2) a ‘hot/affective system’ that leads to ‘aversion to delay that manifests as impulsive behaviour’.

It is this impulsive behaviour that manifests as excessive motor activity and problems in impulse control that are core characteristics of ADHD and are not associated with low WM. Thus, this study hypothesised that the two groups share a common deficit in the ‘cool executive function system’ reflected in the fact that both profiles exhibit lack of attentional control, but only those with the ADHD diagnosis have impairments in the ‘hot executive function system’ that results in hyperactivity and impulsivity.

The ADHD and low WM group, as expected, performed more poorly than the typically developing children on the tests of working memory (necessarily so for the latter group, because they were selected on this basis). Performance was significantly decreased for the low WM group in the two WM subtests that they were initially screened on. This suggests a selection artefact as ‘the two groups did not differ on the verbal WM task that was not used at screening’. In the executive function switching task no significant difference was seen between the low WM and ADHD group when considering accuracy, however, the low WM group showed a significantly slower performance than the ADHD and typical group. The sustained attention executive function task showed that both the ADHD and low WM children were significantly less accurate than the typically developing group and that they both made a significantly greater number of omissions than the typically developing group. The ADHD group also further made a significantly greater number of commission errors than the other two groups. No other significant differences were seen between the low WM group and the ADHD group on any other test of executive function. It is also worth noting that although there are a number of different models proposed for the functional structure of WM, the findings of this paper remain consistent across models on the basis that the ‘storage-only capacities of short-term memory (STM) and the capacity-limited attentional control functions of WM can be distinguished’.

They concluded that the ADHD and low WM groups have very similar profiles of WM ability and executive function impairment. They did however find two important differences: the low WM group were ‘slower to respond on several tasks’ and the ADHD group were ‘more hyperactive and exhibited more difficulties in controlling impulsivity in sustained attention’.  The processing speed impairments in the low WM group were an unexpected finding. They state that ‘it does not appear to be a part of a broader problem in fluid intelligence, as controlling statistically for performance IQ had little impact’. They present the idea that the impairment in response time may be a result of sluggish cognitive tempo (SCT). Sluggish cognitive tempo is a ‘set of symptoms strongly associated with the predominantly inattentive form of ADHD that includes high levels of daydreaming, slow response times, poor mental alertness and hypoactivity.’ It could perhaps be the case that those with a low WM may correspond to those with SCT, a predominantly inattentive subtype of ADHD. At present SCT is not a commonly utilised diagnosis in the UK. The more hyperactive behaviour in the ADHD children however was expected. ‘They violated rules more frequently during a planning task, and made more commission errors on the Continuous Performance Test of sustained attention’. This elevated hyperactivity was also shown in the Connor’s teacher rating as increased impulsivity and oppositional behaviour.

Interestingly they found the two groups to have equivalently high levels of inattentive behaviour. Whilst it has been shown before that children with low WM often exhibit inattentiveness, this paper is the first to show that this inattentive behaviour is of a very similar degree to that of  children with ADHD. In terms of the deficits in working memory, both groups exhibited deficits in the same regions and of a similar magnitude. They had impairment in ‘visuospatial short term memory, verbal working memory and visuo-spatial working memory’; but ‘their verbal short term memory scores fell within the typical age range’. The novel finding here is that there is such a high degree of correspondence between the profile and severity of the WM impairments in the two groups.

Previous research has also suggested a close relationship between ‘controlled cognitive attention and working memory’. This paper takes this understanding one step further suggesting that it is now looking more likely that there is a ‘link between poor working memory and overt inattentive and distractible behaviours’ as seen through the inattentiveness in the behaviour of those children with a low WM. In addition to being comparable in inattentiveness and working memory impairment, the two groups both exhibited ‘high rates of problem behaviours across a wide range of executive function behaviours’. Previous studies have also indicated that problem behaviours associated with executive function are seen in children with a low WM. This paper supports and extends this idea by showing that both groups ‘performed poorly on direct measures of switching, inhibition, sorting, planning, sustained attention and response suppression’.

When considering again, the cool (cognitively based) and the hot (affective) model of executive function, the pattern found in the results from this study suggest that there is some shared general executive deficit between the ADHD and low WM groups. The cool deficits manifest as inattention and low WM in both groups whilst the hot deficits, only present in the ADHD group, manifests as the hyperactivity. The additional delay in response times in the low WM group is the only component that does not fit with this theory, but the authors have identified that it ‘may be symptomatic of a subgroup of children with the predominantly inattentive form of ADHD who are characterised by SCT’.

In conclusion, this paper shows some very interesting preliminary findings into two conditions that are generally considered distinct. Both groups showed equal levels of underachievement when IQ differences were taken into account (‘they are indistinguishable in terms of their poor learning progress in mathematics and reading’). This is of great significance as those with a diagnosis of ADHD will almost certainly have learning support provided to them whilst those with a low WM will not. The educational needs of the group of children with a low WM are being overlooked as they do not show the disruptive hyperactivity of ADHD, even though in many respects the groups are indistinguishable.

Holmes J, Hilton KA, Place M, Alloway TP, Elliott JG and Gathercole SE (2014) Children with low working memory and children with ADHD: same or different? Front. Hum. Neurosci. 8:976. doi: 10.3389/fnhum.2014.00976

Tools of the Mind: An Effective Intervention for High Poverty Schools?

Early childhood interventions are believed to be a key step towards ending the cycle of poverty. This belief is based upon the large evidence base that demonstrates that good childhood development (measured in various ways) is highly predictive of a large variety of positive outcomes later in life.  If we give children the emotional, social and cognitive support they need in their early years it is hoped that this will make a lasting improvement that persists for the rest of their lives and halt the transmission of negative social problems from one generation to the next.

However, high quality intervention studies are rare, making it difficult to know which type of approach will work (if any).  The lack of research is unsurprising due to the fact that this kind of study is very hard to do well. They usually require a huge investment in time, money and effort. Furthermore, they involve a number of complex factors that might affect both the validity and applicability of the results, such as individual teaching style and child demographics.

This is what makes the recent study into the effectiveness of Tools of the Mind [1], an early intervention programme, by Drs Blair and Raver from New York University [2] a particularly interesting research piece. They argue that children receiving the Tools of the Mind program showed a significant improvement in learning in comparison to children in typical kindergarten classrooms. Importantly, the authors claim that some of these benefits continue into the first grade, after the program had finished, and that many of these effects were stronger in high-poverty schools.

In this post, I will be putting Tools of the Mind to the test, loosely using Dorothy Bishop’s framework for identifying red flags in interventions (It’s a good read. For those interested see [3]). So, what is Tools of the Mind? Is it credible? How solid is the scientific evidence behind it? Does it really work and, if so, are the effects worth the effort and cost involved?

What is Tools of the Mind?

Tools of the Mind is an educational program, developed over 18 years and now used in prekindergartens and kindergartens across the USA and Canada. I will be focussing on the kindergarten program here (that’s ‘reception’ for the British). It is based on the Vygotskian approach: the idea that it is important to teach children to master ‘mental tools’ such as attention and emotion-regulation skills that promote intentional and self-regulated learning. This is expected to develop their executive functions, social and emotional competence at a greater rate. In practice, it involves 60 or more Vygotskian-based activities including activities that require children to create their own learning plans, reflect on their learning and work in pairs with a strong focus on intentional make-believe play tied to stories and literature. Tools of the Mind was named an exemplary education program by the International Bureau of Education at UNESCO in 2001. Maybe this sounds marvellous, but does it actually work?

Who is behind the program and what are their credentials?

The program has largely been developed by Drs Bodrova and Leong. Encouragingly, both appear to be experts in the field. Dr Elena Bodrova was, until recently, Principal Researcher at Mid-continent Research for Education and Learning, a non-profit, non-partisan education research and development corporation, and Dr Deborah Leong is the Professor Emerita of Psychology, Metropolitan State College of Denver. They’ve both written a number of papers, articles and books, and, to the best of my knowledge, have each authored 10 papers in peer review journals in topics related to the Tools of the Mind program. I can’t find any red flags here!

Is there credible science behind the program?

The program claims to improve academic achievement and socio-emotional skills through improving executive functions, and in particular, the ability to self- regulate. Executive functions encompass many different processes involved in the management of cognition and actions: the ability to pay attention to and remember relevant details, to plan, to solve problems strategically and to regulate emotions and behaviour successfully. If you think of your brain as an orchestra, executive functions would be the conductor,  organizing many different instruments to play together as a coherent whole, bringing some in and fading others outand changing the pace and intensity of the music. Indeed, they seem to be involved in pretty much every higher-order cognitive process, calling into question whether the idea of executive functions is too vague and general a concept to be of any use and they are sometimes seen as a controversial topic amongst cognitive scientists.

Putting this issue aside for now, performance on tasks purporting to tap executive functions are excellent predictors of educational progress. More and more research is pointing to the development of these as being one of the most important domains in early childhood for positive short and long-term outcomes. The argument goes something like this: in the same way that a bad conductor is likely to produce dreadful music, no matter how excellent the individual musicians are, poor development of executive function is a very strong predictor of poor social and academic outcomes even if the rest of the cognitive system is well developed. Great claims are made about the extent to which these skills can be boosted by intervention. When you combine this with the fact that growing up in a deprived environment is frequently found to have a profound negative effect on executive functions, it seems a credible and good target for a program like Tools of The Mind.

The approach used to develop executive functions is based on the relatively well known and researched Vygotskian Approach. Unfortunately, I don’t have time and space to review this here but the Tools of the Mind website provides a lot of information on the science behind this.

Is there evidence that the intervention is effective from controlled trials?

Blair and Raver provide the first cluster randomized controlled study into the effectiveness of the Tools of the Mind curriculum in comparison to current practices in kindergartens. They looked at its effects on a range of different cognitive and academic skills. During the two year study, an impressive 795 children took part from 79 different classrooms and 29 schools. The schools were randomly assigned to either the control group who simply continued business as usual or the treatment group who received training to implement the Tools for the Mind programme in their schools. Children were tested in the first term of kindergarten, with follow up tests at a mean of 5 months and 1 year later. It’s questionable as to whether this can be considered an active control group, given that there may have been a potential bias for teachers receiving the new and novel Tools training to expect better results in comparison to those that continued as before, but overall I think this is quite a good study design.

They found that children in the Tools of the Mind classrooms were significantly better at keeping information in their working memory (effect size, ES = 0.14), maintaining attention in the face of distractions (ES = 0.12) and processing information (ES = 0.08) at the first follow up. This was accompanied by a greater rate in academic improvements in mathematics (ES = 0.13), vocabulary (ES not given) and reading (ES = 0.07), in comparison to the control classrooms (although this was only conventionally significant for mathematics). The faster rate of improvement in reading (Figure 1, ES = 0.14) and vocabulary (ES = 0.1) extended into the first grade, becoming significant, after the program had finished, suggesting that it has long term effects.

Most of the effects seen were stronger in high-poverty schools where more than 75% of the pupils are eligible for free or reduced-price lunch. In particular, the effect of Tools of the Mind in comparison to the control group was significant in tests that measured the child’s ability to maintain attention despite emotionally arousing distractors (ES = 0.82), fluid IQ (the ability to solve problems in novel situations, ES = 0.46) and vocabulary (ES = 0.43) in high poverty schools.

Whilst this is the only study of the kindergarten program, it should be noted that both the National Institute for early education research (NIEER) and the Peabody research institute (PRI) have been investigating the pre-kindergarten program. Despite the fact that both studies have included only high-poverty classrooms (around 80%+ children receiving free or reduced price lunch) their results appear to have some substantial disagreements. In one study, NIEER compared 88 children in Tools classrooms with 122 children receiving the district’s balanced literacy curriculum. They found that by the end of the first year, children in the Tools classroom had significantly better classroom experiences, far fewer behavioural problems (an indicator of self-regulation) and some improvement in language performance, although not significant after correcting for multiple comparisons, in comparison to children in the control group. They found no improvement in reading or mathematics. In a subsequent study of the same program but with a slightly different sample (85 children in Tools, 62 in the control classrooms) they found that children in the Tools classrooms performed significantly better in tests of executive functions (stoop battery and flanker tasks) in comparison to children in the control classrooms and that this difference was greater, the more taxing the task.

In contrast, PRI have been conducting a much larger study comparing 498 pupils receiving the Tools curriculum with 379 pupils receiving a range of curricula that would normally be found in pre-school classrooms. They found that the Tools curriculum had no significant effect on direct assessments of achievement or executive function or any teacher ratings of language, executive function, or social behaviour by the end of pre-kindergarten in comparison to the control classrooms. Remarkably, when the children were assessed a year later at the end of kindergarten, they found that the comparison children that had received the normal curricula actually had significantly greater gains in achievement and executive function composite scores and on many of their subtests in comparison to those that received Tools of the Mind!

Are these effects worth the effort, time and cost involved?

Being subjective, this isn’t an easy question to answer. Note that the Tools curriculum can’t simply be taken off the shelf and implemented as other curricula might be: it requires about two years of teacher training including in-classroom coaching and a shift in the teacher’s role in the classroom. However, Blair and Raver appear to believe that: ‘teachers received typical levels of training and implemented the curriculum with materials that are well within the budget of the average kindergarten classroom. Information about the actual cost is difficult to find but in general sources place it at £5000 – £7000 per classroom, which seems reasonable [4, 5] and the materials required appear to be simple, inexpensive and readily available.

However, I think the key to answering this question is to consider the effect sizes produced by the program, the number of standard deviations between the mean score of the control and target groups. In general, the effect sizes found for the kindergarten version of Tools of the Mind in Blair and Raver’s study are relatively low: they all hover around the 0.1 mark, 5 months into the program. This is around half the average effect size for childhood interventions in 2006 found by Duncan et al. [6]. In addition, whilst the effects in high poverty schools look much more promising, it should be noted that the confidence intervals are also much larger for these effects.

To get a feel for what this means, we can get a rough estimate for how this effect size translates into months of development. Bloom et al. [7] found that children progress with an effect size of 1.52 in reading and 1.14 in maths over the first year of schooling. Averaging this and dividing by 12 gives as a rough expected progress estimate of 0.1 per month. This means that many of the effects of Tools of the Mind put children only one month ahead of children not receiving the program.  However, when considering vocabulary in high poverty schools, for example, with an effect size of 0.46, this offers nearly 5 months advantage. Of course, this analysis is something of an oversimplification, but it provides a little context as to whether these effect sizes are meaningful.

How do these effect sizes compare to differences between children coming from disadvantaged and advantaged children?  Whilst it is difficult to put numbers on these differences as many researchers use different measures of what counts  as ‘advantaged’ and ‘disadvantaged’, in order to get an idea of this, I’ll compare the ES to a few studies that found differences within the usual range. I’ll steer clear of the issues in measuring something as vague of ‘executive functions’ and concentrate on one aspect of them, working memory. Noble et al [8] found a difference of 0.31 SD between high and low SES children in the first year of school. Given that Tools of the Mind was found to improve working memory by 0.14, this would be the equivalent of closing half the income related gap. Vocabulary also shows promising results. A study comparing children receiving free school meals (FSM) and those that didn’t at the start of school found a gap of 0.62 SD between the groups [9]. The 0.46 ES for vocabulary in high poverty schools would close this gap by three quarters. Unfortunately, they don’t report what this figure was in high poverty schools one year later, which would be a particularly interesting result. Reading and maths are less promising however. The same study found gaps of 0.69 for reading and 0.68 for mathematics between children receiving FSM and those that didn’t. The ES of 0.13 in maths and 0.14 in reading won’t have much effect on closing this gap, which is a shame, because arguably these are the more valuable educational skills.

In conclusion, Tools of the Mind appears to be a relatively well-grounded intervention program. A random controlled study showed that Tools of the Mind improved children’s executive functions and academic skills in comparison to normal kindergarten classrooms. Despite not being one of the most effective interventions available, the costs, effort and time required seem reasonable and the results suggest that some of the benefits of the program are long term. It holds particular potential for high poverty schools, where the effects of the program appear to go some way to closing income related achievement gaps. However, it is questionable whether the control group used can be considered a full active control and it would perhaps be better to see these results replicated in a study that used a similarly new and novel curriculum or simply used just part of the Tools curriculum as the control group. In addition, results from studies of the pre-kindergarten program by two highly distinguished research bodies are inconclusive with a large study indicating that children receiving Tools of The Mind were at a disadvantage in comparison to normal curricula at the start of school. With this in mind, and the fact that we only have one study investigating the kindergarten program, I would suggest that more research needs to be done to establish the overall effectiveness of this curriculum (and ideally to identify which aspects of the Tools have the greatest effect) before we can advocate it as an effective intervention for kindergarten classes in high poverty schools.

[1] http://www.toolsofthemind.org/

[2] The full paper by Blair and Raver can be found here: http://dx.plos.org/10.1371/journal.pone.0112393

[3] http://deevybee.blogspot.co.uk/2012/02/neuroscientific-interventions-for.html

[4] http://www.washingtonpost.com/local/education/dc-school-reform-targets-early-lessons/2011/11/04/gIQAGZ2VCN_story.html

[5] http://economicdiscipleship.com/2010/12/23/profile-tools-of-the-mind/

[6] https://socialinnovation.usc.edu/files/2014/03/Duncan-Two-Policies-to-Boost-School-Readiness.pdf

[7] http://www.mdrc.org/sites/default/files/full_473.pdf

[8] http://onlinelibrary.wiley.com/doi/10.1111/j.1467-7687.2005.00394.x/pdf

[9] http://www.scotland.gov.uk/Publications/2005/02/20634/51605

State of the Nation Report: Linking Research to Policy

We have recently started a new project focused around children that are considered to be “at risk” of poor educational attainment due to their socioeconomic status. Indexes of socioeconomic status include complex measures of interrelated components, with one key component being family income. As a result, socioeconomic status gives an index of poverty that incorporates other forms of deprivation. There is, at present, an increasing effort to provide better links between research on socioeconomic status, its impact on cognition, and government policy. However, when considering educational attainment, there is still little being done to support children from a lower socioeconomic background. The recently released State of the Nation Report [1] indicates the need for further research into this area. It calls for increased interest and action in order to help close the performance gap between children from different socioeconomic backgrounds and reduce the effects of poverty on children.

Whilst the UK is beginning to recover from the economic crisis, social recovery is lagging behind. For example, a family is described as being in relative poverty when the level of family income is less than 60% of the median UK family income. The State of the Nation report from the Social Mobility and Child Poverty Commission states that, before consideration for housing costs, 17% of children in the UK are living in in a state of relative poverty. After housing costs, 27% are living in relative poverty – that is 3.7 million children. Additionally, the report gives values for absolute child poverty, which are also still worryingly high despite government interventions. The House of Commons Scottish Affairs Committee defined absolute poverty as “the lack of sufficient resources with which to keep body and soul together”. [2]

The State of the Nation Report calls for the UK government to re-think their existing approach to child poverty if they wish to achieve the targets set out in the “2020 challenge” to “reduce child poverty by half and prevent Britain becoming a permanently divided society.” The knock-on implications of poverty on educational attainment are well noted in the scientific literature. In 2013, 38.5% of children receiving free school meals, which is a commonly used indicator for poverty, achieved an A*- C grade in Maths and English, compared to 65.3% of children not receiving free school meals. In addition, research published by the Social Mobility and Child Poverty Commission has shown that children from deprived backgrounds that are considered to be high achieving aged 7 fall behind children from the most affluent backgrounds that were considered to be low achieving at the age of 7. This crossing over in attainment levels is seen by 14-16 years of age (see Figure 1) [3]. This pattern of crossing over indicates that the environmental components of socioeconomic status have an impact on a child’s potential for academic attainment above and beyond genetic influences. Since 2005, the attainment gap in education between children from a low socioeconomic background and children from a high socioeconomic background has only closed by 1.6%. The implication is that those children with a low level of educational attainment are 4 times more likely to remain in poverty.

Figure 1. Trajectories from key stage 1 to key stage 4 by early achievement (defined using key stage 1 writing) for the most deprived and least deprived quintiles of socioeconomic status (state school only)SES attainment crossover_KS1 writing

It is well known that working protects against poverty. Children in working households are only a third as likely to suffer from poverty as those in workless households. However, the report also highlights that working is simply not enough to prevent childhood poverty. In 2012/2013 62% of children deemed to be living in poverty lived in a household where at least one person worked. And perhaps most concerning is that if trends continue, “2010-2020 is set to be the first decade with a rise in absolute poverty since records began in the early 1960s.”

With all this considered, further research is needed to provide a clearer picture of how socioeconomic status influences cognition and education. A key focus of this should involve identification of the factors within socioeconomic status and cognition that mark children as being more susceptible to the risk of poor educational attainment. In turn, greater resources need to be devoted to identifying the most effective interventions. These studies are difficult and complicated to conduct, but as The State of the Nation Report identifies, improving the educational prospects of children growing up in poverty provides one of the best mechanisms for creating a more equal society.

In conclusion, although the issues surrounding the socioeconomic impact on education have in part been researched and evaluated, there is yet to be a thorough exploration of all the factors involved.  Further investigation is needed to identify which cognitive, neural and environmental measures provide the most prominent markers of risk and resilience for children growing up in poverty. Research along these lines is sorely needed if government policy is to target those most in need of support.

[1] https://www.gov.uk/government/publications/state-of-the-nation-2014-report

[2] http://www.bbc.co.uk/news/uk-politics-29686628

[3]https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/324501/High_attainers_progress_report_final.pdf