I’ve been doing my sums, making me feel as if I have come back from my summer hols to face some tedious maths coursework that has to be completed before term starts. But this was not coursework. It’s a review of the NHS initiative to improve access to psychological therapies, IAPT.
IAPT is being rolled out throughout England and Wales gradually. It started with just two pilot sites. Then 34 sites began the roll-out from 2008, and a further 79 from 2009. You can see the locations of the sites here, listed separately for each region: IAPT Regions
The way IAPT is being implemented has a couple of striking features. One is that the programme is separate from other NHS activity in mental health, with its own budget. This prevents NHS managers from stealing the money and spending it on other things (although it does not prevent them from running down other mental health services as IAPT develops).
Another is that the sites collect detailed data, allowing IAPT’s effectiveness to be measured. In particular, IAPT measures outcomes. One of the proposals in the recent government white paper Equity and excellence: Liberating the NHS (page 4) is that the whole of the NHS should measure outcomes, and that outcomes should be used to drive improvement in service quality. So the experience of measuring outcomes in IAPT provides some indication of how well this might work throughout the NHS in future.
A report published last month by the North East Public Health Observatory (NEPHO), a Department of Health quango, reviews the data from the 34 sites in the first roll-out year. It is in two parts: the main data review document, and tables of site-specific data in a separate document. Both documents are available here: IAPT Data Review
In this article, unless otherwise stated, page numbers refer to printed pages in the main data review, and SS-numbers refer to site-specific tables. For a PDF page number, add 4 to the printed page number.
Although the data review starts off being easy to read and understand, it quickly gets into detail that is difficult to interpret. I liked the diagram on page 11, a flowchart that makes it easy to understand what is going on.
Unfortunately, that diagram does not tell the whole story. I had to get my calculator out and do my sums to extract the real numbers from the ones given in the review. In this article, I expand that diagram and explain each part in an attempt to tell the whole story from beginning to end.
One of the complexities in the review that makes it difficult to understand is its use of seven different measures of recovery (‘Movement towards recovery’, page 28). To keep things simple, I only looked at the first of them, identified as MTR1.
Another complexity in the review is that it uses percentages instead of actual numbers to report some of the later findings. In this article I reconstruct the actual numbers that the review conceals.
A third complexity is that some of the site-specific tables are incomplete, covering only 31 or 30 sites. I did not look into how this affects the results.
The total number of people referred to the IAPT sites during the year is unknown. I estimate it as 145,865. This number will turn up again towards the end of my story.
I had to estimate it because two of the 34 sites failed to submit any data (page 9). I estimated the total by working backwards from the average of the 32 sites that did submit data. The total for those 32 sites (137,285) is from Box 2 on page 11, or from the bottom of the ‘Total records’ column in SS1.
In these diagrams, start with the top left number. Subtract the number at the right to arrive at the bottom left number. The bottom left number then carries through to the next diagram.
Numbers marked with an asterisk (*) are my estimates. The other numbers come straight from the published review.
*based on average 4,290 referrals per site
Many of those people who were referred to IAPT for treatment did not reach initial assessment. The group who were never assessed represent the largest exclusion, and therefore the greatest opportunity for improving the service.
Only the remaining 79,310 were assessed initially. The review calls these people the ‘study group’.
These numbers are from Box 2 on page 11.
In the system
Some of those people who were referred to IAPT for treatment were still in the system when the data was collected, so although there was some data for them, their outcomes were not yet known. This left 41,724 who were eventually discharged after being referred and assessed, and possibly treated.
These numbers are from Box 2 on page 11, taken at face value. However, the number 41,724 seems to be taken from SS21, where Site 32 has mysteriously disappeared. This makes me think that it is a thousand or so too small.
Only one contact
More than a third of those people who were assessed initially, but who were not still in the system, had only one contact with IAPT. They didn’t return. Sometimes that one contact included some treatment, and sometimes not, but these people cannot really be said to have received significant IAPT treatment.
This left 26,780 people who at least came back for a second appointment. The review calls this group the ‘outcome group’ because what happened to them reflects the outcomes achieved by IAPT treatment.
These numbers are from Box 2 on page 11.
*1,763 no treatment plus 13,181 some treatment
A few of the people who did return after their first appointment appear not to have had any treatment in their subsequent appointments either, because they disappear from the numbers in the review. I estimated their number by working backwards.
This left 26,648 people who came back for a second appointment and were treated. This is the bottom right total on page 100, and in SS25 for MTR1.
Some of the people who returned for treatment had no final assessment, or its result was not recorded, so their final outcome is not known.
This left 23,117 people for whom there was a known outcome. This is the total on page 100 in the upper part (‘Complete data only’) of the ‘Net change’ column, and in SS24 (mistyped as ST24) for MTR1.
Some people were treated even though their initial assessment did not indicate that they had sufficiently severe symptoms to be regarded as ‘cases’ of illness.
This left 19,467 people who were initially assessed as being ‘cases’, who returned for treatment, and who had a final assessment too. This is the total on page 100 in the upper part (‘Complete data only’) of the ‘Recovery’ column, and in SS24 (mistyped as ST24) for MTR1 at the bottom of the ‘Recovery rate for cases’ column.
Some people did not finish their treatment, either because they declined to be treated, they dropped out, they were judged not to be suitable, or for other reasons.
This left 10,023 people who were initially assessed as being ‘cases’, who received complete treatment, and who had a final assessment. This is the number ‘Completed’ on page 100 in the upper part (‘Complete data only’) of the ‘Recovery’ column.
*working backwards, or adding the other numbers in that table cell
For some people, the treatment did not work. They still had sufficiently severe symptoms after completing treatment to be regarded as ‘cases’ in their final assessment.
This left 5,888 people who recovered after initially being assessed as being ‘cases’, receiving complete treatment, and having a final assessment. I calculated this number from the ‘Completed’ data on page 100 in the upper part (‘Complete data only’) of the ‘Recovery’ column, estimating the number recovered as high as possible while still matching the published percentage.
*58.74% of 10,023 is 5,888, then subtract from 10,023 to get 4,135
You might remember that some people’s symptoms were not severe enough for them to be ‘cases’ initially, but they were treated anyway. Well, a few of them got worse. At their final assessment they had become ‘cases’.
Taking this into account, the net number of people who are known to have recovered is 5,694. I calculated this from the top line in the ‘Incidence’ column on page 100.
*8.2% of 2,365 is 194, then subtract from 5,888 to get 5,694
Note that although the number 5,694 does not appear in the review, it is 46.0% of 12,388, which matches the percentage net change on page 100, at the top right.
“Take away the number you first thought of”
Returning to the number of referrals right at the start, the net percentage of people known to have recovered after complete IAPT assessment and treatment is:
*5,694 as a percentage of 145,865
A 4% success rate, that’s 1 in 25, is doubtless somewhat disappointing. However, I took an austere view of the figures in coming to this conclusion.
Some people were no doubt helped by their experience of IAPT but did not have complete data submitted. At the same time, some people were perhaps unaffected or harmed by their experience of IAPT but got better anyway (or claimed they did).
In a future post, I hope to use the same published data to take a much more optimistic view and look at what might have been achieved.