News
Recent news stories have revealed how the algorithm used to determine how much in-home care funding older Australians receive can produce disastrous results, leaving some vulnerable people with less support than they need.
Now, there’s a risk that Support at Home Program’s fundamental mistakes could be replicated with the NDIS. The NDIS is developing a new Support Needs Assessment, built around an adapted version of I-CAN, which is the tool used to evaluate the support people with disability need in everyday life.
I-CAN has a genuine research base. But there is a problem - most of its published validation research has used people with intellectual or developmental disability. The NDIS population is much broader. It includes people with acquired brain injury, autism and psychosocial disability. These are very different conditions, which can present very different problems for assessment. We cannot assume that because an assessment works well in one disability group, it will work equally well in another. We must test it. Properly.
Take someone with an acquired brain injury. A man may make good eye contact, speak fluently and confidently tell an assessor that he manages everyday life well. During a semi-structured interview, like the I-CAN, he may look quite capable. Yet impaired awareness of disability is common following brain injury. He may believe he is managing well, while being unable to organize his everyday life, regulate his behavior or remain safe without considerable support.
The new I-CAN assessment will allow input from people who know him, but having a family member present does not solve the problem. They may be reluctant to contradict him or describe his difficulties in front of him. They need an opportunity to speak separately. Without independent information on this man, the assessment will not accurately assess his support needs.
Psychologists call this ecological validity: does an instrument accurately reflect how someone actually functions in everyday life? Other disabilities present different problems. Someone with psychosocial disability may function well at the time of assessment but deteriorate markedly at other times. An autistic person may present as highly capable during a semi-structured interview but require substantial support to maintain that functioning in everyday life.
Before this new instrument is used to determine funding for these groups, we need evidence that it works well for them. At present, that evidence is not there. And there is another problem. Even if the new Support Needs Assessment accurately measures someone's support needs, it does not follow that it can accurately determine how much NDIS funding that person requires.
Earlier I-CAN research found that support-needs scores explained only part of the amount of paid support people actually received. Support need and the cost of meeting that need is not the same thing. NDIA needs to answer two questions: i) does the new instrument accurately measure the needs of different disability groups? ii) has the method for turning those results into dollars been shown to produce appropriate individual budgets?
The aged-care experience shows why both questions matter, and before any instrument is used to determine NDIS budgets, it should be properly tested across the full range of disabilities it will be used to assess.
Standardization is a good idea for both aged care and NDIS assessments- it can improve consistency and reduce the influence of individual human judgements. But standardizing a poor decision does not make it a good one.
Good psychometrics reduces the amount of error, tells us how much confidence we can place in a result, where its limitations lie, and whether it can reasonably be used for the decisions we want to make. For an older person, getting this wrong can mean insufficient help to shower safely or remain living at home. For someone with disability, it can mean insufficient support to manage everyday life, participate in the community or live independently.
When an assessment determines how much support a person receives to live safely and independently, getting the measurement wrong is not a statistical inconvenience, it can fundamentally change someone's life. The aged care classification algorithm example should act as a warning - the more consequential the decision, the stronger the evidence we should demand before allowing an algorithm to make it.