AI in the NDIS [Mick Barrett]
AI experts are warning that automated planning risks pushing the NDIS towards a search for the average. They say that’s a problem, because there’s no average participant.
Asked to name the one non-negotiable safeguard for the next five years, a panellist at the AI in the NDIS Summit went straight past privacy, past cybersecurity, past every risk the sector has been rehearsing, to land here:
"My big concern is the automated decision-making component — that we'll see AI-generated recommendations for plans, and that they average towards the mean. But we know that every participant's needs are different. If we start to see averaged, recommended plan sizes, that really worries me."
A panel of digital sector leaders, including EY’s Christina Larkin, former NDIA Deputy CEO Anthony Vella, Darren Chua from UTS, and Jessica Martin, founder of Bella Sláinte and Google strategy lead, were examining risks around the increasing use of AI in the NDIS.
Ironically, the most useful thing said about artificial intelligence in the disability sector at the summit was not about artificial intelligence at all. It was about the search for the average.
The NDIS was built on the proposition that support must be shaped around the person, not the mass. An engine trained on what has been funded before is optimised for the opposite. It rewards consistency.
The problem with this is that consistency is not fairness. It can be the precise mechanism by which unfairness is made invisible, defensible and enormous.
The new assessments are undergoing testing now and will be introduced from April next year. Using the I-CAN v6 tool budget model to generate plan funding will be a great idea - if it works.
[continue reading from abilityNEWS newsletter]
The question the agency has not yet satisfactorily answered
Everything turns on one question, and it is a short one. Can a human being change the number?
The NDIA's public position has been consistent: trained staff, not automated systems, approve plans. The I-CAN assessment and accompanying questionnaires inform a decision that is made by a person.
Recent rporting suggests something more complicated. In March, Crikey's Rick Morton reported on internal documents showing senior staff in the NDIA's own technology services division warning the policy design behind New Framework Planning was in chaos, reform was off track, and that the July start was at critical risk.
At the centre of this sits the algorithmic budget model. It’s an engine that’s roaring to go forward - but not necessarily connected to a steering wheel.
There has also been reporting delegates cannot directly amend a budget the model produces: even if a number looks wrong, the answer is to request a fresh assessment with different inputs instead of correcting the output.
If that is accurate, the human in the process is not a decision-maker. They are a witness.
This has echoes from the past. Robodebt was not an artificial intelligence system. It was a much simpler thing: an automated process aimed at people with little capacity to fight back, wrapped in an assurance that humans remained involved. Humans had been left with nothing meaningful to do.
The NDIA should be able to answer this in a sentence. Can a delegate increase a budget generated by the model? Yes or no.
"Human in the loop" is not the test
The panel offered a better standard than the one the sector has been using, and it deserves to travel.
"Very clear governance, very clear guardrails. And then along those processes, not only human in the loop, but meaningful human decision along the way."
Human in the loop has become the reassurance of choice across government and industry. It is nearly meaningless. A person who reviews a hundred model outputs an hour is in the loop. A person who can only press accept is in the loop. A person who has no authority to change the figure in front of them is in the loop.
Meaningful human decision asks something harder. Could this person have reached a different conclusion? Did they have the information, the authority and the time to do it? And is there a record showing they actually considered it?
Applied to the new planning framework, that is the whole story. Not whether a human was present. Whether a human could have said no.
What the data remembers
There is a second problem, quieter and harder to fix.
A model trained on historical decisions inherits historical judgement — including the parts nobody would defend if they were written down.
"There is some inherent bias built into historical data … and we know bias in health data is typically skewed against females. Many of the early studies a lot of healthcare is based upon were based on studies of men."
"That's what concerns me — that we're repeating the past by training these models with data that has inherent bias, and that bias is perpetuated going forward."
Apply that to this scheme and the question becomes uncomfortable quickly. Whose support needs have historically been well recorded, and whose have been minimised?
Women whose pain has been discounted. First Nations participants in communities where services were thin and so demand looked low. People with psychosocial disability, whose needs fluctuate in ways that a point-in-time assessment reads as inconsistency. People in rural and remote areas, whose historical funding reflects what was available rather than what was needed.
A model does not know the difference between a need that was absent and a need that was never met. It sees the same thing in both cases: a smaller number.
Encoded once, that judgement stops looking like a judgement. It starts looking like data.
The evidence trail
Ask a participant to imagine their next plan reassessment. The budget comes back lower. They ask why.
What is the answer?
"That's what the assessment produced" is not one. And from 10 December it may not be a sufficient one in law either. New transparency obligations under the Privacy Act commence that day, requiring organisations to disclose in their privacy policy where personal information is used in computer programs that make, or substantially assist in making, decisions significantly affecting a person's rights or interests — expressly including decisions affecting access to a significant service or support.
That is a description of an NDIS plan.
The obligation is disclosure, not prohibition. Nobody has to stop. But an organisation cannot disclose what it has not mapped, and it cannot explain what it does not understand. Legal commentary on the reforms has pointed directly at Robodebt as the cautionary example, and at the expectation that people affected by an automated decision should be able to seek human review of it.
This was the panel's frankest admission. Asked what an audit trail looks like when the system is opaque, one panellist ran through the three domains that can be assessed — governance and risk, the data itself, then the systems and models — and then conceded the obvious.
"That's very difficult in a black box scenario."
If it is difficult for an assurance specialist, consider the participant. They cannot see the model. They may not know one was used. Their review rights sit downstream of a calculation nobody will show them.
Meanwhile, the regulator is building one too
The planning system is not the only algorithm being pointed at people in this scheme.
The NDIS Quality and Safeguards Commission has been developing what has been described as an intelligent risk engine, part of a four-year, $160 million Data and Regulatory Transformation program, reported as due by August. Officially a decision-support capability, it is designed to identify providers, workers, participants and networks presenting the greatest risk of harm, poor quality, fraud or non-compliance.
Note the third word in that list. Participants.
After a fraud inquiry and with the Australian National Audit Office beginning a program of NDIS audits, the case for better detection is not hard to make. Nobody defends the status quo.
But a risk score attached to a person with disability, generated by a system they cannot see, feeding decisions about scrutiny and access, is a serious thing to build quietly. The Commission published its own AI transparency statement in February, which is more than most agencies manage. The next test is whether the risk engine gets the same treatment.
The case for the defence
The panel was not hostile to any of this, and the story is weaker if it pretends otherwise.
The strongest argument for AI in this sector came from the same stage, from a panellist with senior experience inside the agency:
"I'd like to think AI is part of the solution to absorb the administrative burden — which means practitioners, whether a support worker or allied health, get to spend more time with participants … more time with people, getting great outcomes."
That is a real prize. Documentation is eating the workforce alive.
There is a second, less obvious benefit. Audit and assurance have always been sample-based — pick a sample, hope it represents the whole, hope you have not missed the outliers.
"Previously it was done on sample-based statistical modelling. You wouldn't have the time and resources to look at the whole population, so you'd pick a sample … and hope you'd pick up enough to give you comfort. Hopefully you don't miss any outliers."
"[AI] gives you the ability to analyse large volumes of data and take assurance for the whole population … effectively continuous controls monitoring, 24/7."
In a scheme where the outliers are the entire point, whole-population assurance is genuinely valuable. It could find the participant nobody has visited. It could catch the provider whose numbers have quietly stopped making sense.
The technology is not the argument. What it is pointed at is.
One more thing
In February last year, the Administrative Review Tribunal upheld an NDIA decision refusing to fund a participant's ChatGPT subscription. Among the agency's stated concerns, as reported by Crikey: the risk that the tool would produce false information, and poor value for money.
The agency worried, reasonably, that an AI system might tell a participant something untrue.
It is now building one to work out what they are owed.
What has to happen before this rolls out further
Publish the override rule. State plainly whether a delegate can change a model-generated budget, and if not, say why that is compatible with an individualised scheme.
Publish the automated decision-making disclosure before 10 December, and write it so a participant can read it.
Commission an independent audit of the budget model for bias across gender, First Nations status, psychosocial disability and remoteness — and publish the result.
Guarantee a real review path. Not a rerun of the same model with different inputs. A route to a human who can reach a different answer.
Tell people when a model was involved in a decision about them.
The scheme's founding promise was that support would follow the person. A system designed to find the average is a threat to that promise, and the safeguards need to be in place before the first appeal, not after it.
There is no average participant.
That was always the point.
The next six months
1 July 2026 — New Framework Planning begins for participants aged 18 and over with less complex needs. Mandatory registration expands to supported independent living and platform providers.
August 2026 — NDIS Quality and Safeguards Commission intelligent risk engine reported as due, under the $160 million DART program.
1 October 2026 — Thriving Kids begins for children aged eight and under with developmental delay or autism and low to moderate support needs.
10 December 2026 — Privacy Act automated decision-making transparency obligations commence.
To 2 October 2029 — All participants aged 16 and over transition to New Framework Planning.