Robodebt report calls for automated decision-making review body

The final report into the robodebt scandal has recommended a body be set up to monitor automated decision-making by government agencies.

The report [pdf] also said the government should seek advice on the legality of "end-to-end data exhange" processes operating between Services Australian and the tax office.

In total, the Royal Commission made 57 recommendations following the robodebt scandal, where incorrect debts were raised against welfare recipients by an automated system.

One of the recommendations calls for better legal and ethical governance of automate decision-making processes.

It seeks a new or expanded governing body “with the power to monitor and audit automated decision-making processes with regard to their technical aspects and their impact in respect of fairness, the avoiding of bias, and client usability.”

It also called for the federal government to “consider legislative reform to introduce a consistent legal framework in which automation in government services can operate.”

There should be “a clear path for those affected by decisions” to review and understand implications of automated decision-making and “explaining in plain language how the process works, and business rules and algorithms should be made available, to enable independent expert scrutiny", the report said.

“A trustworthy automated system is a system containing automation that is ethical, lawful and technically robust, coupled with good governance and risk management," the commission found.

"To achieve trustworthiness, the system must be designed with human agency at its centre."

The report said the automation used for Robodebt “involved a system of business rules with no ability to move outside of specific and defined action on the basis of the data received.”

“It was extremely rigid; once the rules had been coded and set in place, the system itself would stay in place until the rules were changed by way of human intervention," the commission found.

The report said people should be told when “they are the subject of automated decision-making” and how to challenge outcomes.

“AI and automated decision-making systems should function in a robust, secure and safe way, with potential risks continually assessed and managed," the commission said.

The report called for “accountability for organisations and individuals developing, deploying or operating AI systems for their proper functioning”.

“The robodebt scheme serves as an example of what can go wrong when adequate care and skill are not employed in the design of a project; where frameworks for design are missing or not followed; where concerns are suppressed and where the ramifications of the use of the technology are ignored," the commission said.

The commission noted there was a prospect of future programs, using increasingly complex and more sophisticated AI and automation, having even more disastrous effects - magnified by the “speed and scale at which AI can be deployed” and the increased difficulty of understanding where and how the failures have arisen.

However, it also noted that AI and automation, done well, could potentially create effective government services.

“The concept of 'when done well' is what government must grapple with as increasingly powerful technology becomes more ubiquitous," the commission said.

Data-matching questioned

The report also calls into question whether data being exchanged between Services Australia and the Australian Taxation Office (ATO) is legal or not.

It recommends the government seek legal advice on these "end-to-end data exchange processes which are currently operating" between the two agencies.

Many such data exchange programs came out of the pandemic era, with the matching used to monitor spending on programs such as JobKeeper and later JobMaker.

"The ATO and [Human Services] should take immediate steps to review and strengthen their operational governance practices as applied to jointly conducted data-matching programs," the commission recommended.

"This should include reviews to ensure that all steps and operations relating to existing or proposed data-matching programs are properly documented; a review of all existing framework documents for existing or proposed data-matching programs; [and] a joint review of any existing or proposed data-matching program protocols to ensure they are legally compliant in respect of their provision for the data exchanges contemplated for the relevant data-matching program."