ANZ examines uneven benefits from AI-augmented coding

ANZ Banking Group is starting to see some uneven benefits distribution from its use of AI pair programming, with training being offered to drive value for different teams.

Chief technology officer Tim Hogarth told a recent GitHub Galaxy24 event that as a result, the bank hasn't settled on metrics to measure the “collective value” that AI pair programming brought to its software engineering operations.

The bank was an early adopter of GitHub Copilot.

While initial results were promising enough to expand access to the technology from a trial cohort of 150 engineers out to 3000 engineers in total, benefits and value have accrued unevenly since - and in some cases even ANZ is unsure why.

“We essentially started with the rollout last year and made it available immediately to 3000 people, and then we're encouraging people to onboard and monitoring the activity,” Hogarth said.

“We're at the point now where there's this portion of our population of engineers who are very actively using this.

“[But] we've still got a long tail of people who are still not using it as a ‘first order’. They’re still building that familiarity [with the technology].”

Hogarth said that GitHub Copilot-specific training is one way the bank is getting more engineers familiar with the tooling and ways in which they might be able to incorporate it into their existing workflows.

“In the proof-of-concept, people were enthusiastic about the trial, but not everyone's ready to change, so we sat down with teams and gave them additional coaching to help them build familiarity and new habits,” Hogarth said.

“It's true, I think, for all of us using AI. It's not always our first instinct [to turn to AI for answers].”

More challenging, however, is “a few niche teams” for whom Copilot didn’t produce much value.

“We’ve got to dig into more of that,” Hogarth said. “We don't know if it's the technology, the team or the nature of the problem they're trying to solve.”

The uneven distribution of value at a team level means that quantifying the “collective value” that Copilot is producing for the bank remains a challenge.

More potential answers than accepted answers

Hogarth also said that of the bank’s use of GitHub Copilot so far, it generated “about two-and-a-half times” more code suggestions than were actually accepted.

“There was a bit of a ‘mystique’ [with Generative AI] that you just type in your problem and bang, you'd automatically select [the answer]. That gave some people conceptions that our engineers would blindly accept solutions,” Hogarth said, adding that the ratio of rejected suggestions to accepted suggestions showed otherwise.

He also said that “suspicions” that the quality of the bank’s code output would suffer had also been unfounded. “We’re actually finding it’s better,” Hogarth said.