Westpac sees 46 percent productivity gain from AI coding experiment

Westpac saw a 46 percent productivity gain, with no reduction in code quality, from software engineers that were aided by generative AI compared to a control group that performed the same tasks exclusively by hand, in a recent in-house experiment.

The AI coding experiment took place in the bank's Growth Labs function, which has been conducting AI experiments over the “last couple of years", according to Westpac chief technology officer David Walker.

Westpac’s Growth Lab - an extension of its innovation unit, ‘Co.Labs' has “been working with generative AI now for "two-and-a-bit years", he said.

Walker said he wanted to test the impact of generative AI on software development, and whether it helped or hindered coders and their output.

The experiment brought 60 engineers together and “randomly divided them into four groups”.

“One of those was a control group. That control group had to hand code basically, they had to do what they would normally do," Walker said.

The other three groups were given generative AI tools from Microsoft, Amazon and OpenAI, and "about three or four hours" to get familiar with the basics.

Each team was then given seven tasks across various coding languages, comprising of assignments including extracting and exporting data, creating unit tests and data transformation.

The test was held “over a number of days,” Walker said, adding that “the headline result was quite stellar.”

“We found that compared to the control team that just had coded it the normal way, we've got a 46 percent productivity gain across the board in terms of generative AI tools supporting the coding of these three teams that were given these tools, which is quite amazing," he said.

The productivity gain was accompanied by no noticeable drop-off in code quality.

"We scanned all the code for vulnerabilities, and looked at it from a maintainability and reliability point of view - all the key metrics that you might look at from when you look at code quality, and we saw no drop-off," Walker said.

He added: “When we looked at the time that it took the hand-coding team to actually complete their tasks on average it was three-and-a-half times longer than it took the other teams to do their tasks."

The bank also evaluated the impact of generative AI assistance on developers with differing experience levels.

Walker said 83 percent of junior engineers were “blown away” and appreciated “there was an aid to help people that are early in their career”.

Meanwhile, more experienced engineers found the tools took care of “laborious tasks” and “allowed them to focus on more complicated aspects of the software”.

Walker said while the experiment was "short and sharp", the outcomes were "fantastic".

Feedback collected from participants showed they were “extremely excited to be able to blend these tools into my daily workflow” with others remarking they were “able to get functioning Python code to do exactly what I wanted just by asking the AI the correct questions."

“Writing Python code is quite a tricky thing if you've never done it before and here we have someone who's the developer, but maybe a specialist in Java, or one of the other codes, able to use [generative AI] to work with it," Walker said. "So [it has the potential to deliver] very powerful outcomes.”

The mesh

Walker said Westpac’s internal engineering platform called ‘the mesh’ is where its hoped results of the experiment can be applied.

“We've had [the mesh platform] up for about five years now and it’s been where our engineers go to build user interfaces, APIs, microservices, all the sort of technical components that we build applications with," Walker said.

“We roughly run about 40 percent of our development through that environment and so that's where we want to start.

“That's our environment where we build a lot of our in-house code and … things like our websites and mobile banking apps and a large number of the applications we build ourselves, all get built on that environment.”

He said the platform is “already a very highly productive environment for our engineers” and that new AI capabilities are expected to “enhance that even further.”

“What we want to do is get all the toil out of the way, get all the noise out of their way and let them just do what they want," Walker said.

“The mesh environment with [generative AI] plugged into it, that'll give about 40 percent of our engineers straightaway access to the capability and we've already got the first teams already live now on it. So, we've already started.”

Walker added Westpac's “starting to heat the engine up” and build momentum towards much of the foundational work it’s been developing over the years.

He said “engineers, in my mind, are first-class citizens in the bank” and that he was excited to offer them the generative AI tooling.

“They're the people that we really want to make sure are as efficient and effective as possible," he said.

Walker added that generative AI “is a big leap forward and therefore giving [engineers] things [to make] their job more efficient and have more fun with and help them grow in terms of their careers is a huge thing.”

Kasisto and AI

Walker said the AI coding experiment is separate from its work with conversational AI company Kasisto, which Westpac is leveraging to develop a finance-specific large language model (LLM).

In April, Westpac said the partnership will help create LLMs that are safer and more accurate than ChatGPT.

Walker said given public LLMs are “trained on the internet, they can say and do anything and they can mislead”.

“We can’t afford that … that's where Kasisto comes in,” he said.

“Kasisto has trained a large language model for the finance sector. 

“Kasisto model’s is very specifically being trained on conversations in banking and that's why they're specialised in that area and that's why we know the safe and not going to mislead or do anything wrong for our customers, or staff.”