How Woolworths uses Google to power its massive analytics uplift

Woolworths is set to use data analytics to localise the range carried by its stores, personalise offers to customers and optimise its entire value chain under a large-scale project revealed today.

Woolworths' Angelo Clayton.

General manager of IT for stores and business intelligence, Angelo Clayton, told Google Cloud Next 19 in San Francisco that the retail giant stood up a new data infrastructure in five months through the back-half of last year.

That infrastructure comprises a Hadoop-based data lake that is being used to bring together data that was previously held in siloed data warehouses across the Woolworths business, including finance, supply chain, and retail.

From there, the data is brought into the Google cloud platform where Woolworths is querying it and increasingly using machine learning and other cloud-based services to extract key insights.

Until now, Woolworths has kept details of its data platform and experimentation a well-kept secret. Some aspects of the project were revealed by iTnews in June of last year.

Woolworths’ openness now about what has occurred appears to be the result of having the platform bedded down to the point where it is anticipating major changes and improvements to occur as a result within months.

It is also likely a warning to rivals such as Coles which have also been working in recent months to stand up similar advanced analytics capabilities.

Immediate use cases emerge

Clayton said that Woolworths had started running proof-of-concepts on Google cloud platform in June last year.

“We didn't just want a platform that could compute [at scale] - it was really about the value-add that the platform would bring over time to us,” he said.

“So not only did we do the compute piece, we also did a lot of work around machine learning and some use cases around natural language processing.”

Clayton said that these more advanced use cases should make appearances within months.

“Some of the use cases we know that we can go and unlock over the coming months,” he said.

“How do we do price and promotions far more targeted to our customers?

“How do we drive localisation around our range - again, through the insights of the data that we've got?

“How do we start to really optimise our entire value chain? We're talking to Google about this at the moment.”

1.5hrs on-prem, 2.3 seconds in Google

Working with Google and its long-term technology partner Tata Consultancy Services (TCS), Woolworths managed to stand up the new data infrastructure in a very fast timeframe.

“Projects like this [usually] take you two to three years to be able to land,” Clayton said.

“For us to be able to do this in the space of four or five months was just an amazing effort.”

Clayton said that Woolworths began to properly explore big data and the value it might bring to the organisation about a year ago.

“It really started with a saturday afternoon session with some of the senior leaders in our business asking what our data ambition was and what are some of the things we thought we can achieve with [a stronger] data asset,” Clayton said.

“As we started to go down that path, we started to realise that data as an asset is something that we can go and unlock and start to drive an analytics mindset throughout our business.”

However, Clayton said that Woolworths’ IT quickly came to the realisation that it didn’t have the data infrastructure to support the leadership ambitions.

That led IT to start scoping what it needed “to put in place to be able to unlock that group data strategy”.

“One thing we needed was to start to put down a capability around trusted, reliable data that’s really easily accessible,” Clayton said.

“One of the first things the business teams said to me as I joined was, ‘We’ve got all of this data, I just cannot get my hands on it’.

“Part of that challenge was the data literally sat in so many different places within the organisation. Each business unit built its siloed data warehouse, and it was really just impossible to be able to bring all that data together in a trusted and reliable fashion.”

Clayton said Woolworths also wanted a data infrastructure that would not date quickly, was cost-effective, enabled self-service of analytics capability by business users, and was secure.

“We have close to 10 million customers,” Clayton said.

“It is imperative that our data needs to be in a place that's secure and only we can get our hands on [it].”

The result is a tiered architecture consisting of a Hadoop-based data lake for group-wide data, with Google cloud platform services layered on top.

The self-service capability is enabled via an implementation of Tableau, which leaked out late last month.

Clayton said that Woolworths was already comfortable with the Google cloud platform (GCP) - it is a large existing G-Suite user and runs other cloud services from the company.

However, Woolworths was unconvinced that Google’s BigQuery service was capable of delivering what the retail giant wanted.

“As a platform, we were quite comfortable with GCP. I guess we were probably a little bit more circumspect about BigQuery as a capability in the GCP stack,” Clayton said.

“Because it's been maturing over the last few years, we put a bit of a challenge to Google and said to them, 'take some of our data and see how quickly you can process it'.

“We gave them 80 billion rows of data. They loaded it on the platform and typically on our on-prem platform, the queries we run against that dataset takes anything between 45 minutes to an hour-and-a-half.

“Google ran the queries on their platform and it took 2.3 seconds, to be precise.

“That was the point we sat up and said, ‘I think there's a capability [here] that could support our ambitions’.”

Clayton said that similar sub-second performance was seen when experimenting with other Google cloud services that Woolworths now expects to use with its new data infrastructure.

Standing up the new infrastructure

Woolworths began standing up the new data infrastructure in July last year.

“Really, by the end of September, we were live on the GCP platform,” Clayton said.

“While that was great, and the platform was up and running, we needed to put the enterprise visualisation tool [Tableau] on top of that, and we did that by October.

“Then, we actually needed to get data into the platform. And literally, by the end of November [2018], we had put our core retail data set onto the platform - two years worth of history moved into Google, ready for the business to be able to consume and work with [it].”