QLD Police builds cloud-enabled search and prediction tools

Queensland Police Service (QPS) has credited cloud adoption for enabling it to procure capabilities like facial recognition technology, crime prediction models and tools for seamlessly searching data sourced by different branches. 

In the past few years, QPS has invested in emerging technologies to solve crimes faster as well as for crime prevention.

It’s expanded its main data retrieval tool to more internal systems; its in-house face-matching system has assisted more than 1000 cases and its machine learning (ML) model for predicting domestic violence offences will be used to support programs that QPS is already rolling out to de-escalate conflicts before they turn violent.  

The capabilities were made possible by managing and storing data centrally with more standardised rules - moving both structured and unstructured data to an AWS data lake, QPS data services manager Nicholas Moss said.

Clues in the silos 

Moss told the 2023 AWS Public Sector Symposium that in QPS’s previous on-premises environment, “what you’d find in our data are a lot of flat files."

“Investigators and analysts were using data in ways that were successful [but not] repeatable or scalable," Moss said.

“When they ran a similar exercise a week later - although they were still successful - there was a totally different way of doing it because it was all subjective; it was all about people and what they knew and certain things on the surface.”

Moreover, “for every project, we’d build a new data warehouse…the data couldn’t be detached from different lines of business systems.”

The amount of data QPS was storing was growing substantially but data management practices and analytics capabilities that could use it to draw metrics on the whole organisation’s performance or to assist investigators in cracking cases were not keeping up.  

“QPRIME - our fundamental records management system - was growing by a terabyte every year; and that growth was mostly in unstructured data. Everything that was unstructured was not really accessible or usable.

“It could be useful if you were looking at a particular event, but if you wanted to scan all our systems to say, ‘Where have we seen this vehicle before?’ it wasn’t easily done.”

Pressure to predict crime 

Moss said that 2014 - the year of the Lindt Café Siege in Sydney and the G20 summit in Brisbane - prompted QPS to be more concerned that clues may be hidden in siloed systems and couldn’t be easily retrieved or made sense of in a holistic context. 

“Preparing for the G20 was an eye-opener for us as an organisation in terms of showing how much information is needed and how we actually source that information etc," Moss said.

“One of the deputy commissioners said to me that they were really worried about having information in our systems available that we couldn't use.

“We’d spent a lot of time focusing on the security of information and the controls and the auditing but not really on the use of the information and that was about to change in this journey.”

Extreme acts of violence against the public and advances in storage and analytics technology had created increasing expectations that police should utilise data in ways that enable them to stop crimes in advance instead of just solving them afterwards.

“We could answer the ‘What happened’? [and] some of the 'Why did it happen', but we couldn't really answer questions about 'What will happen?' or 'What can we make happen?'

“You might have seen in the media that police are often asked, ‘Why couldn’t you see this coming? You had the information.’ - and yes we did, but it was buried under 50 files on systems with different access controls.”

QPS has faced questions about utilising data more proactively even more in recent times.

The murder of three officers in Wieambilla last December ignited efforts to build a better national firearms register, and in March, a coronial inquiry linked the 2017 death of senior constable Brett Forte [pdf] to his inability to access reports of automatic gunfire in the location his murderer lured him to. 

“I recommend that, as a matter of priority, the QPS implement a solution to enable searches to occur across all QPS information systems and intelligence holdings to ensure frontline officers have access to reliable and current information,” the coroner said. 

QSEARCH 

The pressure to integrate QPS’ systems so that more records could be easily retrieved also came internally from frontline officers, Moss said. 

“When we asked police, ‘What do you want’ they said, ‘We want to be able to search things the way you can search Google.”

Migrating siloed data to the “data lake and the [AWS] platform” expanded investigators' main retrieval tool’s access to data in more systems. 

“QSEARCH is a search tool that searches across structured and unstructured data; it's allowing people to surface pieces of information that they couldn't readily do before, now they’re able to bring it together, share it, do the drawings and start to do some analysis.

“And in that project, QSEARCH, it’s got a graph database; it’s got a whole series of things, but it's a very simple interface to use.” 

Building the platform 

Moving different QPS branches’ data to AWS and establishing rules for accessing, sharing and analysing took nine proof-of-concepts over several years. 

“It's not your traditional data lake - it’s not ‘here's your servers;’ ‘here's your backups;’ ‘here’s this’; ‘here’s that’ - it’s data pipelines; it’s Kubernetes containers; it’s actually built to scale and work across multiple data sources.”

Different QPS branches have more experience sourcing and using different types of data based on the business functions, regions or areas of crime that they work in.

Moss’s team reviewed the rules of the legacy systems that different teams used for their data types when designing rules for how the new platform would manage it.  

“From a search capability perspective, it's been built largely around functionalities pre-determined by systems that we know and by rules that we know. What we're trying to really make more available in addition is the unstructured data.”

In expanding rules for managing structured data and creating new rules for managing unstructured data, Moss’s team consulted the divisions with the most experience dealing with the data type. 

“We were trying to go to the actual custodians of - not the information systems - but of the sources of the information; the components that we are actually going to use.

“If anything was about HR, we wanted the chief HR officer to be the one to say, ‘That's ours’, or if it's finance data we wanted the chief finance officer, or if it's about crime stats we wanted the head of policy.”

The consultation process was equally important for establishing the new platform’s metrics for evaluating different branches’ performance.  

“From a performance lens, it helps to know about the different domains that make up the Force.” 

Sometimes this meant consulting with bodies that extended beyond QPS, Moss added. 

“If we looked at youth justice and what data is important for youth justice, it didn't just come from two QPS systems; it came from external sources as well. To create that domain, you have to bring the right people together to collaborate and work on it.”

Another goal of the migration was standardising data governance enough to streamline processes and synchronise insights without enforcing too many rigid requirements on the front end. 

“I don't care if [officers] want to use Excel to do all their modelling…I never want to have a debate about whether they should be using Power BI or Tableau.

“Let the business choose that based on their skills as long as they ensure everyone can get the right data catalogued, sourced and fit-for-purpose as simply as possible.” 

Emerging technologies 

Moss said that “bringing multiple sources together and multiple domains together to get that insight or foresight to ask questions” was the main aim of the transformation. 

“The final level for what we're doing with the platform is providing the tools or some kind of way to make that happen. 

“So it could be specific tools you're building or it could be an off-the-shelf product. And that's where we tried to go in terms of usage.”

QPS’s facial recognition system - QFACE - is an example of one of the capabilities that the integrated data landscape enabled, Moss said.

QFACE compares prompt images against the more than 10 million photos stored in QPS systems for operational purposes such as its mugshot database. 

“How often do police actually see a suspect from CCTV or someone coming in and say, 'Who is that? Has anyone seen this person?’”

Moss added that ML-based tools like QFACE were “not evidentiary products; they’re discovery products.

“The officer might get some good potential matches, and from the matches, they might say ‘Yeah, I think it’s that person but now I’m going to look at what other features they’ve got in the photo’ or ‘Now I’m going to look at their telephone data.’” 

To make the case to identify threats in advance, “the first thing to say to the organisation was, ‘We're not doing Minority Report; we're not going to be predicting things in the way that that happened’.

“The culture’s said ‘You can’t’ and we don't want that; we want some of those things, but we're not trying to do that.” 

Moss said that QPS’s domestic violence prediction model, “can look at a couple having an escalation through more events and get more confident that something is going to happen.

“It's trying to actually say, ‘Who's in our system that we're not really aware of’ because we might only be managing a handful of cases when there are far more at risk. 

“So it’s trying to give the organisation better information so that they can actually do the research.” 

When asked what features the model’s predictions were based on, Moss said, “We didn't specifically include race or location; those factors are normally biased.

“We can’t let everything get out because then people might say, ‘Maybe if I changed something I won’t get picked up', but what I can say is that we have taken [the historical crime data for training the model] over a four to five-year period; it hasn't just been clipped. 

“And each stage has been validated and predictions have been looked at for what's happened in that period; so we've done lots of checks and balances about this before going out.”