Harnessing the power of observational learning in data quality processing and management

SmartStream’s artificial intelligence uses this innovative technique to ensure data quality in business

Banks and financial institutions in Asia rely heavily on data quality to streamline key areas of their operations. However, the complexity of today's marketplace has resulted in huge volumes of data generated in various inputs and formats, which makes it challenging to validate and verify for accuracy and comprehensiveness. Moreover, delivering data as well as comparing huge data sets across contrasting internal databases are often delivered unstructured and, in many cases, with varying formats. 

Managing these data issues as well as verifying data sets to ensure quality and accuracy poses a challenge to many financial companies. Consequently, many institutions continue to rely on spreadsheets to deal with data verification processes, which often come with many risks. It's not uncommon to find human mistakes, incomplete processes, key-man dependencies, and audit trail problems when dealing with manual validation techniques. 

Bombarded by disparate data coming from all directions, the resulting data quality issues can impinge on straight-through processing (STP) workflow efficiency. The results are financial losses due to added costs and penalties, and in some cases, reputational damage. 

Observational learning solution 

Financial institutions are exploring more innovative ways to solve this challenge through the use of observational learning. SmartStream launched ‘Affinity’ in collaboration with global Tier 1 banks – an artificial intelligence observational learning solution to meet the technical demands and business agility for operational data management and data quality processes.

"It's an umbrella technology for us," said Andreas Burner, chief innovation officer for SmartStream. "The interesting thing about observational learning is that it's very hard to master, but it's the key for future banks because we have so many workloads and so much data that what you want to do is to understand better how users work with that data and optimise that work."

Observational learning borrows techniques that children use to do when learning something by mimicking actions of adults instead of being told what to do. 

"It started maybe two years ago in the banking industry. Everybody was complaining that it was very complex, and with so many rules. We all know from psychology that it's much easier to show something than to explain it," Burner said.

"For us, humans, it's much easier—that's why everyone uses YouTube to learn how to cook. It's more difficult to read about cooking than to watch it.” 

"That's what we're trying to use in machine learning—it's called supervised learning. What we did was take that to the next level and apply that to banks and financial institutions," Burner explained. 

Despite its congruence to financial industries, observational learning can also be applied to various industries where data quality is important to complete a function. This benefits those who are shifting from manual Excel-based processes, especially those that utilise macros to compare data for regulatory reports or check the latest data against past reports to see potential changes.

"I believe that technology can be used anywhere. Whenever you have very standardised workflows based on data, it will find out how a user does its workflow and provide suggestions so that the way SmartStream has implemented that component is very generic and can be used across our solutions suite," Burner said.

Observational learning is available in SmartStream Air Version 2, which changes traditional operating models as it behaves like a consumer app. The solution is the latest AI technology to transform data quality and reconciliation processes that would be measured in weeks and months, to just seconds. It’s cloud-based, it doesn't need training, configuration or even IT projects to work. It will be embedded into SmartStream's flagship reconciliation solutions as part of its Affinity AI offering.  

This AI solution approach by SmartStream has earned substantial cost savings in as much as $2M per year—all by increasing match rates and helping business users handle the huge number of data it gets. 

Observational learning in action

Affinity AI uses observational learning to assess users' actions and then creates its understanding of how records are related to each other. It lessens the time it takes for the user to match complex data sets and once the neuronal network is trained, Affinity acts as a virtual user to support businesses dealing with large amounts of data. The more it observes, the more efficient it becomes in delivering the end-user with high-quality results and more accurate matching rates.

"The machine learning process becomes more and more confident when it observes someone more often. The more it sees things the more confident it gets, and with that confidence, we can do a lot in the workflow," Burner said.

"For example, if the process is more than 99% confident, then we can automate that. And we can say that if the process is 70% confident, then we can just suggest it but not automate it. All the while, the user is getting more efficient because it gets a suggestion and doesn't have to do an entire workflow on its own," he added.

Over time, the system's success rate in providing useful recommendations increases as Affinity observes more instances of mismatches and analysts' responses data sets. For instance, in the event of a mismatch in transaction data within the reconciliation process, business analysts will search information from different systems and sources in an attempt to fix the problem. 

SmartStream Affinity's algorithms observe the steps the analysts will take to resolve these exceptions and will pay particular attention to how they address these data quality issues. This process effectively “trains” Affinity, so that it can suggest useful information to users based on prior successful interventions by analysts.

"It reduces manual tasks and provides suggestions. It's a bit like when you're on Amazon and you want to buy trousers, then the system tells you some shoes could also work with that, or there's a belt with the same colour. It's a bit like that—a recommendation. And it makes it efficient for users because the shoes are already next to the trousers, so it's just one click away and doesn't have to search through the entire database. It's really making these applications very efficient," Burner said.

Observational learning in the future

The potential to meet the technical demands and business agility for operational data management and data quality processes may be exceptional for Affinity's observational learning solution, but Burner believes that we are just scratching the surface of what AI can do. 

"We still see when we talk to our customers that they are just starting to adapt to that so we are at the very beginning of that. The banks have a lot of ideas at the moment, we have been in the business for more than 40 years, and they are approaching us with these ideas and we are starting to release one machine learning after the next. We have eight or nine components and it's getting better. But for sure, in the progression in the next few years, those technologies are getting better. Like Tesla, the self-driving car. At the moment it works on highways, then later it works on cities, and then it will work everywhere and it's the same for us. It's a journey,” he added. 

Still, there will always be humans to ride along the AI journey.

"A human can do a lot, and an AI can do a lot. When you combine human intuition with machine learning, then it's unbeatable. I think that the bank of the future will be very much like that. Similar to observational learning where we give the user much more power, much more data, intelligence, and more insight, then a user can then do the clever things and confirm what the AI suggested. Users will also only see what they need to see and will be assisted by AI. Its learning algorithm will tell you to stop and warn you when what you're doing seems to be incorrect," Burner said.

SmartStream focuses on researching ways in which advanced technologies, such as AI and machine learning, can be deployed in its solutions. The company, which is one of the earliest to utilise that technology in its solutions, still thinks AI still has a lot of untapped potential waiting to be discovered.

"We had a lot of moments where technology surprised us because it's so brand new and we, as humans, are just starting to understand these possibilities. It's really amazing," Burner said.