Scaling Artificial Intelligence (AI) in Open Banking: Building Trustly’s Machine Learning Platform

“It is all about the data!” This line stuck with me as I decided to join Trustly as their first Machine Learning Engineer in early 2023. It’s impressive what we have accomplished in such a short time frame since we started building our Machine Learning Platform! The time between model development and model deployment used for decision-making in production dropped from months to about one week. 

We introduced model governance and several pre and post-deployment safeguards to de-risk production errors affecting our traffic and profitability. We also decreased model response time from seconds to just a few milliseconds. The milestones achieved are so many that it becomes hard to count. In this blog, I will describe how Trustly joined the race to become a major player in Machine Learning and AI in the Open Banking space and why we are well-positioned to win it.

Guaranteed Instant Payments Depend on Open Banking Risk Models

If data is the oil that fuels long-term profitability, North American Open Banking is a reservoir yet to be discovered, buried deep down within decades of financial system inefficiencies and complexities. Let’s focus on one of those inefficiencies: traditional payment rails like ACH. ACH in the US takes days to process, and it’s been cumbersome and inconvenient for customers to pay with their bank due poor user experience and long settlement times. For example, can you imagine purchasing crypto assets and receiving them days later? This was the case until Trustly arrived on the market with Guaranteed Payments. 

Now, you can log into your bank, choose from which bank account you want to pay from, and with a few clicks, your payment is complete. The merchant receives their payment, and you receive your digital assets immediately. Any customer can immediately pay straight from their account, from any bank, with the click of a button and without needing a credit card or manually inputting account and routing numbers. Additionally, Trustly’s Open Banking platform helps merchants reduce expensive processing fees (the average credit card processing fee is 2.24%, which adds up to merchants’ payment costs). 

To make the ACH payment happen in real-time, Trustly pays the merchant directly out of its own pocket at the time of the transaction and then later collects the payment from the customer's account. As you can imagine, Trustly may be unable to collect the payment due to fraud (which is still much lower than card-not-present fraud, thanks to the bank's Multi-Factor Authentication) or simply because the customer didn’t have sufficient funds during the transaction. A few risks, such as the ones described above, are involved in providing such an instant payment experience, but that's where machine learning comes in. Machine learning bolsters the Trustly risk engine to mitigate against risky transactions and ensure we can still provide cost-effective guaranteed payments.

Trustly built payment connectors with most banks in North America, covering 99% of financial institutions between Canada and the US. This technology allows Trustly to access and use fresh Open Banking data to conduct risk analysis and decide whether to approve or decline a transaction. Our platform’s capability is directly tied to the success of our risk models. While the stakes are high, the ability to access fresh consumer financial data gives us a competitive advantage to build data products at scale using machine learning.

The Machine Learning Platform: High Stake Decision Making at Scale

We built our Machine Learning Platform to support decision-making for transactional risk evaluation. Let's talk about scale: Trustly connects businesses to thousands of financial institutions in the US and Canada as an account-to-account payment method. 

One of the hardest challenges in Open Banking is the difficulty of establishing a single data standard among all financial institutions. Every institution has its particular ways of managing and sharing its data. Trustly consolidates the financial data across financial institutions and categorizes it in a single standard data model. Our unified data model allows us to build a Feature Store that centralizes data from all the financial institutions connected to Trustly: the same sets of features can, therefore, be used for both major banks and local ones. This high level of standardization allows us to scale our ML operation across institutions and geographies. We currently have more than 1,700 features that can be reused among all our machine-learning models to improve decision-making.

‍ 

‍ 

While features can be plugged and played into any model, we often need to give specific attention to key merchant partners or financial institutions. For example, a major financial institution in the US may require a specific model, while minor institutions with similar behavior can be grouped in a single model. Trustly currently operates several individual Machine Learning models along with other models that challenge the ones in production to ensure a continuous feedback loop. We keep striving to improve our performance. 

While in most industries, Data Science teams often operate as craftsmen, where each model development has a dedicated Data Scientist and its own code, Trustly requires a more industrial approach. As part of our platform, we created a standardized model pipeline covering all aspects from model creation, deployment, and observability, including data preparation, feature selection, hyperparameter tuning, regularization, features validation, orchestration, artifact tracking, and service endpoint wrapping. This pipeline is versioned in an internal library. It is config-driven and is based on configurable steps. 

For example, it is mandatory to apply hyperparameter tuning for model development, but choosing different tuning algorithms through a declarative configuration is possible. Through this effort, we managed to reduce model development time from about 3 months to a single week. We were able to refresh all our models through automated retraining thanks to having this standardization in place.

Last but not least, our most requested models operate and maintain robust Request Per Second (RPS) performance under regular business conditions, and can demonstrate significantly increased capacity during major events like the Super Bowl. Additionally, since we operate in a transactional business, the risk evaluation must respond to strong time constraints. No customer will wait seconds to have their order processed. As we introduced methodological advances in feature selection, regularization, and parallel processing, we reduced up to 10x the response times and have all our new models with p99 latency below 200ms with some models actually responding below 35ms!

Besides scalability, we also need to remember that we offer a guarantee and serve businesses in high-risk verticals like crypto and betting, with a direct impact on our P&L if anything goes wrong with the risk evaluation. If our machine-learning platform goes down, we are in serious trouble! To prevent this risk, we developed strong control measures and processes: Every change in models and policies follows strict governance procedures, like peer review, model cards, automated testing, test coverage checks, stress testing, and several other forms of validations. Additionally, we have safeguards in place to reduce the blast radius and severity of potential incidents. For example, every model is escorted by at least one fallback model that can be activated if the main one becomes unavailable. The fallback model can act as a substitute if a feature group fails to be calculated or the model endpoint starts to timeout, thus allowing us to continue operations without downtime in case of failures. 

Scaling AI in Open Banking

We are a small and young Machine Learning Engineering and Data Scientists team. There is still plenty of foundational work to do. It may not be today, tomorrow, or next year, but we are confident that our work will set the stage for world-class AI in Open Banking. Trustly has cracked the surface and dug so deep that it hit the big reservoir of the North American financial system that is Open Banking data coupled with machine learning. The data is pumping. It is time to continue building our machine learning platform to fuel new Open Banking applications.

‍