Zero impact data pipelines from Mainframe Systems to a modern computing environment
The second largest bank in the US relies on IBM Mainframes to validate, record, and retrieve financial transactions for a wide spectrum of customers. IBM Mainframes are renowned for their resiliency, durability, and security. Mainframes provide unmatched guarantees when it comes to financial transactions. They provide customers with a sense of security and the Bank with a platform that is always available.
top 10 global bank
of all US deposits
updates/day across 10 core tables
accuracy for transactions
< 2 sec
The Bank had flattened revenue growth and a declining customer base over several years. For a Bank to grow and thrive, adding new customers and selling new services to existing customers is vital.
An additional challenge for the Bank was the deregulation of the Banking industry, which resulted in the introduction of many new players competing for the same customer base.
Faced with slowing revenue growth and intense competition, the need to perform analytics on customer transactional data was becoming critical for the Bank. This would allow the Bank to understand and predict the needs of existing and new customers. With this knowledge, the Bank would be able to hyper-personalize the offerings to meet customer needs and improve the odds of conversion.
However, DB2 as the transactional database on the Mainframe was not designed for analytics. Running any kind of analytics on the Mainframes proved to be hugely expensive and slow. Furthermore, rReplacing the Mainframes was not an option. The Bank decided A decision was made to offload analytics to a dedicated computing environment using real-time data pipelines.
Furthermore, replacing the Mainframes was not an option. The Bank decided to offload analytics to a dedicated computing environment using real-time data pipelines.
Across all the business units, the Bank was processing 50 million updates/day. There were more than 200 core tables across multiple Mainframes that needed to be replicated to the target with a latency of 5 seconds or less. The source being DB2 on Mainframe, the strict requirement of uptime of 9x9s, and the huge volume of transactions across multiple business units limited the number of tools that could meet these requirements.
The Bank’s team had several requirements for a tool providing real-time data pipelines:
Zero impact to source
Use an IBM supported method to extract transactions from the Mainframe
Maintain the same order as in source when replaying the transactions on target to guarantee strong consistency (a must for the Bank)
Guarantee 5x9s availability for the pipelines
Robust error handling with automatic recovery
The team selected Arcion after evaluating several tools because it met all their requirements.
The Bank was initially reluctant to adopt the tool. However, Arcion’s willingness to partner with the Bank to ensure a successful outcome impressed them. The Arcion team had more than 140 years of combined experience with mission-critical transactional systems and real-time pipelines, which provided assurance that their product would work well.
With their two-person team, the Bank started out by capturing and replicating data with one Business Unit. This business unit was able to process 1M updates/day across 10 core tables.
With IIDR, changes were captured from the Mainframe and written out to an Event bus. Arcion read and processed the events in real-time, rebuilt the transactions, and applied them in the right order on the Targets. Arcion was deployed in a distributed manner, guaranteeing scale and 5x9s availability.
Several rounds of rigorous testing proved that the transactions were applied accurately 100% of the time and recovery from failures was automatic with zero data loss.
The solution was deployed over several months. With just one data pipeline per table and with end-to-end latency under 2 seconds (DB2 -> CDC -> IIDR -> Kafka -> Replicate XA -> 3 singlestore clusters), changes were captured and correctly applied to 3 geographically separated target clusters.
The Bank was able to accomplish their goals of analyzing customer transactions in real-time without any impact to the Mainframes.
Join our newsletter
Take Arcion for a Spin
Deploy the only cloud-native data replication platform you’ll ever need. Get real-time, high-performance data pipelines today.