Zero impact data pipelines from Mainframe Systems to a modern computing environment

Source
Target
distributed
database

The second largest bank in the US relies on IBM Mainframes to validate, record, and retrieve financial transactions for a wide spectrum of  customers. IBM Mainframes are renowned for their resiliency, durability, and security. Mainframes provide unmatched guarantees when it comes to financial transactions. They provide customers with a sense of security and the Bank with a platform that is always available.

Company Highlights
10
top 10 global bank
11%
of all US deposits
46M
customers
Solution
Highlights
1M
updates/day across 10 core tables
100%
accuracy for transactions
< 2 sec
End-to-end latency

The Bank had flattened revenue growth and a declining customer base over several years. For a Bank to grow and thrive, adding new customers and selling new services to existing customers is vital.

An additional challenge for the Bank was the deregulation of the Banking industry, which  resulted in the introduction of many new players competing for the same customer base. 

Faced with slowing revenue growth and intense competition, the need to perform analytics on customer transactional data was becoming critical for the Bank. This would allow the Bank to understand and predict the needs of existing and new customers. With this knowledge, the Bank would be able to hyper-personalize the offerings to meet customer needs and improve the odds of conversion.

However, DB2 as the transactional database on the Mainframe was not designed for analytics. Running any kind of analytics on the Mainframes proved to be hugely expensive and slow.  Furthermore, rReplacing the Mainframes was not an option. The Bank decided A decision was made to offload analytics to a dedicated computing environment using real-time data pipelines.

Furthermore, replacing the Mainframes was not an option. The Bank decided to offload analytics to a dedicated computing environment using real-time data pipelines.

Challenge

Across all the business units, the Bank was processing 50 million updates/day. There were more than 200 core tables across multiple Mainframes that needed to be replicated to the target with a latency of 5 seconds or less. The source being DB2 on Mainframe, the strict requirement of uptime of 9x9s, and the huge volume of transactions across multiple business units limited the number of tools that could meet these requirements.

Solution

The Bank’s team had several requirements for a tool providing real-time data pipelines: 

  • Zero impact to source
  • Use an IBM supported method to extract transactions from the Mainframe
  • Maintain the same order as in source when replaying the transactions on target to guarantee strong consistency (a must for the Bank)
  • Guarantee 5x9s availability for the pipelines
  • Robust error handling with automatic recovery

The team selected Arcion after evaluating several tools because it met all their requirements.  

The Bank was initially reluctant to adopt the tool. However, Arcion’s willingness to partner with the Bank to ensure a successful outcome impressed them. The Arcion team had more than 140 years of combined experience with mission-critical transactional systems and real-time pipelines, which provided assurance that their product would work well.

Results

With their two-person team, the Bank started out by capturing and replicating data with one Business Unit. This business unit was able to process 1M updates/day across 10 core tables. 

With IIDR, changes were captured from the Mainframe and written out to an Event bus. Arcion read and processed the events in real-time, rebuilt the transactions, and applied them in the right order on the Targets. Arcion was deployed in a distributed manner, guaranteeing scale and 5x9s availability. 

Several rounds of rigorous testing proved that the transactions were applied accurately 100% of the time and recovery from failures was automatic with zero data loss. 

The solution was deployed over several months. With just one data pipeline per table and with end-to-end latency under 2 seconds (DB2 -> CDC -> IIDR -> Kafka -> Replicate XA -> 3 singlestore clusters), changes were captured and correctly applied to 3 geographically separated target clusters.

The Bank was able to accomplish their goals of analyzing customer transactions in real-time without any impact to the Mainframes.

Join our newsletter

Take Arcion for a Spin

Deploy the only cloud-native data replication platform you’ll ever need. Get real-time, high-performance data pipelines today.

Get started for free

5 connectors: Oracle, MySQL, Databricks, Snowflake, SingleStore

Pre-configured enterprise instance

Available in four US AWS regions

Free download

20+ enterprise source and target connectors

Deploy on-prem or VPC

Satisfy security requirements

Join the waitlist for Arcion Cloud (beta)

Fully managed, in the cloud.

Start your 30-day free trial with Arcion self-hosted edition

Self managed, wherever you want it.

Please use a valid email so we can send you the trial license.