Zero impact data pipelines from Mainframe Systems to a modern computing environment

The Bank had flattened revenue growth and a declining customer base over several years. For a Bank to grow and thrive, adding new customers and selling new services to existing customers is vital.

An additional challenge for the Bank was the deregulation of the Banking industry, which  resulted in the introduction of many new players competing for the same customer base. 

Faced with slowing revenue growth and intense competition, the need to perform analytics on customer transactional data was becoming critical for the Bank. This would allow the Bank to understand and predict the needs of existing and new customers. With this knowledge, the Bank would be able to hyper-personalize the offerings to meet customer needs and improve the odds of conversion.

However, DB2 as the transactional database on the Mainframe was not designed for analytics. Running any kind of analytics on the Mainframes proved to be hugely expensive and slow.  Furthermore, replacing the Mainframes was not an option. The Bank decided to offload analytics to a dedicated computing environment using real-time data pipelines.

Challenge

Across all the business units, the Bank was processing 50 million updates/day. There were more than 200 core tables across multiple Mainframes that needed to be replicated to the target with a latency of 5 seconds or less. The source being DB2 on Mainframe, the strict requirement of uptime of 9x9s, and the huge volume of transactions across multiple business units limited the number of tools that could meet these requirements.

Solution

The Bank’s team had several requirements for a tool providing real-time data pipelines: 

• Zero impact to source
• Use an IBM supported method to extract transactions from the Mainframe
• Maintain the same order as in source when replaying the transactions on target to guarantee strong consistency (a must for the Bank)
• Guarantee 5x9s availability for the pipelines
• Robust error handling with automatic recovery

The team selected Arcion after evaluating several tools because it met all their requirements.  

The Bank was initially reluctant to adopt the tool. However, Arcion’s willingness to partner with the Bank to ensure a successful outcome impressed them. The Arcion team had more than 140 years of combined experience with mission-critical transactional systems and real-time pipelines, which provided assurance that their product would work well.

Results

With their two-person team, the Bank started out by capturing and replicating data with one Business Unit. This business unit was able to process 1M updates/day across 10 core tables. 

With IBM Data Replication (IIDR), changes were captured from the Mainframe and written out to an Event bus. Arcion read and processed the events in real-time, rebuilt the transactions, and applied them in the right order on the Targets. Arcion was deployed in a distributed manner, guaranteeing scale and 5x9s availability. 

Several rounds of rigorous testing proved that the transactions were applied accurately 100% of the time and recovery from failures was automatic with zero data loss. 

The solution was deployed over several months. With just one data pipeline per table and with end-to-end latency under 2 seconds (DB2 -> CDC -> IIDR -> Kafka -> Replicate XA -> 3 SingleStore clusters), changes were captured and correctly applied to 3 geographically separated target clusters.

The Bank was able to accomplish their goals of analyzing customer transactions in real-time without any impact to the Mainframes.