Product Release Notes

Main feature updates:

- BigQuery pipeline

- Kafka pipeline

- Confluent pipeline- /dev/null destination support

- Databricks Lakehouse UI rework & Azure/S3 support

Detail fixes:

- Pipeline destination - BigQuery

- Pipeline destination - Kafka

- Pipeline destination - Confluent

- Support /dev/null destination for extractor performance testing

- Allow download of full Replicant trace log

- Allow disabling IPv6 in container

- Cannot save YugabyteSQL applier

- Disable role creation as default for all replication

- Configuration edit not working properly

- Missing restart for finished replications

- Oracle to Big Query Fails due to connection issue

- Google Big Query Applier - Replay Strategy Not Being Recorded

- Query reply-strategy not being printed

- PostgreSQL default port listed as hostname

- Oracle Applier unable to save

- Oracle roles still being fetched even when unchecked in extractor

- Cloud wording on front screen

- Catalog name undefined in the per table target configuration

- Pre-populate default port for all connectors

- Add IN_MEMORY_MERGE applier support

- Add ability to skip row counts to extractor

- Update connection with unsaved changes on test

- Databricks applier - save-file-on-error - default to false

- Add maxFilesPerBulkLoad to applier

- Add Azure and S3 support to Lakehouse

- Realtime replication mode support + MySQL cursor support

- Connector list - alphabetical sort

- MySQL extractor - expose idleBinLogTimeoutSeconds in realtime

Main feature updates:

- BigQuery pipeline

- Kafka pipeline

- Confluent pipeline- /dev/null destination support

- Databricks Lakehouse UI rework & Azure/S3 support

Detail fixes:

- Pipeline destination - BigQuery

- Pipeline destination - Kafka

- Pipeline destination - Confluent

- Support /dev/null destination for extractor performance testing

- Allow download of full Replicant trace log

- Allow disabling IPv6 in container

- Cannot save YugabyteSQL applier

- Disable role creation as default for all replication

- Configuration edit not working properly

- Missing restart for finished replications

- Oracle to Big Query Fails due to connection issue

- Google Big Query Applier - Replay Strategy Not Being Recorded

- Query reply-strategy not being printed

- PostgreSQL default port listed as hostname

- Oracle Applier unable to save

- Oracle roles still being fetched even when unchecked in extractor

- Cloud wording on front screen

- Catalog name undefined in the per table target configuration

- Pre-populate default port for all connectors

- Add IN_MEMORY_MERGE applier support

- Add ability to skip row counts to extractor

- Update connection with unsaved changes on test

- Databricks applier - save-file-on-error - default to false

- Add maxFilesPerBulkLoad to applier

- Add Azure and S3 support to Lakehouse

- Realtime replication mode support + MySQL cursor support

- Connector list - alphabetical sort

- MySQL extractor - expose idleBinLogTimeoutSeconds in realtime

Fix included:

1. Connect to oracle using local tns file

2. Changed build.gradle repository order

3. Fixing SnapshotRecovery after changes introduced by a new MR

4. Native Transform Timestamp Less Than

5. BigQuery set realtime replay strategy as AUTO

6. Do not consider blocked columns from the validation of FKs

7. SQL Server Log based extractor threading issue

8. SQServer --> RedisStream Accumulated Insert Logs during snapshot comes as Insert Oper instead of Replace

9. DB2 LUW Log Reader Handle BINARY Data Type

10. Redis Stream : Code cleanup

11. MongoDB realtime mode error when start position is mentioned

12. Add DB2 LUW as source for Debezium format messages

13. Add source extraction timestamp and operation as new columns to target tables

14. Add liveliness memory stats to metadatadb and stats file

15. getClusteringKeySql throws exception if Snowflake is used as source

16. Credential store support for Databricks

17. Support DefaultCredentialsProvider for authentication on S3_STORAGE_BROKER.

18. Db2/MQ/Kafka->Singlestore --continue-inconsistent-post-failure is not working for dst-query failures

19. Test and Document SSL connection to SAP Sybase ASE

20. Refactor fencing workflow for producer replication in the streaming storage setup

21. Error Handling - SingleStore/Mysql Target

22. Add and test SSL connection support to SingleStore

23. Sync CDC data stream through S3 with a independent snapshot migration

24. Db2 AS400 Table containing Column type as Time is throwing exception

25. Postgresql(source) DDL support

26. Oracle: Provide Alternative to Deletion Behavior Upon Resume

27. Extract exceptions used in forms to standalone classes

28. YamlUtil InputStream load methods changed to accept TriFunction that returns Composer implementation

29. FormUtils.validateNamespace() logic moved to standalone class NamespaceValidator

30. PG-DDL inline created table are having string data type for all columns

31. Oracle as a target results in exception when source isn't also Oracle

32. SQL Server to Oracle DEFAULT CURRENT_TIMESTAMP cannot be translated

33. Fix Databricks connection URL, when the password contains special characters

34. Fix Oracle source CDC for inserting rowid datatype

35. Add Minio support

36. Fix NPE in MysqlRealtimeExtractorForm due to binLogIdleTimeoutS

37. Db2 luw decfloat result in parsing issue

- Oracle to Oracle pipeline

- SQL Server CDC agent support

- Databricks Deltalake UI rework & Azure/DBFS support

- Whole SaaS stack migrated to Java 17

- Computed columns support for MySQL

- Forward test-connection trace log to UI in case of Core error

- Add Azure stage support to Databricks Delta Lake

- Missing ‘restart’ option for stopped replications

- Increase Replicant memory limit

- Update Sign In & Sign Up pages

- Hung on current screen (loading replication data)

- Connection details not opening

- Connection created successfully duplicated

- Replication statuses phasing in and out

- Connection test does not accurately represent what is needed for replication to succeed

- Databricks connection not working with or without stage

- Table filter search not working

- Schema not being refreshed after fetchSchema

- Update the current deadlock solution

- Fix tooltip overflow issues

- Update connection logo image files to fit to container box

- Generated columns support

- Applier BulkLoad charset support

- Increase maximum YAML schema input size

- Management service port issues on WSL2

- Snapshot applier additions - skip table failures and init indexes

- Enable APPENDING write mode in UI

- Oracle to Oracle pipeline

- SQL Server CDC agent support

- Databricks Deltalake UI rework & Azure/DBFS support

- Whole SaaS stack migrated to Java 17

- Computed columns support for MySQL

- Forward test-connection trace log to UI in case of Core error

- Add Azure stage support to Databricks Delta Lake

- Missing 'restart' option for stopped replications

- Increase Replicant memory limit

- Update Sign In & Sign Up pages

- Hung on current screen (loading replication data)

- Connection details not opening

- Connection created successfully duplicated

- Replication statuses phasing in and out

- Connection test does not accurately represent what is needed for replication to succeed

- Databricks connection not working with or without stage

- Table filter search not working

- Schema not being refreshed after fetchSchema

- Update the current deadlock solution

- Fix tooltip overflow issues

- Update connection logo image files to fit to container box

- Generated columns support

- Applier BulkLoad charset support

- Increase maximum YAML schema input size

- Management service port issues on WSL2

- Snapshot applier additions - skip table failures and init indexes

- Enable APPENDING write mode in UI

- Core 23.02.01.9

- MySQL -> MySQL pipeline

- Dashboard rework with filtering by mode and virtual list support (to allow smooth scroll with huge numbers of replications)

- Redesign of the settings menu, various UI fixes

- Support for paid cluster pause/resume/terminate

- AWS password rotation support in the Docker container (both for Metadata DB and replications)

- S3 as the shared config storage for clustered mode

- In addition to the latest Docker tag we now have the test tag - which will point to the next container version with the latest core release (for PoC etc.)

- Core 23.02.01.9

- MySQL -> MySQL pipeline

- Dashboard rework with filtering by mode and virtual list support (to allow smooth scroll with huge numbers of replications)

- Redesign of the settings menu, various UI fixes

- Support for paid cluster pause/resume/terminate

- AWS password rotation support in the Docker container (both for Metadata DB and replications)

- S3 as the shared config storage for clustered mode

- In addition to the latest Docker tag we now have the test tag - which will point to the next container version with the latest core release (for PoC etc.)

- Core 23.01.05.9- Improvements to our on-premises APIs

- Configuration YAML upload fixed

- Expose information about cluster state

- Refresh connection logos with vector ones

- Support for Oracle native log reader (ASM only on Cloud, Redo Logpath additionally for on-premises)

- Audit logging support for user actions and events

- Pass license into Docker as Base64 environment variable

- Support ARM64 architecture for on-premises container image (look for tag latest-arm64 on DockerHub)

- Show days left on license for on-premises

- Support SQL Server Azure as source

- Various UI improvements and bug fixes

- Core 23.01.05.9- Improvements to our on-premises APIs

- Configuration YAML upload fixed

- Expose information about cluster state

- Refresh connection logos with vector ones

- Support for Oracle native log reader (ASM only on Cloud, Redo Logpath additionally for on-premises)

- Audit logging support for user actions and events

- Pass license into Docker as Base64 environment variable

- Support ARM64 architecture for on-premises container image (look for tag latest-arm64 on DockerHub)

- Show days left on license for on-premises

- Support SQL Server Azure as source

- Various UI improvements and bug fixes

Fixes included:

1. Generalize the Oracle extractor throttle.

2. Replicant Crashes for Delete operation in delta mode when Applier is configured in Non-BulkLoad Mode.

3. Configure mode - run into error in new version.

4. Error in Postgresql to Postgresql table creation for tables with default values.

5. MongoDB delete not working.

6. Duplicate PK issue in certain idempotent scenario.

7. Databricks connection file with AWS secrets - failing on recent versions.

8. Test and document ssl connection to Snowflake.

9. Test and document ssl connection to BigQuery.

10. Delta-snapshot mode throws NPE when resuming replication containing compound columns.

11. SQL Server datetime columns with CURRENT_TIMESTAMP default gets created incorrectly on MySQL.

12. Allow specifying metadata config for Redis Stream as target.

13. Replicant fails to start PG->PG due to DEFAULT column value syntax error.

14. Test and Document SSL connection to Databricks.

15. Applier wrong type resolved on XA form loading resulting in parse error.

16. Db2/MQ/Kafka Blocked Columns Causing Parsing Error.

17. Introduce IRealtimeOperationSerializer interface to be used for de/serialization per different CDC format (Kryo/JSON).

18. DB2/MQ/Kafka Db2DelimitedMsgParser Error On IBM Error Messages.

19. NPE in SQL Server fetch-schmas.

20. Move create-table-without-defaults-on-failure option to applier.

21. Moved static logic from DBSchemasForm into SchemasFormUtils and DBSchemasFormParser classes.

22. In config forms, changed visibility of validate method to public and some constructors as well.

23. Moved BaseContext instance holder from ReplicationContext to BaseContext.

24. SecretManager.initializeSecretManager() changed method call to accept vars instead of GeneralConfig.

25. Decoupled parse method from FilterForm/MapperForm/ProcForm/NativeLoadConfigForm/ObjectIdForm/PathForm classes into standalone class hierarchy. Made required modifications to DBMetatataProvider/DBMetadataProviderShared classes.

26. Improve Debezium format data representation.

27. Config to avoid of overlap of tables during extraction.

28. Support bigint data type to Kafka.

29: PostgreSQL - Change default value for connection config log-reader-type.

30. Fix Databricks job-wide snapshot recovery.

31. Databricks - Fix concurrent copy failure.

1. MySQL: support  GENERATED ALWAYS AS

2. In duplicate Pk scenario (parent issue), detect and handle the error by deleting the original row

3. NPE in SQL Server

4. PostgreSQL to Kafka CDC - Null Pointer Exception

5. Additional Error Handling Options for Kafka

6. Support configuring a custom NTP server in general configuration

7. Add --terminate-post-cdc-init flag for full mode replication

8.MongoDB to BigQuery CDC

9. Add CLI dashboard for the maintenance process

10. SQL Server computed columns should be handled more intelligently

11. Testing reinit based DDL replication for CDC source Informix

12. Embed functionality of --clean-stage to the --overwrite flag

13. Renaming parts of the setup

14. Fix fetch-schema for Redis Stream and Kafka as target

15. PostgreSQL: CREATE INDEX fails when using mapper file

16. Index creation attempted when column in block: [ ] list

17. Feature Request - Yugabyte Table split clause

18. Redis Stream and Kafka  ( JSON Format )

19. DBSnapshotExtractorParser wrong class call.

20. Remove OPER_TYPE as PK from delta table

21. Resume replication failing due to replicationcontext not initialized

22. Set the default value retry-transactions to true only for ACID destinations

23. Deprecate and disable databricks _optimize-snapshot by default

24. Mongo Pipeline breaking due to version change

25. Kafka as target : Override snapshot config num-shards = 1 when...

26. Fix Azure Databricks when optimizesnapshot is disabled

27. Add SAS token support for Azure stage.

28. Oracle -> Oracle Add missing recovery key columns in case of fetch-create-sql enabled.

29. Incorporating extractor config computed-columns to support generated columns for MySQL and fixing snapshot of generated columns.

30. Set the blocking option for generated columns to false automatically when using --create-sql/fetch-create-sql.

‍

1. DB2-MQ-Kafka to S2 - Duplicate entry for key 'PRIMARY'

2. Bad SQL cause ORA-00913: too many values

3. DB2-MQ-Kafka to S2 - Update failing with duplicate key error after successful insert

4. Fix parsing of connection stage config

5. Support XML

6. Support multiple replay strategy for BigQuery as target

7. Support MySQL RDS password rotation (Chegg)

8. Support auto-reinit if DDL fails

9. Upgrade S3 SDK version to 2.*

10. MySQL index creation error

11. Option to add prefix to topic

12. Db2 Kafka - Kafka logging improvements

13. Option to allow dot(.) as a delimiter in Kafka topic names

14. Fix system generated split-key and delta-snapshot-key handling for HANA as source

15. Initialize ASE source fields in target JSON format log ( for Kafka and Redis Stream )

16. Transforms Not Applying When Src-schemas is used

17. Log Delta Table Verbose

18. AWS SecretsManager - error parsing secretKeyPath with /

19. Setting csv-publish-method: READ as default when column transformation is used

20. DB2-MQ-Kafka - Update on timestamp column not working

21. Enable compression-type: GZIP for MemSQL/SingleStore bulk load

22. Redis Stream : Improve logging under snapshot config log-row-level-errors

23. Fix NPE in GLOBAL replay consistency validation code

24. Fix issue where we only allow connecting to a named instance by port

25. Fix for SQL Server change tracking and re-enable CI/CD tests for SQL Server

26. Fixed SQLServer fetchObjectIds() function, using schema name to filter tables from SQLServer

1. Add Voodoo LogReader as extraction method for Oracle

2. Redis Streams as a Target

3. Databricks as target : MERGE-Based Strategy ( When we don't have full before/after image )

4. Test Snowflake Iceberg Tables

5. Improve cdc log segment cache

6. Db2 As400 Add logging for commitment control

7. AS400 Support Transaction Rollback

8. Support realtime BigQuery as target using storage Write API

9. Support reinit based DDL Replication for SAP ASE Sybase Source

10. Added support for delta-snapshot both src and dst for SQL Server.

‍

1. Add replication ID to the transaction path for Oracle uncommitted CDC

2. Support Writing All Data to a Single Kafka Topic (From Informix and Postgres)

3. Integration with AWS Secrets Manager (or Similar Service)

4. Write to CDC log segment in chunks

5. Add stage maintenance mode

6. NPE in Snowflake initializer

7. SQL Server incorrectly blocking system schemas

8. --stop-never-slave-server-id param in MySQL is deprecated.

9. RDS PostgreSQL to RDS Postgres - replicate_io_replication_stats_summary_coreoct11_12_coreoct11_12 does not exist

10. Handle error: ORA-02149 Specified partition does not exist

11. SQLServer - fetch-schema not fetching the database specified in connection configuration

12. Test and fix source column transformation

13. MySQL as Source in real-time replication crashes when we insert into column of type year

14. Azure SQL to YugabyteSQL - Column default does not created at target

15. Databricks S3 Role arn parameter

16. Fix for cdc recovery across all pipelines

17. Ability to Surface Alerts in New Relic

18. Enable init-metadata mode for DB2

19. Fix incorrect exception when logging failed transaction

20. DB2 AS400 tables with table name length greater than 10, realtime replication is not working

21. SQL Server heartbeat doesn't work when there are multiple databases being replicated

22. Add a table level check in Oracle for full column logging when Databricks is a target

23. Table level column logging check not working

24. Sybase ASE snapshot but on release binary 22 10 28 1

Features:

1. Separate transaction files by replication ID for SQL Server CDC

2. Oracle Native Export

3. Databricks Unity catalog support

4. Allow use of Salesforce ID or Id Mod Column as Split Key

5. Add support for a credential store for the Replicant SQL Agent

6. GCP Databricks as destination

‍

Fixes:

1. Db2 to Singlestore snapshot, TIMESTAMP column is shown as TIMESTMP in fetch-schema and is converted to varchar in singlestore

2. SQL Server CDC incorrectly creates multiple cursors for schemas within a database

3. Imply does not drop existing table in reinit mode if table already present

4. Change active connection logging to include TPT connections

5. Oracle to Oracle: Real-time SCN problem.

6. Add support for a credential store for the Replicant SQL Agent

7. SQL CDC should send ACKs for each file transfer

8. Decouple exit codes from custom exceptions

9. Upgrade Databricks jdbc driver

10. GCP Databricks as destination

11. Db2 LUW/As400 Losing Records When Transaction Size > 100000

12. PostgreSQL as target doesn't close files after bulk load

‍

Features:

1. SAP ASEas source for realtime/full mode support.

2. OracleDBLINK based snapshot.

3. Support for Kerberos Authentication.

4. Support BigQuery as a target.

5. AddUpsert support for SingleStore.

6. EnableEmail Notifications if Lag is ABOVE a particular threshold value.

Fixes:

1. MariaDB-> Google BigQuery : BigQuery not support AUTO_INCREMENT or IDENTITY.

2. Fixextractor NPE when replicating Oracle views.

3. DB2AS400 Schema with Schema name length greater than 10, realtime replication isnot working.

4. Changing Imply Implementation for their new apis.

5. Fix issues with TiDB as a target identified by PingCAP.

6. Invalidvalue when replicating table with set datatype in realtime mode.

7. ExtendDb2 XML TRANSACTION MQ message type to Kafka-Transactional log-storage.

8. Addsupport for DB2 TRANSACTION message type in DELIMITED message format.

9. Olapduplicate row insert error.

10. Snowflake target improvements.

11. MongoDBconnection needs to be established through url or connection config properties(host, port, username and password).

12. SAP ASECDC Performance testing.

13. Snowflake table cdcStageType not working for insert.

14. Number of rows mismatch while replicating updates ondatabricks using MERGE.

15. TestTargets with ASE CDC Source.

16. fixingper table replay strategy in realtime mode.

17. Snowflake bug fixing for delete query.

18. Snapshot mode not finished when trying to replicate table with SET Type.

‍