OpenLogReplicator – first log-based open source Oracle to Kafka CDC replication

I have figured out that writing about somebody else’s software is boring. Why not create your own.

So here it is:

• All code GPL v.3
• Just C++ code – absolutely no Java/Python or any other interpreted language
• Purely reading Oracle Redo Log from disk – zero additional load to the database instance
• High performance architecture from the beginning – no lazy slow code
• Minimum latency approac
• Memory-based approach – no storing of intermediate files on disk

Currently is implemented:

• Possible to compile for Linux x/64 only
• Supported only Oracle 11.2.0.4
• Replicated Redo Codes: just single row Insert (OpCode 11.2)
• Supported full transactionality: begin, commit, rollback, savepoint
• Supported types: numeric, char, varchar2, timestamp, date
  

If I have more time, more documentation will appear here.

How to run?

git clone https://github.com/bersler/OpenLogReplicator
vi Debug/makefile
make
cp OpenLogReplicator.json.example OpenLogReplicator.json
vi OpenLogReplicator.json
./OpenLogReplicator

Make sure that you set the proper paths for needed dependencies for libraries:

• Oracle client 11.2.0.4
• RapidJson library
• The Apache Kafka C/C++ library
  

Make sure the JSON file contains proper information. Everything is very easy and logical.

For who are inpatient and don’t want to compile – here is release 0.0.3 compiled for Linux x64. To execute it run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib
Debug/OpenLogReplicator

Have fun, but please do not send me any complains about not working code. I will maybe write here some help&docs when I have time. I could of course add more functionality, but I didn’t have time. You have the code – you can do it by yourself!

Sample input is:

create table adam3(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
insert into adam3 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;

In Kafka you should have:

{"scn": "4856388", dml: [{"operation":"insert", "table": "SYSTEM.ADAM3", "after": {"A": "100", "B": "999", "C": "10.22", "D": "xxx       ", "E": "yyy", "F": "2018-12-10T21:27:42"}}]}

Please have in mind that only single row insert operations are now supported. insert .. select, insert append, etc. would not work. Just 1-row INSERT operations.