Streaming table changes to Kafka

Six Stars

Streaming table changes to Kafka

Hi,

 

I'm trying to capture SQL Server table changes in real-time and send them to a Kafka topic.

 

I know that I can use a CDC table and connect it to Kafka directly (CDC table -> Kafka), but is there another alternative using Talend (something like CDC table -> Talend -> Kafka, etc)?

 

I've tried using Talend's CDC feature, but it seems that it only works for batch jobs and not real-time... Do correct me if I'm wrong though.

 

 

Regards,

Veronica

 

Thirteen Stars

Re: Streaming table changes to Kafka

Hi,

 

any solution would be not a real-time (unless you not design sending a message direct from triggers), all will be - loop by schedule or infinite loop

so, you could run Talend job in loop - 1 sec delay or less

 

or you could check in internet SQL CDC to Kafka and find few other solution for read SQL Server CDC tables, but it still will be not a real real-time

-----------
Six Stars

Re: Streaming table changes to Kafka

Hi,

 

Yes, I am aware that I can create real-time streams from SQL Server directly to Kafka, but I was wondering whether it's possible to do it in a more user-friendly way with Talend.

 

Thanks for the input!

Thirteen Stars

Re: Streaming table changes to Kafka

as I know - not, all what you can do it schedule job in a loop (schedule or design Job as infinite loop)

with some other tools it same - you must define query interval

-----------

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.