How to efficiently load 30mil rows from Kafka into Postgresql w/TOS?

One Star

How to efficiently load 30mil rows from Kafka into Postgresql w/TOS?

Hi Team,
I have a scenario where i want to load 30million rows of data using Kafka into PostgreSQL. My job is going to be quite straightforward with few filters and tmap. 
I would like to know:
1. What are the best configuration settings for TOS in such scenario?
2. how can i reduce cpu/memory consumptions?
3. what is the most efficient way to load huge data using TOD? Shall i do it in one go or in batches?
4. And also, what could be the average time to load such huge data using TOS?
Thanks in advance.
Rera
Moderator

Re: How to efficiently load 30mil rows from Kafka into Postgresql w/TOS?

Hi Rera,
I have a scenario where i want to load 30million rows of data using Kafka into PostgreSQL. My job is going to be quite straightforward with few filters and tmap.

tMap is cache component consuming much memory. For a large set of data, try to store the data on disk instead of memory.
Here is an option "Use Batch Size" in tPostgresqloutput which is used to activate the batch mode for data processing.
What does your filter look like? What's your current row rate(rows/s)? Is it a normal speed?
Please take a look at document about:TalendHelpCenter:Exception outOfMemory
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now