Need batch wise data pull from source (tFileInputDelimited) in TOS 6.0

One Star

Need batch wise data pull from source (tFileInputDelimited) in TOS 6.0

My requirement is I need to fetch every 1000 records from CSV file/ tNetsuite ERP and move to elastic search as batches in specific interval times. I didn't find any component to solve my requirement. In TOS 6.0 we have a connector tMySqlInput. In this connector in advance settings we have "Enable Stream" with check box. If we select the checkbox, we can utilize the batch wise data to output file or tMysqlOutput. 
1. How can I see the code for tMysqlInput?
2. Is there any other component that will solve my requirement? If yes, please share the link.
3. Is there a way to design the Custom component?
Please let me know if need more information.

Community Manager

Re: Need batch wise data pull from source (tFileInputDelimited) in TOS 6.0

Hi 
1. Click on the Code tab aside the Designer, you will see the java code of job.
2. If the data is stored in a CSV file, you can use tFileInputDelimited, I don't understand you use a tMysqlInput component, can you please explain your requirement a bit?
3. Yes, you can develop custom components, refer to the following article:
https://help.talend.com/pages/viewpage.action?pageId=189661214

Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Need batch wise data pull from source (tFileInputDelimited) in TOS 6.0

Thanks a lot Shong. I got solution from your reply on custom component.
My requirement is I need to fetch only 10,000 records per day from Netsuite ERP using tNetsuiteInput connector and store it in Elasticsearch using tElasticsearchIndex component in TOS 6.0 as Json format. How can I do that, please suggest.
Community Manager

Re: Need batch wise data pull from source (tFileInputDelimited) in TOS 6.0

Sorry,  I am not family with Elasticsearch, the tElasticsearchIndex component is not documented yet, so I can't give you some suggestion right now.;(
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Need batch wise data pull from source (tFileInputDelimited) in TOS 6.0

For TFileInputDelimited we have limit option in advance settings. we can provide the input as 1000 or required number of records to fetch. It got resolved 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog