Parsing multiple input JSON files

One Star

Parsing multiple input JSON files

Hello All,
We have a requirement to read data from multiple json files stored at a particular location and transform all data and finally insert it to a postgre db. 
I am using Talend tFileInputJSON component to read json file, but i can read only 1 JSON file. 
Since i am fairly new to Talend, looking for experts opinion here on how to tackle this situation. 
Any support appriciated.
Thanks,
Manesh
Five Stars

Re: Parsing multiple input JSON files

Welcome to Talend! 
follow the instructions to get your job done. 
Your job design should be like this. 
tFileList--iterate--tFileinputJson---tMap--youroutput.

Add tfileList component, set the directory path where all files stored. 
Add file mask as per your file names default with "*.*". 
Connect tFileList to tFileInputJson with "iterate" flow. 
you will have " file name with path" in tFileList global variables. add this global variable to tFileInputJosn file path text box. 

global variable which store the file name with path. 
((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
One Star

Re: Parsing multiple input JSON files

Thanks for your help Umesh.. Multiple file parsing Worked perfectly !
One question, can tMap hold data from all the json files before inserting into db or do i need to insert one by one ?
Thanks,
Manesh
Five Stars

Re: Parsing multiple input JSON files

use tBufferoutput and input component to hold the result in buffer then insert this result in db. 
One Star

Re: Parsing multiple input JSON files

It's often easier to convert the JSON to a CSV flat file first then import that into tables.
There are some free converters like json-csv.com
One Star

Re: Parsing multiple input JSON files

Thanks Umesh and Holga for your inputs. I will try to follow it. Basically my JSON has few nested loops and i need to put some custom logic before inserting in database,so converting into csv may not help my purpose. 
Five Stars

Re: Parsing multiple input JSON files

if you share input json file, then we can suggest better solution. 
One Star

Re: Parsing multiple input JSON files

Thanks Umesh,

Below is my sample json file with very few fields, 
{"prod_code":"2568945","default_sku_cd":"010101001","sku_list":,"color_details_list":,"size_details_list":,"length_details_list":,"price":"12.00",}

So here I want to achieve 2 things,
1st Job : read each file like above and Insert sku_list row by row into db, but before that i need to map fields like color_code in color_details_list array to get respective color name or if any field is null then replace by "-" before inserting record. 
2nd Job : Insert level 1 fields (ex. prod_code,default_sku_cd,price etc) but before that loop through sku_list  for few details (ex. sku_price,color_code,color_name) for sku_code matching with default_sku_cd and then add all these details as  a single row. 
Since i am fairly new to Talend, unable to decide what approach to take to find solution for above problems. Any directions, help is highly appreciated. 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download