Shows null values for tExtractJasonFields

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: and @CK395: I have got my json data output for second scenario using java like this:

Date Instructor message student message
Date1 1 2
Date2 1 1
Date 3 2 2

Is it possible to straight away output it into db?
Seven Stars

Re: Shows null values for tExtractJasonFields

Yes you can if your java component has schema defined to be inserted into db

then simply connect your data flow to db_output component and configure the db_output with same schema and valid 

configuration of other values  and you are good to go.

 

 

 

 

Regards

Chandra Kant

Eight Stars

Re: Shows null values for tExtractJasonFields

@CK395 and @rhall_2_0: my java is trying to read from file.json just as a test and formatting data the way I want. But I need to plug in same code in flow: trest client- tjava- db output.

How do I read the output of trest client in java (where my existing code will then manipulate the json format)? Is there any method or property you know? This is where I’m stuck as getting the read from file won’t work in my case as I have got parallelism happening later in the flow. So the best way would be to get output read by tjava.

Please help!
Seven Stars

Re: Shows null values for tExtractJasonFields

hi,

i think tjava cannot provide an output flow to be inserted into db.

you better go for tJavaFlex.

 

 

Regards

Chandra Kant

Eight Stars

Re: Shows null values for tExtractJasonFields

@CK395 and @rhall_2_0: I am really unable to get output read from trest client using javarow or javaflex component.

Currently as a workaround I have got this:

Trest client- toutputjson (creates the output json file) - tjava (which calls routine. This does manipulating of json data and saving into csv file)- tinputdelimeted which will read csv file- output into dB. I’m reluctant to have this workflow as it creates many types of files and given during the process I will create many files. This is not the ideal solution.

Your help is really appreciated if you can show me how to read output result from trest client then that way I don’t have to create json file and instead java component will read rest client output and feed it to csv.

Let me know your thoughts and please share some screenshots.
Highlighted
Sixteen Stars

Re: Shows null values for tExtractJasonFields

What column are you reading from after your tRestClient? It should be the String column, not the Body column.

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: yes I know it is string. But what do I write in javarow or javaflex to get entire output as string or object type from trest client? Then I can manipulate in java code later.
Sixteen Stars

Re: Shows null values for tExtractJasonFields

Connect your tRestClient to a tJavaFlex and use this code (assuming the row is "row1")....

System.out.println(row1.string);

You should see the JSON from the webservice call printed to the console window now. If not, your tRestClient is not returning anything.

 

It might help to see a screenshot of your job AND a screenshot of your tRestClient's output schema. Sometimes the output schema can become corrupt

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: here there sample json data:

{
"2012-01-24":{
"instructorMessages":1,
"studentMessages":2
},
"2012-01-27":{
"studentMessages":1
}
}

And simple job like this:

Sixteen Stars

Re: Shows null values for tExtractJasonFields

This code....

System.out.println(row1.string);

....was to show you how to use the JSON in the tJavaFlex. I don't want to see the JSON. Was what you posted achieved by using the above code? 

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: thanks for your reply. Whatever you have that will just resolve part of my problem where I might reduce one step in job.

Now that I have code, I am going to read as string from trest client then call routine/function(this manipulates whole json data and converts into csv) in javaflex and then from tinput delimited I will read csv file to put it into db.

Let me know your thoughts or is there any way I can improve the job by reducing steps.
Sixteen Stars

Re: Shows null values for tExtractJasonFields

Are you saying that this helped you get hold of the JSON string? I'm not sure I understood what you meant in your post. Do you have a colleague who can help with the Java? I ask because it is very difficult to debug code remotely. It might make more sense to work with a colleague on this to share skills

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: yes that’s correct. I have a colleague who is helping on java. As it’s beyond my capability.

But keeping java code aside, from the talend design do you see anything what I’m trying to achieve? Like any other space for improvement in terms of job design or this is the way you reckon?
Sixteen Stars

Re: Shows null values for tExtractJasonFields

Get the job working and then come back if you have any questions on potential improvements. At the moment, there is nothing to improve.

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: This is how my job looks like:

 

Screen Shot 2018-11-23 at 3.23.29 pm.png

 

Where tJavaflex looks like this:

 

Analytics analyticsAPI = new Analytics();

analyticsAPI.parseJsonData(row2.string, (String) globalMap.get("tempFileURI",student_id,sis_user_id,course_id,course_code));

 

It actually calls routine Analytics and returns the output in .csv file which I am then putting into S3 using PutLadedData. Once this job is successful then another subjob runs and copies all the files from S3 to Redshift using below steps:

 

Screen Shot 2018-11-23 at 3.27.11 pm.png

 

Is there anything I can change or seems correct?

Sixteen Stars

Re: Shows null values for tExtractJasonFields

It looks OK, but if you are running the job that writes the data to a file in parallel, you will have to make sure that it is one file with a unique filename per parallel iteration. You cannot write to the same file in parallel.

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: Thanks for your reply. Yes I’m creating unique file (adding course id and student id in file name) per parallel process.
Sixteen Stars

Re: Shows null values for tExtractJasonFields

If it works, then you're good :-)

Eight Stars

Re: Shows null values for tExtractJasonFields

@rhall_2_0: yes it works fine and this way I’m able to run 25 parallel iterations which is quite good. This reduced my job time from 1 and half hour to 12 min 😀.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.