Dynamic Schemas for JSON

One Star

Dynamic Schemas for JSON

Hi,
I am developing an ETL job whereby a variety of different JSON schemas are converted to RDBMS. Each key in the JSON is mapped 1 to 1 to a column in a table. Each JSON schema has about 5 columns that are the same across all schemas, the rest are different. Right now, I am manually creating a JSON file in metadata for each schema, and manually mapping it to the corresponding table. However, in a few months, the amount of different JSON schemas received will increase exponentially, so manually creating a metadata file and mapping for each schema becomes unfeasible.
Is there a way I can build a job that generically takes any JSON schema I throw at it, and automatically maps the value to a column in a table (where the name of a column = the JSON key), maybe using tSetDynamicSchema, or write/extract dynamic fields?
Thanks,
Nilan
edit: Just to add, each different type of schema goes into a different table. The data from the JSON defines what the target table is, which is currently achieved by setting the relevant key/value from the JSON as a global variable, then using that variable as the table name in tMySQL input.
edit2: I have also seen the page: https://help.talend.com/pages/viewpage.action?pageId=5671283 which shows how to deal with changing data structures. However the dynamic data type isn't supported by tFileInputJSON, so there may be a workaround for this?
One Star

Re: Dynamic Schemas for JSON

bump
One Star

Re: Dynamic Schemas for JSON

bump
facing the same issue
Moderator

Re: Dynamic Schemas for JSON

Hi,
Please open a feature jira issue of DI project for dynamic schema requirement in json components on Talend Bug Tracker.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Dynamic Schemas for JSON

Hi All,
This issue seems pretty old. And almost none of the links are working the communication.
I'm also facing the same issue. Could anyone suggest some solution to it..?
Regards,
Mitul
Five Stars

Re: Dynamic Schemas for JSON

Hi All,
This issue seems pretty old. And almost none of the links are working the communication.
I'm also facing the same issue. Could anyone suggest some solution to it..?
Regards,
Mitul
Five Stars

Re: Dynamic Schemas for JSON

Hi All,
Elaborating the issue..
Source System : CouchDB - JSON format..
Target System : SQL Server..
There is another application running on top of the ETL system, Which generates the required views over the couchDB, as well as, creates the table to the SQL server and then triggers the ETL job.
Kindly suggest some solution.
Regards,
Mitul
Five Stars

Re: Dynamic Schemas for JSON

Adding to that.. I'm using Talend Sandbox 6.2.. There, for tMSSQLinput component I could not find the datatype as dynamic as well..
Please advise the solution.
Regards,
Mitul Vyas
Five Stars

Re: Dynamic Schemas for JSON

Hi,
Could you please respond?
Regards.
Mitul
Five Stars

Re: Dynamic Schemas for JSON

Hi, Could You please Respond?
Thanks.
Mitul