I'm quite new to Talend however i got the assignment to make a system that can dynamicly read multiple, different and varying Json schema files. these then have to be mapped together and pushed to our generic database.
Can anyone confirm this is possible and if so how ?
any input is appriciated, thanks
In order to do pretty much anything with JSON files, and certainly any processing, you'll need to know what the full schema would be, even if some of those fields won't always be present.
If you have this, then the tFileInputJSON component will allow you to read in the file and create a flow of data, which can then be manipulated as required using Talend's range of components.
The tFileInputJSON component does have a bit of a learning curve, but is both powerful and flexible, and you'll almost certainly want to set up each different JSON file format as metadata, in order to use the wizard, rather than configuring the components manually.
If there is no schema available, then you're really going to hit a brick wall no matter which tool you're using, unless of course you can find something designed specifically for what you have in mind, and I'm not aware of anything like this.
It may be possible to manipulate and merge the contents of the raw files, but this is unlikely to be robust, and depending on the structure of files, may not even be an option.
You mention that you'll be pushing the data to a database. Does this have a fixed schema to which the JSON data needs to be mapped, or are you just planning to put the JSON into a single text field? If your target database has a fixed schema, then you'll definitely need to know the JSON schema for your files.
If you could provide test examples of the JSON files you'll be dealing with, and maybe the destination database schema, then hopefully we'll be able to suggest a suitable approach.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema