Four Stars

Talend Jobscript API

Hello,

 

I want to implement a totally dynamic job in talend.

By dynamic, I mean that I would have a file, which will have the metadata -the source, number of columns, their names, targets, their data transformations etc. I am hoping to use the jobscript API in talend and parametrize all or most of the components. 

 

Questions-

  1. Can we write if else loops and for loops in the job script?
  2. Can the jobscript be used with a context file?
  3. Does jobscript give this sort of flexibility or are there any other suggestions for the same ?

 

Thanks in advance!

2 REPLIES
Thirteen Stars

Re: Talend Jobscript API

Hi Remytom,

 

I have seen a number of your questions and they all seem to be about creating jobs automatically or creating the mythical "one job for all scenarios". It just isn't possible I'm afraid, otherwise the Talend product would just be a button which would do it all for you. Talend sales would clearly love this as they could truthfully say "My grandmother can do it" and there would be a lot of data integration specialists out of work. 

 

However, if you want to use the Jobscript functionality you may be able to automate the creation of jobs with very similar structures. 

 

This link may help in understanding the Jobscript and job creation API (https://help.talend.com/reader/7YfsQ62BmVj8qZKXzineUg/d0QVvShdlGcIqowdsla3lw).

With regard to your questions......

1. You can do with the Jobscript file what you can do by actually manually dropping components onto your work space. You can use Loops and IF conditions in Talend (although not how you would expect them to be written in Java, for example), so yes you can do this. You can also write Java in the JobScript file....but it must be for a Java component.

 

2. You can import contexts from a file using the Implicit Context Load. This can be figured using code very similar to this in the JobScript file....

addParameters {
	addElementParameters {
		JOB_RUN_VM_ARGUMENTS : " -Xms256M -Xmx4096M ",
		JOB_RUN_VM_ARGUMENTS_OPTION : "true",
		SCREEN_OFFSET_X : "0",
		SCREEN_OFFSET_Y : "736",,,,,,,,,,,,,,,,
		IMPLICITCONTEXT_USE_PROJECT_SETTINGS : "true",
		STATANDLOG_USE_PROJECT_SETTINGS : "true",
		MULTI_THREAD_EXECATION : "false",
		PARALLELIZE_UNIT_SIZE : "25000",
		IMPLICIT_TCONTEXTLOAD : "true",
		FROM_FILE_FLAG_IMPLICIT_CONTEXT : "false",
		FROM_DATABASE_FLAG_IMPLICIT_CONTEXT : "true",
		IMPLICIT_TCONTEXTLOAD_FILE : "\"\"",
		FIELDSEPARATOR : "\"\"",
		DB_TYPE_IMPLICIT_CONTEXT : "tMSSqlInput",
		DB_VERSION_IMPLICIT_CONTEXT : "MYSQL_5",
		URL_IMPLICIT_CONTEXT : "\"\"",,
		DRIVER_CLASS_IMPLICIT_CONTEXT : "\"\"",
		HOST_IMPLICIT_CONTEXT :
		"routines.ImplicitContextLoadUtils.getImplicitContextParameterValue(\"TalendContextHost\",rootPid,pid)",
		PORT_IMPLICIT_CONTEXT :
		"routines.ImplicitContextLoadUtils.getImplicitContextParameterValue(\"TalendContextPort\",rootPid,pid)",
		DBNAME_IMPLICIT_CONTEXT :
		"routines.ImplicitContextLoadUtils.getImplicitContextParameterValue(\"TalendContextDbName\",rootPid,pid)",
		LOCAL_SERVICE_NAME_IMPLICIT_CONTEXT : "\"\"",
		PROPERTIES_IMPLICIT_CONTEXT :
		"routines.ImplicitContextLoadUtils.getImplicitContextParameterValue(\"TalendContextAdditionalParams\",rootPid,pid)",
		SCHEMA_DB_IMPLICIT_CONTEXT : "\"\"",
		USER_IMPLICIT_CONTEXT :
		"routines.ImplicitContextLoadUtils.getImplicitContextParameterValue(\"TalendContextUser\",rootPid,pid)",
		PASS_IMPLICIT_CONTEXT :
		"H3dPaI+z82DZMG+nSMylCTAOIpvF93163DlYfsTyanFfPv1zHabbNxAzqKLT4IQJdkwg8STkuezl0lEMFAojWRexu/vfgtM0YVs+7dUgwubICbiDLRr8SMb6meeE+x1pIJVgjkPQgDs=",
		DBFILE_IMPLICIT_CONTEXT : "\"\"",
		DBTABLE_IMPLICIT_CONTEXT : "\"Context_Variables\"",
		QUERY_CONDITION_IMPLICIT_CONTEXT :
		"routines.ImplicitContextLoadUtils.getImplicitContextParameterValue(\"TalendContextEnvironment\",rootPid,pid)",
		LOAD_NEW_VARIABLE : "Warning",
		NOT_LOAD_OLD_VARIABLE : "Warning",
		PRINT_OPERATIONS : "false",
		DISABLE_ERROR : "true",
		DISABLE_WARNINGS : "true",
		DISABLE_INFO : "true",
		ON_STATCATCHER_FLAG : "false",
		ON_LOGCATCHER_FLAG : "false",
		ON_METERCATCHER_FLAG : "false",
		ON_CONSOLE_FLAG : "false",
		ON_FILES_FLAG : "false",
		FILE_PATH :
		"\"C:/Talend/6.1.1/cmdline/studio/commandline-workspace/.metadata\"",
		FILENAME_STATS : "\"stats_file.txt\"",
		FILENAME_LOGS : "\"logs_file.txt\"",
		FILENAME_METTER : "\"meter_file.txt\"",
		ENCODING : "ISO-8859-15",
		ENCODING:ENCODING_TYPE : "ISO-8859-15",
		ON_DATABASE_FLAG : "false",
		DB_TYPE : "tJDBCOutput",
		DB_VERSION : "ORACLE_11",
		URL : "\"\"",,
		DRIVER_CLASS : "\"\"",
		HOST : "\"\"",
		PORT : "\"\"",
		DATASOURCE : "\"\"",
		DBNAME : "\"\"",
		LOCAL_SERVICE_NAME : "\"\"",
		PROPERTIES : "\"\"",
		SCHEMA_DB : "\"\"",
		USER : "\"\"",
		PASS : "0RMsyjmybrE=",
		DBFILE : "\"\"",
		TABLE_STATS : "\"\"",
		TABLE_LOGS : "\"\"",
		TABLE_METER : "\"\"",
		CATCH_RUNTIME_ERRORS : "true",
		CATCH_USER_ERRORS : "true",
		CATCH_USER_WARNING : "true",
		CATCH_REALTIME_STATS : "false",,
		HEADER_ENABLED : "false",,,,,
		FOOTER_ENABLED : "false",,,
	}

You will have to also configure your JobScript with context variables, like below...

addContextParameter {
NAME: BatchControl_BatchStatus,
TYPE: id_Long,
VALUE: "",
COMMENT: "",
PROMPT: "BatchControl_BatchStatus?",
REPOSITORYCONTEXTID: "BatchControl"
}

3. With the JobScript you have the ability to quickly generate jobs *IF* you have learnt the details of building JobScripts *OR* are cribbing from an existing job and simply changing parameters. I would not recommend trying to learn the JobScript code without first learning Talend. If you get that far you may as well build a bare bones job, convert  it to a JobSript base and create something to tweak that.

 

Rilhia Solutions
Four Stars

Re: Talend Jobscript API

Thank you for the response along with the code snippets! Very helpful.