tFileList does not clean his global variables

Highlighted
Seven Stars

tFileList does not clean his global variables

Scenario: a loop on dates (for example) and within any iteration a tFileList component look for specific files.

Well, when the component does not find any file, the global variables (in particular tFileList_2_CURRENT_FILEPATH) are not cleaned.

 

the bold line you see below is a:

System.out.println(globalMap.get("tFileList_2_CURRENT_FILEPATH"));

performed by a tJava component which follows the tFileList component itself.

 

 

...

...

...

[DEBUG]: xxxxx - tFileList_2 - Start to work.
[DEBUG]: xxxxx - tFileList_2 - ParametersSmiley Very HappyIRECTORY = context.transport_archive_dir+((String)globalMap.get("application_key"))+".log.archived/" | LIST_MODE = FILES | INCLUDSUBDIR = false | CASE_SENSITIVE = YES | ERROR = false | GLOBEXPRESSIONS = true | FILES = [{FILEMASK=((String)globalMap.get("application_key"))+".log."+(new SimpleDateFormat("yyyy-MM-dd")).format(globalMap.get("date"))+".gz"}] | ORDER_BY_NOTHING = true | ORDER_BY_FILENAME = false | ORDER_BY_FILESIZE = false | ORDER_BY_MODIFIEDDATE = false | ORDER_ACTION_ASC = true | ORDER_ACTION_DESC = false | IFEXCLUDE = false | FORMAT_FILEPATH_TO_SLASH = false |
[INFO ]: xxxxx - tFileList_2 - Start to list files
[INFO ]: xxxxx - tFileList_2 - Current file or directory path : /apploghost_pri-archive/syslog-ng/archive/var/syslog-ng/PROGRAMS/TRANSPORT/TPMBG-APP40.log.archived/TPMBG-APP40.log.2018-12-11.gz
[INFO ]: xxxxx - tFileList_2 - File or directory count : 1
[DEBUG]: xxxxx - tFileList_2 - Done.
/apploghost_pri-archive/syslog-ng/archive/var/syslog-ng/PROGRAMS/TRANSPORT/TPMBG-APP40.log.archived/TPMBG-APP40.log.2018-12-11.gz
...
...
...
[DEBUG]: xxxxx - tFileList_2 - Start to work.
[DEBUG]: xxxxx - tFileList_2 - ParametersSmiley Very HappyIRECTORY = context.transport_archive_dir+((String)globalMap.get("application_key"))+".log.archived/" | LIST_MODE = FILES | INCLUDSUBDIR = false | CASE_SENSITIVE = YES | ERROR = false | GLOBEXPRESSIONS = true | FILES = [{FILEMASK=((String)globalMap.get("application_key"))+".log."+(new SimpleDateFormat("yyyy-MM-dd")).format(globalMap.get("date"))+".gz"}] | ORDER_BY_NOTHING = true | ORDER_BY_FILENAME = false | ORDER_BY_FILESIZE = false | ORDER_BY_MODIFIEDDATE = false | ORDER_ACTION_ASC = true | ORDER_ACTION_DESC = false | IFEXCLUDE = false | FORMAT_FILEPATH_TO_SLASH = false |
[INFO ]: xxxxx - tFileList_2 - Start to list files
[INFO ]: xxxxx - tFileList_2 - Current file or directory path : /apploghost_pri-archive/syslog-ng/archive/var/syslog-ng/PROGRAMS/TRANSPORT/TPMBG-APP40.log.archived/TPMBG-APP40.log.2018-12-12.gz
[INFO ]: xxxxx - tFileList_2 - File or directory count : 1
[DEBUG]: xxxxx - tFileList_2 - Done.
/apploghost_pri-archive/syslog-ng/archive/var/syslog-ng/PROGRAMS/TRANSPORT/TPMBG-APP40.log.archived/TPMBG-APP40.log.2018-12-12.gz
...
...
...
[DEBUG]: xxxxx - tFileList_2 - Start to work.
[DEBUG]: xxxxx - tFileList_2 - ParametersSmiley Very HappyIRECTORY = context.transport_archive_dir+((String)globalMap.get("application_key"))+".log.archived/" | LIST_MODE = FILES | INCLUDSUBDIR = false | CASE_SENSITIVE = YES | ERROR = false | GLOBEXPRESSIONS = true | FILES = [{FILEMASK=((String)globalMap.get("application_key"))+".log."+(new SimpleDateFormat("yyyy-MM-dd")).format(globalMap.get("date"))+".gz"}] | ORDER_BY_NOTHING = true | ORDER_BY_FILENAME = false | ORDER_BY_FILESIZE = false | ORDER_BY_MODIFIEDDATE = false | ORDER_ACTION_ASC = true | ORDER_ACTION_DESC = false | IFEXCLUDE = false | FORMAT_FILEPATH_TO_SLASH = false |
[INFO ]: xxxxx - tFileList_2 - Start to list files
[INFO ]: xxxxx - tFileList_2 - File or directory count : 0
[DEBUG]: xxxxx - tFileList_2 - Done.
/apploghost_pri-archive/syslog-ng/archive/var/syslog-ng/PROGRAMS/TRANSPORT/TPMBG-APP40.log.archived/TPMBG-APP40.log.2018-12-12.gz

...

...

...

 

I can check also for  ((Integer)globalMap.get("tFileList_2_NB_FILE"))  of course, but I was expecting to have the global variable set by the component cleaned before any of its execution.

 


Accepted Solutions
Employee

Re: tFileList does not clean his global variables

@Lorenzo

 

Thanks for the diagram and I got a better idea of your problem.

 

You are using OnSubJobOk and OnComponentOk after getting the data from filter and we may not get the desired way of data flow here.

 

Instead, can we break the flow to two simple flows here? Following are the steps to do it.

 

a) After tfilterrow, push the output data to a tHashOutput

b) To execute the plain logs, use the below components.

     tHashInput (from previous stage output) -> tFlowtoIterate -> tFlisList ---- (if)----- > tRunJob Child

c) Again read the same hash to execute the gZipped logs

     tHashInput (from previous stage output) -> tFlowtoIterate -> tFlisList ---- (if)----- > tRunJob Child

      This time, mark the option to clear the hash after reading .

 

   I believe, it should resolve your issue. If it has helped in issue resolution, could you please mark the topic as resolved? Kudos are also welcome :-)

 

Warm Regards,

 

Nikhil Thampi


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)

All Replies
Employee

Re: tFileList does not clean his global variables

Hi,

 

    Could you please share a screen shot of your job and component detail screenshot so that we will get a better idea about the use case?

 

Warm Regards,

 

Nikhil Thampi


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Seven Stars

Re: tFileList does not clean his global variables

Sure.

 

job.png

Employee

Re: tFileList does not clean his global variables

@Lorenzo

 

Thanks for the diagram and I got a better idea of your problem.

 

You are using OnSubJobOk and OnComponentOk after getting the data from filter and we may not get the desired way of data flow here.

 

Instead, can we break the flow to two simple flows here? Following are the steps to do it.

 

a) After tfilterrow, push the output data to a tHashOutput

b) To execute the plain logs, use the below components.

     tHashInput (from previous stage output) -> tFlowtoIterate -> tFlisList ---- (if)----- > tRunJob Child

c) Again read the same hash to execute the gZipped logs

     tHashInput (from previous stage output) -> tFlowtoIterate -> tFlisList ---- (if)----- > tRunJob Child

      This time, mark the option to clear the hash after reading .

 

   I believe, it should resolve your issue. If it has helped in issue resolution, could you please mark the topic as resolved? Kudos are also welcome :-)

 

Warm Regards,

 

Nikhil Thampi


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.