Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

Four Stars

Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

I am trying to load several files with common names from S3 into Redshift.

 

Filename samples:

20190117_WEEKLY_TEST_AB.psv.gz

20190117_WEEKLY_REAL_AF.psv.gz

20190117_WEEKLY_TEST_AF.psv.ctl.gz

20190117_MONTHLY_TEST_AB.psv.gz

 

How would I tell Talend to load all of the files that are only  "*_WEEKLY_*_*.psv.gz" ? (first two files listed)

 

I tried entering this into the Key S3 Setting in my tDBBulkExec but the asterisks are being read as literal asterisks.

 

Any ideas on what to do?

Thanks,

Dan

Thirteen Stars

Re: Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?


@dinzana wrote:

 

I tried entering this into the Key S3 Setting in my tDBBulkExec but the asterisks are being read as literal asterisks.

 

Any ideas on what to do?

Thanks,

Dan


tDBBulkExec - definitely do not understand wildcard, it always work (and for any databases) with a single file

 

try use tS3List and iterate 
I can not test it right now, but even if tS3List do not accept wildcard you can list all and filter iterations by name pattern - RunIf trigger

-----------
Four Stars

Re: Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

Thanks I will take a look into that!!!

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.