Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

Four Stars

Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

I am trying to load several files with common names from S3 into Redshift.

 

Filename samples:

20190117_WEEKLY_TEST_AB.psv.gz

20190117_WEEKLY_REAL_AF.psv.gz

20190117_WEEKLY_TEST_AF.psv.ctl.gz

20190117_MONTHLY_TEST_AB.psv.gz

 

How would I tell Talend to load all of the files that are only  "*_WEEKLY_*_*.psv.gz" ? (first two files listed)

 

I tried entering this into the Key S3 Setting in my tDBBulkExec but the asterisks are being read as literal asterisks.

 

Any ideas on what to do?

Thanks,

Dan

Forteen Stars

Re: Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?


@dinzana wrote:

 

I tried entering this into the Key S3 Setting in my tDBBulkExec but the asterisks are being read as literal asterisks.

 

Any ideas on what to do?

Thanks,

Dan


tDBBulkExec - definitely do not understand wildcard, it always work (and for any databases) with a single file

 

try use tS3List and iterate 
I can not test it right now, but even if tS3List do not accept wildcard you can list all and filter iterations by name pattern - RunIf trigger

-----------
Four Stars

Re: Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

Thanks I will take a look into that!!!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download