Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

Four Stars

Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

I am trying to load several files with common names from S3 into Redshift.

 

Filename samples:

20190117_WEEKLY_TEST_AB.psv.gz

20190117_WEEKLY_REAL_AF.psv.gz

20190117_WEEKLY_TEST_AF.psv.ctl.gz

20190117_MONTHLY_TEST_AB.psv.gz

 

How would I tell Talend to load all of the files that are only  "*_WEEKLY_*_*.psv.gz" ? (first two files listed)

 

I tried entering this into the Key S3 Setting in my tDBBulkExec but the asterisks are being read as literal asterisks.

 

Any ideas on what to do?

Thanks,

Dan

Highlighted
Forteen Stars

Re: Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?


@dinzana wrote:

 

I tried entering this into the Key S3 Setting in my tDBBulkExec but the asterisks are being read as literal asterisks.

 

Any ideas on what to do?

Thanks,

Dan


tDBBulkExec - definitely do not understand wildcard, it always work (and for any databases) with a single file

 

try use tS3List and iterate 
I can not test it right now, but even if tS3List do not accept wildcard you can list all and filter iterations by name pattern - RunIf trigger

-----------
Four Stars

Re: Is it possible to use wildcards for an S3 key (filename) to allow Talend to load all files meeting the wildcard criteria?

Thanks I will take a look into that!!!

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog