Is there an easier way to retrieve only the last line of a file?

One Star

Is there an easier way to retrieve only the last line of a file?

I need to retrieve only the last non-empty line of a file (the footer).

I can't use the mutli-schema inputs on this file, because the first field may vary so it can't be used as the schema-type key.

I suppose I could get the input with tFileInputFullRow, then use tFilterRows with a regular expression that matches only the content of the last line... but that would be slow and potentially buggy. (Edit: Oh, now I notice the tFileInputRegex component which could do both of these steps.)

Another option would be to use the tJava* with some simple custom code like http://www.javaprogrammingforums.com/java-code-snippets-tutorials/29-how-read-last-line-file.html

Or can you think of a better way to do this?

(* Correction: You would have to use a tJavaFlex, since it's the only custom code one which doesn't require an input flow. Since no looping is needed, you can put all your custom code in the starting block.)
Seventeen Stars

Re: Is there an easier way to retrieve only the last line of a file?

hi ewall

what about Tail command ?

regards
laurent
Six Stars

Re: Is there an easier way to retrieve only the last line of a file?

It is quite a strange request for an ETL tool because usually record doesn't have a natural order by itself (order is imposed by the user and semantics and fields).

Anyway To have the last "physical" row the best I can think of is:
- The solution invoking an external command like tail will be ok but surely not portable (if you need portability)
- Create an artificial increment field in talend and get only the row with max value (but involves a full scan)
- Count the rows and then use in talend tSampleRow (but again involves a full scan)
- Use custom java code to just extract the last physical row and pass it to talend flow (best performance, not a full scan)
- Use custom java code fully implementing the "tail" unix command in java (search google for it)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Have you checked out Talend’s 2019 Summer release yet?

Find out about Talend's 2019 Summer release

Blog

Talend Summer 2019 – What’s New?

Talend continues to revolutionize how businesses leverage speed and manage scale

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog