finding a string in a flat file

One Star

finding a string in a flat file

Hello !
I want to be able to find a string in a flat file using TOS 3.0.3 in perl... Can someone tell me how to do that please ?
Actually I'm looking for the string "error" in messy log files.
I've tried to link a tFileList to a tFileDelimited and to a tPerl containing something like
grep {error} $_globals{tFileList_1}{CURRENT_FILE};

but it seems to be looking for my string "error" only in the title of the files. Is there a variable for the current row read by tFileInputDelimited ?
I've also tried to use regular expressions with a tFileInputRegex, but I have to admit that I don't really master this kind of thing.
What would be the best solution ?
Thank you for your help Smiley Happy
One Star

Re: finding a string in a flat file

interesting question. Sounds easy but seems not to be...
First the simple (but bad) solution:
Use tFileInputRegex with the following regex mask: ".*error.*" or tFileInput followed by tFilterRow (".*error.*" again to filter). Now you have separated all lines with error in it. Now you could count the rows or anything else.
The bad point in my solution is: If you only would like to know is there a string error or not, you do parse always the full file.
What would you like to do with the information from your file?
What is your input data looks like?
One Star

Re: finding a string in a flat file

Hi Volker and thank you very much for your answer Smiley Happy
My input datas are very messy log files that have nothing in common one with another. They can contain 3 lines or 300 lines or more, they can contain tables or not.... all kind of different types of files (format should be .log or .dat)
I would like to be able to find the string error and retrieve the line containing the error as well as the text right below it that can contain details about the error. And I would like to put all these information in an excel file.
Do you think I can do that using Talend ? Do you need other details to be able to help me ?
Thank you Smiley Happy
One Star

Re: finding a string in a flat file

I think you could do that.
So lets do a short brainstorming:
* Define two context variables (actualOutputLine, maxOutputLine). The first will be used in the job and will not need a value. With the second one you could define how many lines after error you will print.
* As input use a tFullRow (because you need the whole row without any more structure to parse)
* Put your stream into a tPerlRow with the following code. You need one more attribute at the output (outputLine) than in input. Because I don't use perl generation it is more pseudo code and may not work. (But I'm sure someone out of the community will post a correct solution):
if ($input_row =~ /error/) {
$context= $context;
} else {
$output_row= $input_row;
$output_row= $context;

* Now use a tFilerRow and filter for each row with outputLine > 0.
* Use a component for output (excel for example).
Just out of interest: Which kind of (log?) file do you parse?

Re: finding a string in a flat file

I have only kept the previous line in memory.
I have 2 files to parse:
error 1
details about the problem

error 2
no smart details
another line
and yet another one

The code in the tPerlRow is:
$output_row = $input_row;
$output_row = $_globals{line};
$_globals{line} = $input_row;

And the output is:
Starting job topic5784 at 15:27 12/03/2009.
| tLogRow_1 |
| error_line | error_detail_line |
| error 2 | no smart details |
| error 1 | details about the problem |
Job topic5784 ended at 15:27 12/03/2009.
One Star

Re: finding a string in a flat file

You guys are just great, thank you very much ! It's working just fine, thank you !! Smiley Happy
Volker, the files I parse are log files generated by different kind of applications developed by some colleagues of mine.
One Star

Re: finding a string in a flat file

Thanks for feedback.
So it is a more "private" format. I asked because I thought about writing a component for parsing log4j logs.


Talend named a Leader.

Get your copy


Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables


Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables


Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema