Four Stars

Custom the flush buffer size" on the tFileOutputDelimited

 

When we can use the "Custom the flush buffer size" on the tFileOutputDelimited.

 

Default what value will be passed, if we not check the box.

 

What is the maximum value we can provide (in rows)?

1 REPLY
Employee

Re: Custom the flush buffer size" on the tFileOutputDelimited

Hi,

 

     Its a very good question. 

 

     If we are not giving any custom number to this parameter, the entire incoming data will be flushed to file in one go. If you go to the code tab, you can see it.

 

if (outtFileOutputDelimited_1 != null) {
            outtFileOutputDelimited_1.flush();
outtFileOutputDelimited_1.close();

}

      But if you are giving a custom value (say 200 rows), the component will flush data to file once it reach this upper threshold. 

if (nb_line_tFileOutputDelimited_1 % 200 == 0) {
outtFileOutputDelimited_1.flush();
}

 

    In this case, a 1000 row input data will be loaded to file by 5 data flushing loops (200 rows *5).

 

     The data will be in memory till its flushed to file. So unless you specify a custom flush buffer size, there is a chance that entire memory is consumed by the incoming data, if the incoming data size is very large. But if you put very less value, it will result in more I/O operations and which result in less throughput.

 

      There is no single magic value for this parameter as the underlying server memory decide the maximum capacity to store data before data flush to file. It also depends on the size of each row ( eg. a row with 10 columns vs a row with 200 columns). Based on your use case, you need to send the data and fine tune the performance.

 

If the answer has helped you, could you please mark the topic as solution provided? Kudos are also welcome :-)

 

Warm Regards,

 

Nikhil Thampi