topic Re: Extract sample rows from delimited input file in Talend Studio

Extract sample rows from delimited input file

JZMatrix1 — Fri, 15 Nov 2024 23:54:57 GMT

I'm trying to accomplish something similar to what is in https://community.talend.com/s/feed/0D73p000004kMH3CAM, using TOS 7.3, and it seems as though the "Extract Line at Random" is no longer available for the tFileInputDelimited component. The closest option I've been able to find is the tReservoirSampling component, which isn't available in TOS. Does this functionality still exist within TOS 7.3?

** I've been away from using Talend for a number of years, so finding the difference in 7.x since my last 5.x/6.x days **

Thank you,

Jason

Re: Extract sample rows from delimited input file

gjeremy1617088143 — Tue, 27 Jul 2021 13:45:22 GMT

Hi, @Jason Ziemba

I use a Talend data Services platform 7.3 and I can find the extract lines at random in the advanced settings of the tFileInputDelimited.

Send me love and kudos

Re: Extract sample rows from delimited input file

JZMatrix1 — Tue, 27 Jul 2021 14:55:35 GMT

My IDE is the Talend Open Studio for Data Integration (7.3.1.20200219_1130). Below is the screenshot of my tFileInputDelimited, advanced tab. Unfortunately, the random line option seems to be omitted from this platform in the latest version.

Re: Extract sample rows from delimited input file

gjeremy1617088143 — Wed, 28 Jul 2021 09:04:43 GMT

I think, you can do this for example :

read the file sent it into a hash

get the globalMap NB_LINE generated by the fileInput component :

create a globalMap of type int with value :Numeric.random(1,the globalMap NB_LINE generated by the fileInput component)

then read the hash --> use tflowtoiterate--> use a tfilterRow with this condition (CURRENT_ITERATION == (your globalMap))

Re: Extract sample rows from delimited input file

gjeremy1617088143 — Thu, 29 Jul 2021 12:30:21 GMT

you can also :

read the file, add a new field with a sequence , put into a hash read the NB_line

create a globalMap of type int with value :Numeric.random(1,the globalMap NB_LINE generated by the fileInput component)

read the hash

use a tfilterRow with this condition (field with sequence == (your globalMap))

Re: Extract sample rows from delimited input file

JZMatrix1 — Thu, 29 Jul 2021 12:57:09 GMT

Thank you for these suggestions. I did a different project that required a bit of similar logic and used a combination of global variables and a JavaRow component to pull in random numbers, determine if it was within the desired range, and then acted on it. I could do the same with this implementation, but got excited when I read about a checkbox that did all of that 'out of the box'.