Skip to main content
Announcements
See what Drew Clarke has to say about the Qlik Talend Cloud launch! READ THE BLOG
cancel
Showing results for 
Search instead for 
Did you mean: 
priyanka08
Contributor III
Contributor III

Count Second appearance of Record

Hi 

I  received the monthly file like in  Jan -21 data ,I have Jan-20  and Jan-21 data .However in Jan -22 data ,I have only Jan-21 and Jan-22 data. Which leads to two entry for Jan -21  . I want to pick the Jan -21 data only from the latest file and remove the old data for Jan-21

Input

priyanka08_0-1663832426191.png

Need to create a flag =0 and then remove the row where flag =0 is mentioned

priyanka08_1-1663832472364.png

 

 

 

Labels (3)
1 Solution

Accepted Solutions
rubenmarin

Hi, you can load fiels from the newest to the oldest in a for..next bucle, and use exists() to check if the data is already loaded.

1- Load all filenames and extract the date part as a date

2- Do a sorted load of the previous table to sort filenames from the newest to the oldest

3. Do a for i=0 to NoOfRows(FileNamesTable)

3.1 Using Peek store the filename in a variable

3.2 Load the file using the variable

3.3 In the load create a key thet identifies the data loaded, ie: Region &'_'& Date as KeyLoaded

3.4 In the where use Exists to check if the key was already loaded Where not Exists('KeyLoaded',Region &'_'& Date)

View solution in original post

2 Replies
rubenmarin

Hi, you can load fiels from the newest to the oldest in a for..next bucle, and use exists() to check if the data is already loaded.

1- Load all filenames and extract the date part as a date

2- Do a sorted load of the previous table to sort filenames from the newest to the oldest

3. Do a for i=0 to NoOfRows(FileNamesTable)

3.1 Using Peek store the filename in a variable

3.2 Load the file using the variable

3.3 In the load create a key thet identifies the data loaded, ie: Region &'_'& Date as KeyLoaded

3.4 In the where use Exists to check if the key was already loaded Where not Exists('KeyLoaded',Region &'_'& Date)

priyanka08
Contributor III
Contributor III
Author

@rubenmarin thanks it worked