topic Insert unique value in Talend Studio

Insert unique value

_AnonymousUser — Tue, 17 Dec 2013 06:31:07 GMT

hi,
I have two file File1, File2. Both have two field name,id.
File1:
name ID
akki 123
akshath 124
akki 123
File2:
name ID
aks 125
akki 123
aks 125
These data i want to insert to table. At the end table should have only unique data.
i.e
name ID
akki 123
akshath 124
aks 125
From each file i can get unique row using tUniqueRow component, but how to get unique row between files.
My job is some what like this:
tFileList---->tFileInputde---->tMap---->tUniqueRow---->tVerticaOutput.
So here only file level duplication can be eliminated. i want to achieve row level duplication.
This is just an example i have thousands of file like this.
Please do suggest the answers.
Regards,
Akshath Hegde

Re: Insert unique value

Anonymous — Tue, 17 Dec 2013 06:54:05 GMT

Hi,
All the thousand of files are same structure?
If so, you can use tfileinputxxx_1-->main-->tUnite-->tuniqrow-->UnqiRow-->toutputxxx
tfileinputxxx_2-->
See my screenshots
Best regards
Sabrina

Re: Insert unique value

_AnonymousUser — Tue, 17 Dec 2013 09:24:21 GMT

Hi,
Thanks for quick reply. But i cant use 1000 tFileInputDelimited for single job right!!! Im fetching file using tFileList because all file is of same extension and same structure. And after some time i may receive some more files, again i have to add tFileInputDelimited !!!. This doesn't look efficient.

Regards,
Akshath Hegde

Re: Insert unique value

_AnonymousUser — Wed, 18 Dec 2013 05:33:01 GMT

Hi,
Anything else we can do for this??
Note:files are taken one at a time.
Regards,
AKKI

Re: Insert unique value

Anonymous — Wed, 18 Dec 2013 05:46:42 GMT

Hi,
I think you can get this using tfilelist -->iterate-->tfileinputdelimited-->log
In tfileinputdelimited we can use default global variable "((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))"
something like this.
As you said you have many files, it means tables should be read dynamicaaly.
So as Sabrina mentioned in other forumtopic. to get tables dynamically we should have subscribed version.
If you are using scbscribed version i think you can do this.
Regards,
Vijayvanamala