topic Re: Filter comments out of input file in Talend Studio

Filter comments out of input file

imholzj — Sat, 16 Nov 2024 06:20:13 GMT

I have a csv file that has leading comments, how can I filter them out?

# use this file for additional users
# all values must be present
id,userName,date
"1","foo","01.01.2019"
"2","bar","02.01.2019"
.
.
.

Re: Filter comments out of input file

Anonymous — Thu, 14 Mar 2019 13:45:29 GMT

use Header = 2 in tfiledelimted

look at the screeshot below

Re: Filter comments out of input file

akumar2301 — Thu, 14 Mar 2019 14:48:21 GMT

If your 1st Field is Integer like "1" or 1(as in you case) , just remove "Die On Error" , All the record will be rejected( Including header)

You can capture these using Reject Link from Input Component

Re: Filter comments out of input file

imholzj — Thu, 14 Mar 2019 15:04:48 GMT

I am looking for a more general solution, allowing people to add more comments if necessary. Seems like hardcoding a 2 would prevent that.

jji

Re: Filter comments out of input file

imholzj — Thu, 14 Mar 2019 17:03:12 GMT

I'm going with a tFilterRow for a solution:

Advanced mode with:

// filter out all comments
!( java.util.regex.Pattern.matches("^\\s*#.*", row1.Status) )

In this case "Status" is the first field defined in the schema (which happened to be String.) I'm not sure how to do this if it weren't.

Thanks,

jji

Re: Filter comments out of input file

Anonymous — Thu, 14 Mar 2019 17:35:47 GMT

@imholzj but if the file is in below format

VERSION = 2007
DATE = 10/23/2017
EXPDATE = 10/1/2018
###########################################
FAMILY NURSE CARE OF LLC
HARRY GILL, MD, PHD, PC
LEAWOOD FAMILY EYE CARE PA

By using the tfilterrow in advance setting we can only remove one line with ## symbols

Please look at screenshot below i ran the test