topic Re: tSystem - sed command - I am close, just need a little help in Talend Studio

tSystem - sed command - I am close, just need a little help

talendtester — Tue, 06 Nov 2018 23:26:40 GMT

When I am in the command line, this command works on Windows for using the GNU sed:

sed -i ":a;N;$!ba;s/\"\n\"«/\"\"«/g" myFile.csv

I have a 115GB file which has multiple row which are breaking to multiple lines.

I need to find and replace all occurrences of double quote new line character double quote and special character « (which is my delimiter).

For example:

line_one_here"\n
"«the_rest_of_line_one_here

line_two_here

line_three_here

I am trying to get this to work in tSystem:

"sed -i \":a;N;$!ba;s/\"\n\"«/\"\"«/g\" "+ context.myFile

Error is:

sed: -e expression #1, char 12: unterminated `s' command

Similar to this post:

https://community.talend.com/t5/Design-and-Development/Not-able-to-run-Linux-command-using-tSystem-Component/td-p/106483

Re: tSystem - sed command - I am close, just need a little help

Anonymous — Wed, 07 Nov 2018 00:21:02 GMT

Whats wrong with the line breaks?

It is not a problem for parsing CSV!

If the values are capsulated in an exclosure like " the line break does not disturb will be treated simple as content.

Re: tSystem - sed command - I am close, just need a little help

talendtester — Wed, 07 Nov 2018 00:34:31 GMT

The problem is when the 115GB file was created, the .csv option wasn't turned on

Re: tSystem - sed command - I am close, just need a little help

Anonymous — Wed, 07 Nov 2018 00:47:10 GMT

I have done a quick check what is the result of your sed java expression:

In a tJava:

System.out.println("sed -i \":a;N;$!ba;s/\"\n\"«/\"\"«/g\" ");

The result is:

sed -i ":a;N;$!ba;s/"
"«/""«/g"

It does not looks like your original working command line above.

Your problem is the line break misused currently. Try this as java expression in the tSystem component.

"sed -i \":a;N;$!ba;s/\\\"\\n\\\"«/\\\"\\\"«/g\" "

compared to your working command from your post:

sed -i ":a;N;$!ba;s/\"\n\"«/\"\"«/g"

this is the result of my changed java escaped command line:

sed -i ":a;N;$!ba;s/\"\n\"«/\"\"«/g"

I suggest you are trying your expression and print it out with System.out.println and tweak it until it looks like your expected command line.

Re: tSystem - sed command - I am close, just need a little help

talendtester — Wed, 07 Nov 2018 01:14:33 GMT

I think this works:

context.myCommand="sed -i \":a;N;$!ba;s/\\\"\\n\\\"«/\\\"\\\"«/g\"
myFile.csv";
System.out.println(context.myCommand);

Re: tSystem - sed command - I am close, just need a little help

Anonymous — Wed, 07 Nov 2018 01:23:00 GMT

Where the hell is my response? I have for about an hour send your exactly this answer!

@talendtester wrote:
I think this works:

context.myCommand="sed -i \":a;N;$!ba;s/\\\"\\n\\\"«/\\\"\\\"«/g\"
myFile.csv";
System.out.println(context.myCommand);

Re: tSystem - sed command - I am close, just need a little help

talendtester — Wed, 07 Nov 2018 17:31:58 GMT

@lli thank you for your fast help!

Not sure, I got the email update notice and I attempted to respond to it (instead of the earlier post) but the forums gave me a message saying I didn't have permissions and then when I refreshed the page your newer post was gone.