topic Re: UTF-8 BOM Encoded File Processing in Data Quality

UTF-8 BOM Encoded File Processing

Anonymous — Sat, 16 Nov 2024 07:40:41 GMT

We are getting a daily file in UTF-8 BOM encoding because of which our Talend ETL Job always misses the first row of the file

Sample Data in File:

P, 1234, $10

Q,1235,$20

R, 1236, $15

Our actual flow is like

tFileList ==>> tFileInputDelimited ==>> fReplicate ==> tFilterRow ==> tMSSqlSCD

Actually tFileInputDilimited is able to process all rows but when we use tFilterRow, but it always misses first row of every particular file

The condition for tFilterRow is column0 Equals "P"

When we configured tLogRow we found few special characters prefixed with the first rows of all files. Example ???P

Also when we opened our CSV files in Notepad++ we discovered that File is encoded in UTF-8-BOM

We have option only for UTF-8 in Advanced settings of tfiledilimited

Let us know how can we process UTF-8-BOM file using Talend job

Thanks & Regards

Re: UTF-8 BOM Encoded File Processing

Anonymous — Wed, 12 Sep 2018 09:55:48 GMT

Hello,

Are you using talend Data Preparation Free Desktop? Is there any error in app.log?

Best regards

Sabrina

Re: UTF-8 BOM Encoded File Processing

Anonymous — Wed, 12 Sep 2018 10:00:57 GMT

Hi Justin,

When do you get this error? When trying to add a dataset or a preparation step? If so, did you follow the steps to disable App Nap as mentioned in the getting started guide? See https://help.talend.com/reader/mime3pPIS~faiBvVb~FT7w/b006EaGseLmCHC2Tx__deg for more details.

If you are facing the issue as soon as you access the product UI, can you share the log file (Library/Application Support/Talend/dataprep/logs/app.log)?

Regards,

Gwendal

Re: UTF-8 BOM Encoded File Processing

Anonymous — Sat, 15 Sep 2018 14:25:57 GMT

Hello

Thanks for the advice.

I have now disabled App Nap, but still get the error when undertaking a preparation step.

Unfortunately the directory and log file that you refer to cannot be found on my computer.

Justin

Re: UTF-8 BOM Encoded File Processing

Anonymous — Mon, 17 Sep 2018 12:44:04 GMT

Hi Justin,

A quick check before going any further: which version of Data Preparation Free Desktop are you using? That is either visible in the footer of the application (for older versions) or in the "About Data Preparation" menu on the top-right (for the latest versions).

Thank you,

Gwendal

Re: UTF-8 BOM Encoded File Processing

Anonymous — Mon, 17 Sep 2018 20:09:57 GMT

It is version 2.5.1.

Re: UTF-8 BOM Encoded File Processing

Anonymous — Thu, 27 Sep 2018 15:38:11 GMT

Hello @Moe,

The application log file is located here:

~/Library/Application Support/Talend/dataprep/logs/app.log

or if you use an absolute path:

/Users/{your-username}/Library/Application Support/Talend/dataprep/logs/app.log

Re: UTF-8 BOM Encoded File Processing

Anonymous — Thu, 27 Sep 2018 20:38:57 GMT

As mentioned before I’m afraid there is no Talend folder in my Application Support folder - see screenshot below. I have also searched for the app.log file and this is not present on my computer. Please could you advise me what to do.

Thank you

Re: UTF-8 BOM Encoded File Processing

Anonymous — Sat, 06 Oct 2018 22:44:43 GMT

Hello

I am still having this problem on my new MacBook Pro. However I have also installing the same software onto my older iMac and it works fine without a problem. However, I do need to be able to use on my MacBook Pro.

Both machines are running the same version of Mac OSX Version 10.13.6 (17G65). The iMac is 21.5-inch, Mid 2011; 2.5 GHz Intel Core i5; 8 GB 1333 MHz DDR3. The MacBook Pro is 13-inch, 2017; 2.3 GHz Intel Core i5; 8 GB 2133 MHz LPDDR3

Can you offer any further advice to help it work on the MacBook Pro? Thanks

Re: UTF-8 BOM Encoded File Processing

Anonymous — Tue, 12 Feb 2019 13:06:47 GMT

Having the same issue, tried the script:

defaults write org.talend.dataprep NSAppSleepDisabled -bool YES

But still having those errors, and in the logs you find:

2019-02-12 14:00:16.553 INFO 4086 --- [qtp1370856792-60] o.t.d.d.service.BaseDataSetService : New DataSet #2bbbd9fc-5c1c-421e-98e0-5b2895d9d378, name: Saved Cases, type: text/csv, from: local
2019-02-12 14:00:16.576 WARN 4086 --- [qtp1370856792-20] org.eclipse.jetty.server.HttpChannel : /api/datasets

To see the whole post, download it here
OriginalPost.pdf

Re: UTF-8 BOM Encoded File Processing

Anonymous — Thu, 14 Feb 2019 10:29:03 GMT

@Andries Can you please tell me what is the details of your local file you are trying to upload (size, number of rows / columns) ?

I also reproduce the error with a dataset (10k rows / 32 columns) but i can open it and view my data.

Can you open/view your dataset please ?

Re: UTF-8 BOM Encoded File Processing

Anonymous — Thu, 14 Feb 2019 11:13:41 GMT

It is with every import, every file, every data preparation step... always

Re: UTF-8 BOM Encoded File Processing

Anonymous — Thu, 14 Feb 2019 11:47:35 GMT

@Andries Can you please update the Info.plist file (in the /Applications/Talend Data Preparation Free Desktop.app/Contents folder or right click on the Talend Data Preparation icon and select view the contents of the package).

Then add the entry (like in the attached screenshot)

<string>-Dhystrix.command.default.execution.timeout.enabled=false</string>

Capture d’écran 2019-02-14 à 12.17.48.png

Re: UTF-8 BOM Encoded File Processing

Anonymous — Thu, 14 Feb 2019 15:47:41 GMT

Thank you, that seems to solves it !!!