topic Re: Loading Script output shows huge no of lines fetch's in App Development

Loading Script output shows huge no of lines fetch's

nandhaadjame200 — Mon, 05 Nov 2018 05:46:05 GMT

Hi There

In my loading script, i'm loading .CSV files on daily basis and incremental basis. I observed in the output when loading my script like the no. of lines fetched from .CSV file is showing as Lakh's & Crore's but the particular .CSV is having only around 100's & 1000's of records only. Also attached the output files here.

Please help to identify the issue and fix.

Thanks in Advance.

Thanks

Nandhakumar

Re: Loading Script output shows huge no of lines fetch's

vamsee — Mon, 05 Nov 2018 16:28:37 GMT

Hello,

I have looked at you attached script and also the log. I do see some of the statements where the activity QVD loads more than a 1 lakh files.

Check if the qvd's and also you could write a trace statements and sub counters like

SUB Counter (TableName)

Let vCount='Rows fetched for table ' & '$(TableName)' & ': ' & Num(NoOfRows('$(TableName)'),'#,##0');
Trace $(vCount);

ENDSUB

CALL Counter('Activity'); /** Inside the for loop and after IF statements **/

Re: Loading Script output shows huge no of lines fetch's

vamsee — Mon, 05 Nov 2018 16:31:10 GMT

IF the issue still persists, try loading some handful of files into a different folder, change the $(vRoot), $(vQVDRoot) and check.

This way you could also test your if statements for Activity file not found.

Re: Loading Script output shows huge no of lines fetch's

nandhaadjame200 — Mon, 05 Nov 2018 18:08:35 GMT

Hi Vamsee

First Thanks a lot for your reply.

I've tested based on your guidance and found like - I've a csv file with 1000 records but it is showing as "Lines Fetched: 1529" in output. I don't know why.

Please help to fix this. Thanks in advance.

Thanks

Nandhakumar

Re: Loading Script output shows huge no of lines fetch's

vamsee — Mon, 05 Nov 2018 18:40:42 GMT

As you are concatenating your CSV's the value shown is a sum of records in the current file and the previously loaded files.

Eg:

LOAD 'Test' as Column

Autogenerate(100);

Concantenate(A)

LOAD 'Test' as Column

Autogenerate(200);

Would result in 300 records for Table A;

Re: Loading Script output shows huge no of lines fetch's

nandhaadjame200 — Mon, 05 Nov 2018 20:14:05 GMT

Yes, objective is to do incremental loading of CSV files which comes in on daily basis. But here i'm i tested with single CSV file for now and it is still showing excess record fetched in output but file is actually having less records than shown count.

Thanks

Re: Loading Script output shows huge no of lines fetch's

vamsee — Mon, 05 Nov 2018 20:32:30 GMT

If you loading only one file then this should not happen.

Try loading the CSV file directly in a different qvf and check the number of records. If the records are 1000, as expected then there is something wrong with the code

Activity.qvd already has data.

Re: Loading Script output shows huge no of lines fetch's

nandhaadjame200 — Tue, 06 Nov 2018 09:51:13 GMT

Hi Vamsee

I found the bug. I've to drop the "Activity" Table after the reload process for each metric because the same table is used in another metric script with same headers, so the records remains in the table and duplicated until script ends. So, i've dropped the "Activity" table in each metric script and finally fixed the bug and now it is all working good.

Appreciate your prompt reply and your guidance.

Thanks

Nandhakumar