Duplicate rows when storing data into csv

[nigel987]

Hi,

I'm storing fields from a table into a csv file in the following way:

Ads:

LOAD

  LoginID,

  LoginCreated,

  LoginDeletionDate,

  AdID,

  AdCreated,

  AdStartDate,

  AdPublished,

  SourceID,

.... (and some more)

Resident Dataset;

STORE * from Ads into $(pathQVDs)Seller\ads.csv(txt);

This works fine, but many rows in the csv are exactly the same but are duplicated several times (some 5 times, some 30 times).

Here's some sample lines:

LoginID,LoginCreated,LoginDeletionDate,AdID,AdCreated,AdStartDate,AdPublished

123456,28.09.2015,,134752458,28.09.2015,28.09.2015,,,12.11.2015,,6

123456,28.09.2015,,134752496,28.09.2015,28.09.2015,,,12.11.2015,,6

When I create a table object and add the fields in the same order, I have only one line! So I'm really sure that the lines are exact duplicates.

I have no idea why this happens. Has anyone experienced the same issue?

Many thanks for any hint!

KR, Nigel

[Chanty4u · Accepted solution]

u can try

Load

Disctinct...

.....

....

store (qvd)

[Chanty4u]

sample:

Hi, use the keyword "DISTINCT" and RowNo() function.

I.E.

TABLE1:

LOAD DISTINCT

    FIELD_01,

    FIELD_02,

    FEILD_03,

    ...

    FIELD_09

    ;

with this keyword all rows with the same values on all fields are reduced to one row and the table are load with no duplicate rows.

TABLE2:

LOAD

     RowNo()  AS NUMBER,

     *

RESIDENT TABLE1;

DROP TABLE1;

with RowNo() you can enumerate the rows in a table.

[PrashantSangle]

Hi,

In provide sample lines AdID different so those lines are not duplicate.

If it really contain duplicate check your value in DataSet.

Regards

[settu_periasamy]

Hi,

The Table Object doesn't display the duplicate values (if all field values are Same)..

You can check it in the Table Viewer, It will shows how many rows in the table..

for eg. When i load the below 2 rows, the table object will display single row only.. see the screen shot.

LOAD * Inline [
Dim1,Dim2,Dim3
A,B,C
A,B,C
];

Table Viewer shows : 2 Rows

[nigel987]

Hi Chanty,

beginner's mistake I missed the DISTINCT

thanks!

[Chanty4u]

Hmmm noprb....

u can make others as helpful. which is other try fr ur requirment.

Thanks

Suresh chanty