5 Replies Latest reply: Dec 14, 2014 2:51 AM by Massimo Grossi RSS

    Ignore 0 when load QVD, keep optimazed

      Hello Guys!

       

      I have really huge amount of data (one qvd file is about 250 million of rows, and I need to load and process several files). The problem is that these qvd files contains duplicates of rows.

      For example I have two rows for the same Item. One row contains Sum of sales and 0 as a Price, but the other one contains 0 in Sales and positive Price.

      ItemPriceSumOfSales
      100010100
      1000110

      I want to reduce data by joining parts of the same table:

      t1:

      LOAD

           Item,

           Sum

      From QVD Where Sum>0;

       

      left join (t1)

      LOAD

           Item,

           Price

      From QVD Where Price>0;

       

      But if i use "where" in a clause it breaks optimized load and all my data loads terrifying slow.

      Can somebody advice me how to keep optimized load and keep only rows with positive values?