8 Replies Latest reply: Dec 5, 2012 2:04 AM by sujeet singh RSS

    Synthetic Field Performance Issue

      Hi there,

       

      I am attempting to create a QV model from an ERP-grade application. For this, I have "delegated" joins to QV, thinking it was the best approach. I am running this model on a laptop (with 4GB dedicated memoy to the Windows instance I am working from).

       

      Basically, I am attempting to create a sort of a star-schema. I am simplifying the issue, this is what my script looks like:

       

      qualify * 
      unqualify A, B, C, D 
      
      // Transactional data
      [Dataset1]: 
      SQL SELECT A, B, C, D, <...>
      
      qualify * 
      unqualify A, B
      
      // 'dimension'
      [MasterData1] 
      SQL SELECT A, B, <...>
      
      qualify * 
      unqualify A, C
      
      // 'dimension'
      [MasterData2] 
      SQL SELECT A, C, <...>
      

       

      FYI, this is the data yield:

      • DataSet1: 100k records
      • MasterData1: <50 records
      • MasterData2: <5k records

       

      I am satisfied with the result, it's extracting and processing in less than 5 seconds with a very low processor/memory footprint.

       

      Connecting to XXX
       Connected
       DataSet1 << AAA 112,026 lines fetched
       MasterData1 << BBB 29 lines fetched
       MasterData2 << CCC 4,096 lines fetched
       $Syn 1 = A+B
       $Syn 2 = A+C
       $Syn 3 = $Syn 1+$Syn 2 
      

       

      Next I am adding one more MasterData to the script:

       

      // 'dimension'
      [MasterData3] 
      SQL SELECT A, D, <...>
      

       

      I have tried with a MasterData yielding 1 record or 1M records, I get to the same result: I am maxing out ma machine's memory.

       

      Connecting to XXX
       Connected
      DataSet1 << AAA 112,026 lines fetched
      MasterData1 << BBB 29 lines fetched
      MasterData2 << CCC 4,096 lines fetched
      MasterData3 << DDD 1 lines fetched
       $Syn 1 = A+B
       $Syn 2 = A+C
       $Syn 3 = A+D 
      

       

      What could explain the sudden explosion in memory usage?

       

      Edit:changed the title for more visibility