Qlikview task fails in QMC

[KishoreSuriF]

Hi All,

We are unable to complete a task in QMC , failing repeatedly after executing at 2 or 3 hrs time , but actually it takes 3 mins only

We are getting the error as below in the log details. Please help us asap , it effecting business.

NoConcatenate
  20231219T114729.396+0000 0455     WT_VERSION_2:
  20231219T114729.396+0000 0456     LOAD *
  20231219T114729.396+0000 0457               
  20231219T114729.396+0000 0458          ,COD_POWERTRAIN_TYPE                         as COD_POWERTRAIN_TYPE_ORIGINAL 
  20231219T114729.396+0000 0459          ,DES_MAKE                               as DES_MAKE_PWT_TYPE
  20231219T114729.396+0000 0460          ,subfield(DES_MODEL, ':',2)                      as DES_MODEL_PWT_TYPE
  20231219T114729.396+0000 0461          ,DES_VERSION                             as DES_VERSION_PWT_TYPE
  20231219T114729.396+0000 0462          ,COD_FUEL                               as COD_FUEL_PWT_TYPE
  20231219T114729.396+0000 0463          ,COD_HYBRID                             as COD_HYBRID_PWT_TYPE
  20231219T114729.396+0000 0464          ,COD_OTHER_FUEL                           as COD_OTHER_FUEL_PWT_TYPE
  20231219T114729.396+0000 0465          ,COD_POWERTRAIN_TYPE                         as COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE
  20231219T114729.396+0000 0466     Resident WT_VERSION
  20231219T114729.397+0000        90 fields found: COD_DATASET, COD_VERSION, COD_VERSION_UNIQ, COD_JATO_GLOBAL_SEGMENT, COD_JATO_REGIONAL_SEGMENT, COD_DATA_SOURCE, COD_VOLUME_TYPE, COD_VERSION_TYPE, COD_PRECALC, COD_COUNTRY, COD_MANUF_COUNTRY, COD_MANUF_GROUP, DES_MAKE_GROUP, DES_MODEL_GROUP, COD_MANFTR, DES_MAKE, DES_MODEL, DES_TRIM_LEVEL, DES_LL_MANFTR_GROUP, DES_LL_MANFTR, DES_LL_MAKE_GLBAL, DES_LL_MAKE, DES_LL_MODEL, DES_LL_TRIM, DES_LL_VERSION, DES_LIMITED_VERSION, COD_BODY_STYLE, NUM_DOORS, COD_CAB_STYLE, COD_ROOF_TYPE, NUM_SEATS, COD_FAC_BODY_STYLE, NUM_LITERS, NUM_CC, NUM_CYLINDERS, COD_ENGINE_CONFIG, NUM_KW, NUM_HP, NUM_FISCAL_HP, COD_FUEL, COD_OTHER_FUEL, COD_HYBRID, COD_TURBO, COD_TRANSMISSION_TYPE, COD_TRANSMISSION, NUM_GEAR, COD_DRIVEN_WHEELS, COD_WHEELBASE_TYPE, COD_REAR_WHEELS, NUM_AXELS, NUM_MODEL_YEAR, NUM_MODEL_GENERATION, COD_PLATE, COD_CONV_ROOF, COD_DRIVER_LOC, COD_DATA_SET, FLG_MODEL_MIXED, NUM_LENGHT_MM, NUM_WIDTH_MM, NUM_HEIGHT_MM, NUM_WHEEL_BASE_MM, NUM_LENGHT_IN, NUM_WIDTH_IN, NUM_HEIGHT_IN, NUM_WHEEL_BASE_IN, NUM_LOAD_SPACE_VOLUME, NUM_LOAD_SPACE_CU, COD_DATA_SOURCE_2, COD_COUTRTY_ASSEMBLY, NUM_GVW, NUM_KERB_WEIGHT, NUM_PAYLOAD, NUM_GVW_LBS, NUM_KERB_WEIGHT_LBS, NUM_PAYLOAD_LBS, JOB_ID, COD_JATO_COUNTRY_VERS, COD_SOURCE_DB_VER, FLG_INTERFACE, COD_POWERTRAIN_TYPE, DES_VERSION, KEY_VERSION, COD_POWERTRAIN_TYPE_ORIGINAL, DES_MAKE_PWT_TYPE, DES_MODEL_PWT_TYPE, DES_VERSION_PWT_TYPE, COD_FUEL_PWT_TYPE, COD_HYBRID_PWT_TYPE, COD_OTHER_FUEL_PWT_TYPE, COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE, 
  20231219T123343.663+0000      1,556,692,963 lines fetched
  20231219T123343.677+0000 0467 LEFT JOIN
  20231219T123343.677+0000 0468     LOAD DES_MAKE                               as DES_MAKE_PWT_TYPE
  20231219T123343.677+0000 0469          ,replace(DES_MODEL, DES_MAKE&' ', '')                 as DES_MODEL_PWT_TYPE
  20231219T123343.677+0000 0470          ,text(DES_VERSION)                           as DES_VERSION_PWT_TYPE
  20231219T123343.677+0000 0471          ,COD_FUEL                               as COD_FUEL_PWT_TYPE
  20231219T123343.677+0000 0472          ,COD_HYBRID                             as COD_HYBRID_PWT_TYPE
  20231219T123343.677+0000 0473          ,COD_OTHER_FUEL                           as COD_OTHER_FUEL_PWT_TYPE
  20231219T123343.677+0000 0474          ,COD_POWERTRAIN_TYPE_ORIGINAL                     as COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE
  20231219T123343.677+0000 0475         ,if(COD_POWERTRAIN_TYPE_FIXED='', null(),COD_POWERTRAIN_TYPE_FIXED) as COD_POWERTRAIN_TYPE_FIXED
  20231219T123343.677+0000 0476     FROM
  20231219T123343.677+0000 0477     [..\..\..\..\..\..\..\\QvSourceDocs\ROOT\Data\Received\DataOverRide\PowertrainTypeOverRide.csv]
  20231219T123343.677+0000 0478     (txt, utf8, embedded labels, delimiter is ',', msq)
  20231219T123343.682+0000        8 fields found: DES_MAKE_PWT_TYPE, DES_MODEL_PWT_TYPE, DES_VERSION_PWT_TYPE, COD_FUEL_PWT_TYPE, COD_HYBRID_PWT_TYPE, COD_OTHER_FUEL_PWT_TYPE, COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE, COD_POWERTRAIN_TYPE_FIXED, 
  20231219T123345.835+0000      1,819,260 lines fetched
  20231219T123345.835+0000      Joining/Keeping
  20231219T125610.062+0000      -129 = 
  20231219T125727.851+0000      Execution Failed
  20231219T125727.946+0000      Execution finished.

[marcus_sommer]

Just to clarify some things. 1.5 billions of records aren't too much data for an appropriate sized server-cluster - even much more would be possible. But the things needs to be done in the right way which means of course to implement sensible incremental approaches.

Further make sure that there are no record-id's included or timestamps (could be split to dates and times) - and in general to take a look on the number of distinct field-values. Also don't apply any row-level formatting.

Beside this your data look like a crosstable which is in most cases better resolved to regular records - even if the number of records increased significantly (a fact-table with more a 90 fields looks quite odd - means you should in general consider which data are facts and which are dimensions). In this regard you may not the join-stuff anymore. If there is really a need to add any information horizontally a mapping instead of a join would be simpler, more powerful, less risky and also reducing the needed resources. 

[KishoreSuriF]

Thanks Marcus.

By left join, we are joining single column and not the rows data.

However, will check as per suggestion and will get back here.

[marcus_sommer]

A join matched horizontal information by adding column(s) and/or rows based on the specified row-level data (key-fields) or might be excluding records. Better is: Don't join - use Applymap instead - Qlik Community - 1467592.

[KishoreSuriF · Accepted solution]

After following multiple approaches, we didn’t succeed.

so, we restored the back dated files and QVDs and reloaded successfully.Now everything is going good. If anything required we will get back to you