Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi All,
We are unable to complete a task in QMC , failing repeatedly after executing at 2 or 3 hrs time , but actually it takes 3 mins only
We are getting the error as below in the log details. Please help us asap , it effecting business.
NoConcatenate
20231219T114729.396+0000 0455 WT_VERSION_2:
20231219T114729.396+0000 0456 LOAD *
20231219T114729.396+0000 0457
20231219T114729.396+0000 0458 ,COD_POWERTRAIN_TYPE as COD_POWERTRAIN_TYPE_ORIGINAL
20231219T114729.396+0000 0459 ,DES_MAKE as DES_MAKE_PWT_TYPE
20231219T114729.396+0000 0460 ,subfield(DES_MODEL, ':',2) as DES_MODEL_PWT_TYPE
20231219T114729.396+0000 0461 ,DES_VERSION as DES_VERSION_PWT_TYPE
20231219T114729.396+0000 0462 ,COD_FUEL as COD_FUEL_PWT_TYPE
20231219T114729.396+0000 0463 ,COD_HYBRID as COD_HYBRID_PWT_TYPE
20231219T114729.396+0000 0464 ,COD_OTHER_FUEL as COD_OTHER_FUEL_PWT_TYPE
20231219T114729.396+0000 0465 ,COD_POWERTRAIN_TYPE as COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE
20231219T114729.396+0000 0466 Resident WT_VERSION
20231219T114729.397+0000 90 fields found: COD_DATASET, COD_VERSION, COD_VERSION_UNIQ, COD_JATO_GLOBAL_SEGMENT, COD_JATO_REGIONAL_SEGMENT, COD_DATA_SOURCE, COD_VOLUME_TYPE, COD_VERSION_TYPE, COD_PRECALC, COD_COUNTRY, COD_MANUF_COUNTRY, COD_MANUF_GROUP, DES_MAKE_GROUP, DES_MODEL_GROUP, COD_MANFTR, DES_MAKE, DES_MODEL, DES_TRIM_LEVEL, DES_LL_MANFTR_GROUP, DES_LL_MANFTR, DES_LL_MAKE_GLBAL, DES_LL_MAKE, DES_LL_MODEL, DES_LL_TRIM, DES_LL_VERSION, DES_LIMITED_VERSION, COD_BODY_STYLE, NUM_DOORS, COD_CAB_STYLE, COD_ROOF_TYPE, NUM_SEATS, COD_FAC_BODY_STYLE, NUM_LITERS, NUM_CC, NUM_CYLINDERS, COD_ENGINE_CONFIG, NUM_KW, NUM_HP, NUM_FISCAL_HP, COD_FUEL, COD_OTHER_FUEL, COD_HYBRID, COD_TURBO, COD_TRANSMISSION_TYPE, COD_TRANSMISSION, NUM_GEAR, COD_DRIVEN_WHEELS, COD_WHEELBASE_TYPE, COD_REAR_WHEELS, NUM_AXELS, NUM_MODEL_YEAR, NUM_MODEL_GENERATION, COD_PLATE, COD_CONV_ROOF, COD_DRIVER_LOC, COD_DATA_SET, FLG_MODEL_MIXED, NUM_LENGHT_MM, NUM_WIDTH_MM, NUM_HEIGHT_MM, NUM_WHEEL_BASE_MM, NUM_LENGHT_IN, NUM_WIDTH_IN, NUM_HEIGHT_IN, NUM_WHEEL_BASE_IN, NUM_LOAD_SPACE_VOLUME, NUM_LOAD_SPACE_CU, COD_DATA_SOURCE_2, COD_COUTRTY_ASSEMBLY, NUM_GVW, NUM_KERB_WEIGHT, NUM_PAYLOAD, NUM_GVW_LBS, NUM_KERB_WEIGHT_LBS, NUM_PAYLOAD_LBS, JOB_ID, COD_JATO_COUNTRY_VERS, COD_SOURCE_DB_VER, FLG_INTERFACE, COD_POWERTRAIN_TYPE, DES_VERSION, KEY_VERSION, COD_POWERTRAIN_TYPE_ORIGINAL, DES_MAKE_PWT_TYPE, DES_MODEL_PWT_TYPE, DES_VERSION_PWT_TYPE, COD_FUEL_PWT_TYPE, COD_HYBRID_PWT_TYPE, COD_OTHER_FUEL_PWT_TYPE, COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE,
20231219T123343.663+0000 1,556,692,963 lines fetched
20231219T123343.677+0000 0467 LEFT JOIN
20231219T123343.677+0000 0468 LOAD DES_MAKE as DES_MAKE_PWT_TYPE
20231219T123343.677+0000 0469 ,replace(DES_MODEL, DES_MAKE&' ', '') as DES_MODEL_PWT_TYPE
20231219T123343.677+0000 0470 ,text(DES_VERSION) as DES_VERSION_PWT_TYPE
20231219T123343.677+0000 0471 ,COD_FUEL as COD_FUEL_PWT_TYPE
20231219T123343.677+0000 0472 ,COD_HYBRID as COD_HYBRID_PWT_TYPE
20231219T123343.677+0000 0473 ,COD_OTHER_FUEL as COD_OTHER_FUEL_PWT_TYPE
20231219T123343.677+0000 0474 ,COD_POWERTRAIN_TYPE_ORIGINAL as COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE
20231219T123343.677+0000 0475 ,if(COD_POWERTRAIN_TYPE_FIXED='', null(),COD_POWERTRAIN_TYPE_FIXED) as COD_POWERTRAIN_TYPE_FIXED
20231219T123343.677+0000 0476 FROM
20231219T123343.677+0000 0477 [..\..\..\..\..\..\..\\QvSourceDocs\ROOT\Data\Received\DataOverRide\PowertrainTypeOverRide.csv]
20231219T123343.677+0000 0478 (txt, utf8, embedded labels, delimiter is ',', msq)
20231219T123343.682+0000 8 fields found: DES_MAKE_PWT_TYPE, DES_MODEL_PWT_TYPE, DES_VERSION_PWT_TYPE, COD_FUEL_PWT_TYPE, COD_HYBRID_PWT_TYPE, COD_OTHER_FUEL_PWT_TYPE, COD_POWERTRAIN_TYPE_ORIGINAL_PWT_TYPE, COD_POWERTRAIN_TYPE_FIXED,
20231219T123345.835+0000 1,819,260 lines fetched
20231219T123345.835+0000 Joining/Keeping
20231219T125610.062+0000 -129 =
20231219T125727.851+0000 Execution Failed
20231219T125727.946+0000 Execution finished.
Just to clarify some things. 1.5 billions of records aren't too much data for an appropriate sized server-cluster - even much more would be possible. But the things needs to be done in the right way which means of course to implement sensible incremental approaches.
Further make sure that there are no record-id's included or timestamps (could be split to dates and times) - and in general to take a look on the number of distinct field-values. Also don't apply any row-level formatting.
Beside this your data look like a crosstable which is in most cases better resolved to regular records - even if the number of records increased significantly (a fact-table with more a 90 fields looks quite odd - means you should in general consider which data are facts and which are dimensions). In this regard you may not the join-stuff anymore. If there is really a need to add any information horizontally a mapping instead of a join would be simpler, more powerful, less risky and also reducing the needed resources.
Thanks Marcus.
By left join, we are joining single column and not the rows data.
However, will check as per suggestion and will get back here.
A join matched horizontal information by adding column(s) and/or rows based on the specified row-level data (key-fields) or might be excluding records. Better is: Don't join - use Applymap instead - Qlik Community - 1467592.
After following multiple approaches, we didn’t succeed.
so, we restored the back dated files and QVDs and reloaded successfully.Now everything is going good. If anything required we will get back to you