Error: Unknown LOAD statement error caused by data???

[dirk_fischer]

Hi community,

I have a strange Problem and hope, somebody can help me out. I have an application, where I use the same Code in a loop for datasets from different jobs. In the load Statement, I use an ApplyMap for a Where condition. If the where condition returns no datasets, the statement works fine. If the where condition is fullfilled, an unknown load Statement error occurs.

Does anybody have an idea how to tackle this Problem? The challange is that the script in the faulty situation runs more than 10 Hours, before the error occurs.

I add the relevant sections of the logfile for the faulty case as well as for the successful case.

Any Suggestion would be much appreciated.

Best regards,

Dirk

[marcus_sommer · Accepted solution]

I could imagine that you are running into any timeout and/or hitting any limits on your machine, like not enough available RAM.

Your dataset seems to be a bit larger - at least what could be seen in your shortened log - with about 515 M records within the raw-data step which loads resident to apply an order by and later this table are loaded within a mapping table with an aggregation and conditions … maybe it's just too much.

I suggest that you split the task and check carefully for optimizations like implementing (more) incremental approaches and similar. Especially large mapping-loads take often a significantly amount of time if they need to pick the records from a larger table and even aggregate some stuff. In my experience is applying some pre-loads to prepare the mapping-content and to load then from there much faster and I mean not the pure mapping-load else the overall time of all necessary pre-loads.

- Marcus

[dirk_fischer]

Hi Marcus,

thank you very much for your feedback. I will try to aggregate before creating the mapping table. Unfortunately I have to do an initial load for this job, which is pretty big.

I don't think, the Problem is caused by a timeout, because the task is set to 5880 minutes before expiring. But "not enough RAM" might very well be an issue, because I saw that the operations monitor shows 100% RAM used on the machine in the same hour, when the failure occurred.

Regards,

Dirk