I have a script which brings in data from a SQL server. The source table is enourmous - multiple billions of rows of data. In the SQL query, it gets aggregated down to about 700 million rows.
This ran perfectly fine for over a year. The source dataset continued to grow as new data came in. Then, one day, it just started hanging. The job wouldn't fail, it would just go on indefinitely. The load used to take about 4 hours. Suddenly, it was running for 24+ hours before we'd abort it.
We ran the query in SQL directly, and it ran in 4 hours. I brought the data straight into QlikView without aggregating, and tried to do the aggregation in the script on the QV side instead of the SQL server. It loaded the table just fine from SQL, then failed on the aggregation.