Talend Open Studio Scripts Not Working

[lizzy1]

I recently updated my Talend Open Studio from 7.x to 8.x. All of my talend scripts run from batch files on a server, triggered by a scheduled task.

Currently, when I personally run the .bat files, as an administrator, they run perfectly. But when I run them as a scheduled task, they do not work. All of my jobs output to Google BigQuery, but their sources are varied, including REST APIs, SOAP APIs, and Microsoft SQL databases.

Most of my jobs consist of a component to drop a BigQuery table, retrieve the data from the source, and then write a BigQuery table. The tables do get dropped, so I'm sure the job is being triggered, but no new data gets written. The BigQuery output component is set to Append, and the box is checked to create the table if it doesn't exist.

The issue appears with all of my jobs. The scheduled task runs with maximum privileges and has not changed since I upgraded talend (and java). The task history says it has run with no issues. I'm sure the environment variables are set correctly, since the scripts run correctly by hand.

What else should I try to troubleshoot this problem?

[lizzy1 · Accepted solution]

I believe I have figured this out myself. It's down to the specific way that the BigQuery components worked in 7.x.

In the previous version, to drop a table, I used tBigQueryInput to execute "drop table". This always created an error, so it was attached to the rest of the job via "On Component Error". It was a hack, but it worked. In the current version, doing the drop table does not result in an error (this is, generally speaking, a good thing!).

So when the job was run as a scheduled task, the table was dropped, and since no error happend, the rest of the job didn't run. When I was running it manually to recover the missing table, the table did not exist, so tBigQueryInput did produce an error, so the rest of the job did run.

I am fixing the jobs by using the new "truncate" option in tBigQueryOutput, instead of having to use the input component to drop the table.

[Anonymous]

Hello,

Thanks for letting us know you have resolved this issue by yourself.

Best regards

Sabrina