Hi *,
looks like I am too stupid for this.
I am running "Talend Open Studio for Data Integration", Version: 5.1.2, Build id: r90681-20120913-0734.
Like so many developers I need to run a migration in two different environments, one on Windows (development, from my workstation), one on Unix (production).
For testing purposes I have defined two Unix users, "migdev1" and "migdev2", with home directories "/export/home/migdev1,2"
The directory structure on the windows environment mirrors the one of "migdev1".
In Talend I have defined a master job that calls several subjobs. In the project settings I have activate "Implicit context load" from fille with filename "context.PARAMS_FILE". In the "Use project settings" I have checked all relevant subjobs. See images.
In the default context I have set
PARAMS_FILE=/export/home/migdev1/conf/irmigration.ctx
to fit the Windows environment.
On windows the masterjob runs error free. On Unix I have two run situations:
as user "migdev1" in "/export/home/migdev1" with config file "/export/home/migdev1/conf/irmigration.ctx"
and
as user "migdev2" in "/export/home/migdev2" with config file "/export/home/migdev2/conf/irmigration.ctx"
Running the exported masterjob on Unix as user "migdev1" works fine, as it basically is the same context environment as on Windows.
Running the exported masterjob on Unix as user "migdev2" fails, because although the masterjob pulls the correct context file, the subjobs do not "inherit" the correct variable values nor do they read the correct config file:
Definitions in Talend-Projext/Contexts/Default:
Masterjob, outputs with tJava.System.out.println():
*** A000_Migration_Master Version 1.0.0.12 as of 23/10/2012: Start Migration ... *** LOG_FILE_DIR = /export/home/migdev2/log *** REJECT_FILE_DIR = /export/home/migdev2/log/reject *** DISCARD_FILE_DIR = /export/home/migdev2/log/discard *** ERROR_FILE_DIR = /export/home/migdev2/log/error *** *** Subjob called by masterjob, outputs with tJava.System.out.println(): *** /export/home/migdev1/conf/irmigration.ctx (Permission denied) *** A0005_Display_Configuration: current actual context *** --------------------------------------------------- *** Runtime context parameters *** --------------------------------------------------- *** PARAMS_FILE = /export/home/migdev1/conf/irmigration.ctx *** --------------------------------------------------- *** DISCARD_FILE_DIR = /export/home/migdev1/conf/irmigration.ctx/logs/discard *** ERROR_FILE_DIR = /export/home/migdev1/logs/error *** LOG_FILE_DIR = /export/home/migdev1/logs *** REJECT_FILE_DIR = /export/home/migdev1/logs/reject *** ---------------------------------------------------
The subjobs obviously try to access the param file in "migdev1", which they can't open because of rights restrictions.
Why does the masterjob find the right config file, and the subjobs do not?
Any suggestions?
Frustrated,
Steve
1) change all paths in the default context from absolute to relative paths, like
"/export/home/migdev1/conf/irmigration.ctx" to "./conf/irmigration.ctx"
or
"/export/home/migdev1/log/reject" to "./log/reject"
2) export the masterjob (here: "A0001_Migration_Masterjob_0.1") from Talend into a ZIP and copy/ftp it to the Unix file system into the HOME directory of the migration user.
3) log in on the Unix system as the mig user and run the bin/run.sh script with the masterjobs name ("A0001_Migration_Masterjob") as parameter:
#!/bin/bash # --------------------------------------------------------------------------------- # Name : run.sh # Purpose : extract and run TALEND jobs from ZIP archives # Author : sse/tjo # Version : see below # Date : see below # Comment : none # --------------------------------------------------------------------------------- export VERSION=1.0.0.12 export STAND=24/10/2012 export LOG_FILE_DIR=$HOME/log cd $HOME # --------------------------------------------------------------------------------- echo echo echo ">>> run.sh: Version $VERSION as of $STAND" # --------------------------------------------------------------------------------- # Test: Parameter 1 = Talend Job name : must not be null # --------------------------------------------------------------------------------- if then echo ">>> usage: run.sh TALENDJOB" exit -1 fi cd ~/jobs # --------------------------------------------------------------------------------- # Unpack archive # --------------------------------------------------------------------------------- if then echo ">>> run.sh: auto extract $1 archive ... " rm -rf $1_0.1 unzip -u -qq $1_0.1.zip fi # --------------------------------------------------------------------------------- # Check if TALEND run script exists # --------------------------------------------------------------------------------- if then echo ">>> No shell script $1_0.1/$1/$1_run.sh found." exit -1; fi echo ">>> run.sh: running $1_0.1/$1/$1_run.sh" echo # --------------------------------------------------------------------------------- # Run script # --------------------------------------------------------------------------------- cd $1_0.1/$1 datum=`date +'20%y%m%d-%H%M%S'` # link the config and log directories from $HOME down to the jobs work directory rm -rf ./conf ./log ./data ln -s $HOME/conf . ln -s $HOME/log . ln -s $HOME/data . /bin/bash $1_run.sh 2>&1 | tee $LOG_FILE_DIR/run-$datum.log echo echo ">>> Done." # --------------------------------------------------------------------------------- # End of file run.sh # ---------------------------------------------------------------------------------
So in the end I linked/copied the config file down to the jobs run directory, thereby making it visible for the jobs context loads.
The shell script might need some adaption (like: unzip syntax et al)
Steve
