Implicit Context Load issues and no end

Anonymous — Wed, 24 Oct 2012 10:05:06 GMT

Hi *,
looks like I am too stupid for this.
I am running "Talend Open Studio for Data Integration", Version: 5.1.2, Build id: r90681-20120913-0734.

Like so many developers I need to run a migration in two different environments, one on Windows (development, from my workstation), one on Unix (production).
For testing purposes I have defined two Unix users, "migdev1" and "migdev2", with home directories "/export/home/migdev1,2"
The directory structure on the windows environment mirrors the one of "migdev1".
In Talend I have defined a master job that calls several subjobs. In the project settings I have activate "Implicit context load" from fille with filename "context.PARAMS_FILE". In the "Use project settings" I have checked all relevant subjobs. See images.

In the default context I have set
PARAMS_FILE=/export/home/migdev1/conf/irmigration.ctx
to fit the Windows environment.
On windows the masterjob runs error free. On Unix I have two run situations:
as user "migdev1" in "/export/home/migdev1" with config file "/export/home/migdev1/conf/irmigration.ctx"
and
as user "migdev2" in "/export/home/migdev2" with config file "/export/home/migdev2/conf/irmigration.ctx"
Running the exported masterjob on Unix as user "migdev1" works fine, as it basically is the same context environment as on Windows.
Running the exported masterjob on Unix as user "migdev2" fails, because although the masterjob pulls the correct context file, the subjobs do not "inherit" the correct variable values nor do they read the correct config file:
Definitions in Talend-Projext/Contexts/Default:

PARAMS_FILE          /export/home/migdev1/conf/irmigration.ctx
DISCARD_FILE_DIR     /export/home/migdev1/log/discard
ERROR_FILE_DIR       /export/home/migdev1/log/error
LOG_FILE_DIR         /export/home/migdev1/log
REJECT_FILE_DIR      /export/home/migdev1/log/reject

Unix Shell Script:

export PARAMS_FILE="$HOME/conf/irmigration.ctx"
/bin/bash $1_run.sh --context_param PARAMS_FILE=$PARAMS_FILE

Masterjob, outputs with tJava.System.out.println():

***  A000_Migration_Master Version 1.0.0.12 as of 23/10/2012: Start Migration ...
***  LOG_FILE_DIR     = /export/home/migdev2/log
***  REJECT_FILE_DIR  = /export/home/migdev2/log/reject
***  DISCARD_FILE_DIR = /export/home/migdev2/log/discard
***  ERROR_FILE_DIR   = /export/home/migdev2/log/error
*** 
***  Subjob called by masterjob, outputs with tJava.System.out.println():
*** 
/export/home/migdev1/conf/irmigration.ctx (Permission denied)
***  A0005_Display_Configuration: current actual context
***  ---------------------------------------------------
***  Runtime context parameters
***  ---------------------------------------------------
***  PARAMS_FILE      = /export/home/migdev1/conf/irmigration.ctx
***  ---------------------------------------------------
***  DISCARD_FILE_DIR = /export/home/migdev1/conf/irmigration.ctx/logs/discard
***  ERROR_FILE_DIR   = /export/home/migdev1/logs/error
***  LOG_FILE_DIR     = /export/home/migdev1/logs
***  REJECT_FILE_DIR  = /export/home/migdev1/logs/reject
***  ---------------------------------------------------

The subjobs obviously try to access the param file in "migdev1", which they can't open because of rights restrictions.
Why does the masterjob find the right config file, and the subjobs do not?
Any suggestions?
Frustrated,
Steve

Re: Implicit Context Load issues and no end

Anonymous — Wed, 24 Oct 2012 12:28:44 GMT

Ok,
I think I have found a solution outside Talend.
Directory structure on the development system:

./conf   -> config file
./log    -> log files in subdir structure
./bin    -> shell scripts

1) change all paths in the default context from absolute to relative paths, like
"/export/home/migdev1/conf/irmigration.ctx" to "./conf/irmigration.ctx"
or
"/export/home/migdev1/log/reject" to "./log/reject"

2) export the masterjob (here: "A0001_Migration_Masterjob_0.1") from Talend into a ZIP and copy/ftp it to the Unix file system into the HOME directory of the migration user.

3) log in on the Unix system as the mig user and run the bin/run.sh script with the masterjobs name ("A0001_Migration_Masterjob") as parameter:

#!/bin/bash
# ---------------------------------------------------------------------------------
# Name    : run.sh
# Purpose : extract and run TALEND jobs from ZIP archives
# Author  : sse/tjo
# Version : see below
# Date    : see below
# Comment : none
# ---------------------------------------------------------------------------------
export VERSION=1.0.0.12
export STAND=24/10/2012
export LOG_FILE_DIR=$HOME/log
cd $HOME
# ---------------------------------------------------------------------------------
echo
echo
echo ">>> run.sh: Version $VERSION as of $STAND"
# ---------------------------------------------------------------------------------
# Test: Parameter 1 = Talend Job name : must not be null
# ---------------------------------------------------------------------------------
if 
then
		echo ">>> usage: run.sh TALENDJOB"
		exit -1
fi
cd ~/jobs
# ---------------------------------------------------------------------------------
# Unpack archive
# ---------------------------------------------------------------------------------
if 
then
		echo ">>> run.sh: auto extract $1 archive ... "
		rm -rf $1_0.1
		unzip -u -qq $1_0.1.zip
fi
# ---------------------------------------------------------------------------------
# Check if TALEND run script exists
# ---------------------------------------------------------------------------------
if 
then
		echo ">>> No shell script $1_0.1/$1/$1_run.sh found."
		exit -1;
fi
echo ">>> run.sh: running $1_0.1/$1/$1_run.sh"
echo
# ---------------------------------------------------------------------------------
# Run script
# ---------------------------------------------------------------------------------
cd $1_0.1/$1
datum=`date +'20%y%m%d-%H%M%S'`
# link the config and log directories from $HOME down to the jobs work directory
rm -rf ./conf ./log ./data
ln -s $HOME/conf .
ln -s $HOME/log  .
ln -s $HOME/data .
/bin/bash $1_run.sh  2>&1 | tee $LOG_FILE_DIR/run-$datum.log
echo
echo ">>> Done."
# ---------------------------------------------------------------------------------
# End of file run.sh
# ---------------------------------------------------------------------------------

So in the end I linked/copied the config file down to the jobs run directory, thereby making it visible for the jobs context loads.
The shell script might need some adaption (like: unzip syntax et al)
Steve

topic Re: Implicit Context Load issues and no end in Talend Studio

Implicit Context Load issues and no end

Re: Implicit Context Load issues and no end