In Spark Batch and Spark Streaming, you need to think and design differently due to the multi-parallel processing of these frameworks. You will use context variables only to initialise the job. And the initialisation should happen as soon as the job begins, meaning these variable values should be passed to the job from the TAC. This is because a Spark Batch or Spark Streaming job will run on many nodes, and will have many executor context. Hence, you cannot rely on context variable, because it is not global to all the executor context anymore. Once some spark logic start executing in an executor context, any attempt to manipulate the context variable will be local to that executor context only, and not the whole job. That's why we do not provide these components in the Spark Batch and Spark Streaming so as to avoid Talend developers from using an antipattern.
You need to figure out how to read variables. An example can be to read it from a database and load the values into an RDD. You can then access the RDD from all your nodes. It will be in memory and fast. You can then read from the RDD and apply the value to globalmap etc. We have not abstracted this logic yet as it creates an RDD. Since RDD is immutable, each time you update a variable value in the RDD, you will create another RDD.
In the same line of thought, you cannot use AMC in Big Data Batch/Streaming jobs.
Once some spark logic start executing in an executor context, any attempt to manipulate the context variable will be local to that executor context only, and not the whole job. That's why we do not provide these components in the Spark Batch and Spark Streaming so as to avoid Talend developers from using an antipattern.
of course this all only visions, but I think - Yours (Talend) position is also not a 100% correct
1) You are use context variables in all Your examples as well (and for batches and for real time streaming) 2) most of time we are (all other world) also use context for initialize job on start - database connection information, ip-addresses, folders, login information (99% of context usage) 3) TAC - store context variables individual for Job
in this case You are also provide also bad anti-patterns: - or have in Job pre-defined PRODUCTION group as well as DEV, and store PRODUCTION values on DEV environment - or setup it individual on PRODUCTION server after implementing
why not allow initialize context same as for ETL? I understand what are You mean - sometime in ETL developers use context.variable as variable in a middle of Job
But must bigger number of them have DEV/QUAL/PROD (I meat up to 7) of context groups defined in Repository and after this - half of them store all values together, and other half - manually set values manually after implementing
May be I not understand, logic, but I not found this anywhere in Talend docs or threads - ok, we have 100+ jobs implemented in TAC, and for some security reasons decide change database connection password for ETL jobs all simple - change on server csv (or in a table) but what about: - routes - batch jobs - stream
yes, it could require - kill and restart for catch new values, but still much more easy rather then edit all in TAC, or redeploy all jobs
Just think about this side of the problem, simple examples with simple context group and password hadoop/hadoop - not always work in real life 🙂
I had to execute the redundant query in multiple components since we dont have a coponent available for storing the results and using it else where. I create a hive component executed the query and stored the results in rdd and assigned the results to GlobalMap. The value is one two columns and one row and it will not change through out the job. The hive component writes to tjava and global map get assigned. I used on ok subjob to call the other hive components where globalmap values are used.
