topic store the dynamic values in context variable in Talend Studio

store the dynamic values in context variable

Anonymous — Thu, 03 Aug 2017 08:34:00 GMT

Hi team,

I have source oracle table and there is one column called txn_date.

values are:

txn_date

02-06-2017

04-06-2017

07-06-2017

04-06-2017

Requirement: How to create folder based on date for examples:

folder 1: c;/Txn_dt_02-06-2017/

folder 2 :c:/Txn_dt_04-06-2017/

folder 3 :c:/Txn_dt_07-06-2017/

But no duplicate folder even if same records with same date arrive.

and is possible to store single context variable to store unique txn_dt values?

Thanks

Shree

Re: store the dynamic values in context variable

TRF — Thu, 03 Aug 2017 09:05:46 GMT

Hi,

You can set a context variable value dynamically at runtime (context.yourVariable = "whatYouWant") but usaully for this king of usecase it's preferable to set a global variable.

Your design process should look like:

tOracleInput -> tUniqRow (eliminates doublons on date values) -> tJavaRow (here set the desired global variable) -> tSystem (to create directory)

To set the global variable, use the following:

globalMap.put("outputFolder", inuput_row.txn_date);

As there is not dedicated somponent to create an empty folder, use tSystem for this purpose with the following command:

"cmd /c mkdir " + ((String)globalMap.get("outputFolder"))

Hope this helps.

Re: store the dynamic values in context variable

Anonymous — Thu, 03 Aug 2017 09:06:31 GMT

Hi Shree
Do you want to store the source data to file under the folder or just create a folder without writing data?

Regards
Shong

Re: store the dynamic values in context variable

Anonymous — Thu, 03 Aug 2017 09:23:23 GMT

I see where @shong is going with the above. If you are intending to add data to the folder, there is no need to worry about this problem.

1) You cannot create a folder twice. Once it exists, it exists

2) When you want to write a file to a folder in Talend, you can set the file to create the folder if it does not already exist

Unless there is some other requirement here, you really do not need to worry about this.

Re: store the dynamic values in context variable

Anonymous — Thu, 03 Aug 2017 09:55:18 GMT

Hi Shong.

Sound good, if we can store records as well based on dates. ex: 4th day records-> create folder(04-06-2017)/txn_dt_4th_data.csv. similar to other days as well.

Means time need to store the date values in context variable in single or mutiple context variable.

After that, i am going to use this context variable for some excel file creation based on some columns like gender='M' etc.

Thanks

Shree

Re: store the dynamic values in context variable

Anonymous — Thu, 03 Aug 2017 10:42:06 GMT

Hi,

As per suggestion whether it will create distinct folder based on date unique values?

Thanks in advance.

Regards

Shree

Re: store the dynamic values in context variable

TRF — Thu, 03 Aug 2017 12:36:37 GMT

I think I gave you the answer (or not?)

Re: store the dynamic values in context variable

Anonymous — Fri, 04 Aug 2017 05:43:04 GMT

Hi,

I have tried as your input.

I have facing error while creating folder using tSystem component. I have attached screen shot for references.

One more point rather than using tUniquerow , we can override distinct in tOracleInput itself?

Error message:

A subdirectory or file null already exists.

Thanks

Shree

error.PNG

Re: store the dynamic values in context variable

cterenzi — Fri, 04 Aug 2017 16:15:45 GMT

The components that create output files have an option "Create directory if does not exist"

Better to use those so that the only directories created are ones with files in them. The component will only create a directory once.

Re: store the dynamic values in context variable

Anonymous — Tue, 08 Aug 2017 05:33:16 GMT

Hi,

Still i am facing same error.

I am attaching job for your reference. Could you please check it.

Thanks

Shree

mutiple_folder_creation.zip

Re: store the dynamic values in context variable

Anonymous — Tue, 08 Aug 2017 09:29:14 GMT

I'm afraid I cannot open your job since I am currently using an older version of Talend. But it is clear that you are trying to create a folder that already exists using the mkdir command. Instead of using tSystem to create directories why not use a tLogRow to print out the directories it is going to create to see how the duplication occurs.

Also note, you are trying to create a directory called "null". This happens in Java when you are concatenating Strings and one of those Strings is null.

You need to refine your data collection and data validation.

Re: store the dynamic values in context variable

Anonymous — Tue, 08 Aug 2017 10:35:18 GMT

Hi,

Still i am getting same error.i have tried with tLog_row, i getting null, How to track. i am attaching screen shot for references.

tJavaRow_1:

globalMap.put("C:/outputFolder", input_row.newColumn1);
output_row.newColumn2 = input_row.newColumn2;

Any inputs would be helpful.

Thanks

Shree

error11.PNG

Re: store the dynamic values in context variable

Anonymous — Tue, 08 Aug 2017 10:41:26 GMT

You do not appear to be passing the value of input_row.newColumn1 to your output. You are setting your globalMap value (strangely to a key of "C:/outputFolder"....why?), but not passing the value to the output_row.newColumn1 column. Your tLogRow will be displaying this. We will need to see your tSystem configuration.

Re: store the dynamic values in context variable

Anonymous — Tue, 08 Aug 2017 12:16:59 GMT

Hi,

Please find the attached screen shot.

Easy to trace where i am doing wrong.

Thanks

Shree

2.PNG
1.PNG

Re: store the dynamic values in context variable

Anonymous — Tue, 08 Aug 2017 12:56:50 GMT

OK, I think I know what is going wrong. But it is difficult to tell for sure, so you will need to try this.

1) Disconnect your tJavaRow from your tUniqRow

2) Connect a tFlowToIterate to your tUniqRow

3) Use an iterate link from the tFlowToIterate to your tSystem component

4) In your tSystem component use the following code....

"cmd /c mkdir " + ((String)globalMap.get("row2.newColumn1"))

FYI the tFlowToIterate works by adding the row to the globalMap using the row name and column name as the key. Each iteration keeps those values for the whole of the flow of the iteration.

Re: store the dynamic values in context variable

Anonymous — Wed, 09 Aug 2017 05:27:33 GMT

Hi ,

Thanks for suggestion.

Now, able to created directory based on unique txn_date.

Observation: if directory already exist it through error.

error msg: A subdirectory or file 07-12-0012 already exists.

2) Year is not properly value. i will check this one.

Final Requirement:

Source records:

Columns are:

txn_date, num

12-06-2017,12

13-06-2017,24

12-06-2017,22

14-06-2017,24

12-06-2017,13

13-06-2017,24

tOracleInput_1 ==> tFileOutputExcel_1

1) Create unique txn_dt directory. Ex: 12-06-2017, 13-06-2017 etc

2) Inside that directory create xlsx file. date_wise_load.xlsx ( like 12-06-2017)

3) Load that date records.

Output File:date_wise_load.xlsx

txn_date,num

12-06-2017,12

12-06-2017,24

12-06-2017,13

4) If directory already exist though error and send notification either email. Keep job running.

Thanks once again for right direction

Regards

Shree

Re: store the dynamic values in context variable

Anonymous — Wed, 09 Aug 2017 09:25:48 GMT

You do not need to do this. Below is a MUCH simpler way of achieving your goal.

1) Create the folder path String as before

2) Do not use a tSystem, use a tFileTouch component to create a file (you choose the name) in the folder you want to create. Select "Create direcory if does not exist".

3) Delete the file you have just created

There is no need to keep track of what folders you have create since this method will not error if you recreate the same folder.....because you cannot do that this way.

Re: store the dynamic values in context variable

Anonymous — Wed, 09 Aug 2017 10:18:49 GMT

Hi ,

Sorry, i am missing some loops now.

I have attached screen shots.

Could you please check and tell me the flow.

Thanks

Shree

Folder creation_2.PNG
Folder creation_1.PNG

Re: store the dynamic values in context variable

Anonymous — Wed, 09 Aug 2017 10:22:09 GMT

At the end of your path you need to add a filename (maybe "del.me"). This will create a file inside the folder. At the moment it is assuming your folder is a file. Once the "del.me" file is there, you can delete the "del.me" file.

Not sure what you meant about loops?

Re: store the dynamic values in context variable

Anonymous — Wed, 09 Aug 2017 10:57:55 GMT

Hi,

I am using below flow.

tfiledelimited_1 => tUniqueRow_1 => tFlowToIterate_1 => tFileTouch_1

tFileTouch_1:

Properties:
File: "C:/Users/DELL PC/Desktop/reports/SOURCE FILE 3/" + ((String)globalMap.get("row2.newColumn2"))
Checked: create direcotry if does't exist

I am able to create file with empty records. but folder are created.

Please let me know any changes required?

Thanks

Shree