how to store the folder name which us having date? when the job run its should start picking from the last extrated date.(incremental load).?

[amit_n]

Extracting the app insights data from client azure blob storage to local bob storage. Data is in the folder which is having date. First run we need to extract full app insights data from blob storage to local blob storage. From the next run job should load incremental data based on the last run date. Please let me know how to storage the last run date. next run it should be incremental. 

folder name:

2018-07-20

2018-07-21

folder will be having insights data.

[vapukov]

hi, 

so, what is your question?

- how store last result date? Answer - just save it somewhere - file or database

- how to store folder name which us having date? need to transform it to more clean form

[TRF]

You can store context variables into a CSV file for this purpose.
Check this link where I give an example you can follow to construct your own solution https://community.talend.com/t5/Design-and-Development/store-the-dynamic-values-in-context-variable/...

[amit_n]

Actually i am not using any values inside the file. i am extracting the app insights data based on folder name which is having date.

folder name: 2018-07-01

based on the folder name i need to do incremental load.

next day it will pick the 2018-07-02 date folder.

[vapukov]

Hi

this is exactly what was suggested:

- finish your work

- save folder name into csv file (or database)

for initial iteration it could be any date from which You want begin - 2001-01-24

next day:

- read csv file (or database)

- start loop from extracted date +1 day until not today (there You will need convert string to date and back to string and use dateadd functions)

- save new value after finishing Job

that all

[amit_n]

Please find the below screen shot.

In the pic . getting the data from the azure blob.

tazurestorageList = data in the folder 2018-09-30

                                                            2018-10-01

                                                            2018-10-02

inside the folder data file with .blob extension

tazurestorageGet = get the files along with the folder.

tazurestoragePut = put the data from client blob to local blob storage.

Here i have to store the folder name which is last extracted - 2018-10-02

From tomorrow i need to start from the last run. date+1

Capture1.JPG