use of iterateToFlow and FlowToIterate with ExtractJSONFields

[SKEDA]

Hi there, I'm trying to export data from clockify.me a time-keeping application. Part of the requirement is to get a list of projects using a REST call, and then make a REST call for each Project to get the list of Tasks for that project. From that I need the amalgamated list of ProjectId, TaskId, TaskName to feed into a tMap. To complicate things, clockify only allow 10 API calls per second so I need to put a delay/sleep in the 2nd API calls that lookup the Task info for each Project.

I have used designed like RESTcall1 -> ExtractJSONFields (projectId) -> FlowToIterate-> iterate -> Sleep (1s) -> RESTcall2 -> ExtractJSONFields (projectId, taskId, taskName) - tLogRow

I get the expected output in my tLogRow but it's separate outputs (I assume as it's an iteration)... if I connect this into tMap it doesn't work (lookup never matches). In tMap I am mapping Projects, Tasks, and Times to an output xls. The Times have ProjectId and TaskId, with lookup on Projects (Times.projectId=Project.projectId), and lookup on Tasks ( Times.projectId = Tasks.projectId & Times.TaskId=Tasks.TaskId).

I think this is because I need to use an iterateToFlow or some other way of mashing the iterative output back into a single flow ?

I got around the issue by doing the iterative part in a subJob and outputting to a csv file, then in the parent I open the file and use it as a lookup on my tMap.

It works but feels pretty cludgy. Is there a way to get it to work without dumping to a file and reloading?

I attempted to use an iterateToFlow after the 2nd API Call but I couldn't figure out what columns/values I had to put there. The ExtractJSON didn't seem to create any globals and if I entered the column names of the ExtractJSONField schema into the Values in the iterateToFlow it didn't seem to work (lookups failed to match anything). I'm not even sure it's the right component to use? - see attempt.png attached for an example of this.

[Anonymous · Accepted solution]

Hi

Store the output into memory instead of a file?

tHashOutput replace tFileOutputDelimited

tHashInput replace tFileInputDelimited

Go to add them to palette if they aren't available, please refer to this knowledge article.

Regards

Shong