Pass Column Schema Through Component Without Using It (Use It In "next" Component)

[jdope]

Goal:

Take source data from one system and migrate to a second system.  After the data is inserted into second system; correlate the "source id" with the new "destination id". 

Ultimately the question is how can I have a column that is ignored by one component (putData) and simply passed through to the next component (correlateMap)? 

srcData: has a "source id" that cannot be used by putData, but needs to be correlated to the RESULT of putData; so although it should not be used by putData it needs to be available in correlateMap.

correlateMap

Reconcile the "old" id with the "new" id; however "Id_Ignore_In_putData" is null because I had to remove the data for putData to work correctly...

I cannot figure out how to simply pass a column along without actually using it...any tips greatly appreciated. 

[TRF]

Hi,

I'm afraid you cannot (or if it's possible, I'd like to know how).

So, you have to remove the undesired column before to arrive in the tSalesforceOutput component, then, using a tMap join with the original input, retrieve the plain record to go to tFileOutputDelimited.

This is a common situation.

[Anonymous · Accepted solution]

You can do this, but it will be slower. Connect your source to a tFlowToIterate. All of your column's values will be stored in the globalMap. Connect the tFlowToIterate to a tFixedFlowInput and set up every column (but the one you don't want to send) in the tFixedFlowInput. Assigne the columns with the values stored in the globalMap. The pass them to your put component. Then after that, just add the removed column back in with a tMap. The globalMap variable value will still be valid for this row.

This will be slower than allowing the data to process normally, but it will work.

[TRF]

I agree with the solution proposed by @rhall but as said, be careful in case of high volume.

In this case, you need to remove the undesired field and retrieve if from the original flow when needed.

Using a CSV file or a tHash is usually the solution to store the data, but of course you need a key to retrieve the expected record!

[Anonymous]

Do you have another "key" in your data? Is there are a combination of fields that would act as a key? If that is the case you can do what @TRF has mentioned (using the Hash components) and just join your ID back in using the key field(s). Otherwise you are stuck with the other way I mentioned. 

[jdope]

Thanks for all the replies, I very much appreciate it.  

Yeah I thought about just using another tMap to "rejoin" the data after the component, but there is no guarantee of a composite unique key (and the only actual key is being lost because it has to be dropped prior to the putData component).

It sounds like the approach is to use some un-ideal overhead (as mentioned by rhall_2_0).  Seems surprising that is not part of the standard component architecture.  

Thanks again!