Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Papademuchos
Creator
Creator

Thanks shong for your response - that's helpful. After playing around a bit more I was able to figure out how to do what I want using tExtractDynamicFields and adding in a hard coded list of columns that I want to ignore in a tMap that follows. But the blog shows an option that would let me play with the columns based on column properties. I'll circle back around and try that out!

 

For anyone else that finds this thread and may be wondering what I did, here are my steps:

  1. tDBInput to read all rows from my source table
  2. tExtractDynamicFields lets me list out the columns I want and keep all other columns in a dynamic field
  3. tMap let's me indicate which fields to pass through
  4. tDBOutput to Insert or Update to my target table

 

 

1 tDBInput: - use a "Select *" to retrieve all columns

My schema looks like this:

In my example, all 140 tables have id, code & description. The tDBInput component is smart enough to load those columns into my defined columns, and put "everything else" into the "other_columns" dynamic field.

 

2 In the tExtractDynamicFields component I've included my common fields, all of the potential "computed" type fields that might be present in my tables, and the "other_columns at the end. Again, Talend is smart enough to extract those columns present in my source "other_columns" dynamic field, map them to the new column name that matches, and leave everything else in the "other_columns" field at the end.

 

 

3 In tMap I'm simply leaving the computed columns behind and only passing through the ones that I want. "other_columns" has everything else for that particular table.

 

 

4 Finally tDBOutput does an Insert of Update with known columns, and the unknown columns ("other" -> dynamic type)

 

I hope this helps someone! It took me a bit of research to put it together.