My source fields are Datetime. in format YYYY-mm-dd HH:mm:ss. My target Database Redshift also has this field defined as Timestamp.
1.So if i read the source field as "String" datatype then the date values get loaded correctly.
2.But if i read them as date and load the data to my target table,then there is an offset of 1 hour added to these values. To handle this, i have added
-Duser.timezone=UTC
the above condition and then date values get reflected correctly. But i have added another field in my stage table i.e Load_at to track the current data load time. It also shows up the time in UTC timezone and not the current timezone.
SO which approach is correct? Should i always read date fields from source system as "string" datatype and load OR use the above function -Duser.timezone=UTC to correct my date time before loading my target Tables? Both the approach give same result.
