[Data Integration][Pipeline Project][S3 Landing] Does the Data Landing Task use compressed data transfer when uploading CSV files to S3?
Hello Support team
I am creating a Data Pipeline project with the following connections.
・Source : Oracle
・Storage : Snowflake
・Landing : AWS S3
Does the Data Landing Task use compressed data transfer when uploading CSV files to S3?
I ran the data landing task and confirmed that the CSV files actually stored in S3 are not gzip-compressed.
Therefore, I understand that the Data Movement Gateway server is not simply gzip-compressing the CSV files and uploading them as-is.
Could you let me know if Data Landing Task is implementing any measures (for example such as those listed below) to reduce the amount of data transferred during upload?
- The data being transferred is automatically compressed by the API used by The Landing Task.
- The Landing Task first sends a gzip-compressed file, decompresses it on S3, and then re-uploads the decompressed file.
Also, if data transfer is not compressed by default, please let me know if there are any settings to enable compression.
The Data Landing Task does not gzip-compress CSV files during upload to S3, and it does not send a gzip file and decompress it on S3 afterward. The files stored in S3 are uploaded in their generated CSV format.
There is no default setting in the Data Landing Task to enable gzip compression for the transfer. If compression is needed, it would need to be handled separately (for example, by compressing the files after landing).
The Data Landing Task (via the Data Movement Gateway) does not compress CSV files by default when uploading to Amazon S3—files are sent as-is, without gzip or other compression.
Nor does it:-
Automatically compress via any built-in S3 API feature.
Upload a gzip, decompress on S3, then overwrite the CSV.
There are no settings in the Landing Task for enabling transfer compression on CSV.
Options to Reduce Transfer or Storage Size:-
While CSV uploads remain uncompressed, you can lower data size or transfers by:-
Switching to Parquet format:- Parquet is columnar and supports built‑in compression (e.g., Snappy, GZIP), reducing both transfer and storage footprint.
Post-upload compression:- Use AWS Lambda or S3 Batch to gzip CSVs after upload, though this adds pipeline complexity.
The Data Landing Task does not gzip-compress CSV files during upload to S3, and it does not send a gzip file and decompress it on S3 afterward. The files stored in S3 are uploaded in their generated CSV format.
There is no default setting in the Data Landing Task to enable gzip compression for the transfer. If compression is needed, it would need to be handled separately (for example, by compressing the files after landing).
As far as I understand, what Rahul_Kale is referring to as “Switching to Parquet format” cannot be done within the landing tasks of a “Data pipeline project.”
