Loading File/s from SFTP location

[nabeelaslam1994]

Hello Team,

I am trying to check and see how can i load file/s from an SFTP location to Snowflake? I also have a stream of files coming in daily to be loaded to snowflake. Which source connection type should i be using?

Thanks,

Nabeel

[john_wang · Accepted solution]

Hello @nabeelaslam1994 ,

Thanks for reaching out.
For the file source endpoint, I do not think it's a good approach to let Replicate read files from an SFTP location because:
1- Replicate cannot request FTP/SFTP to get/pull new files. the files should be push/put to Replicate Server local OS folder for better post-processing operations (change files cleanup etc)
2- To avoid Replicate detects a new file and start to read the incomplete file/s (which the ftp transfer not finish yet eg the big files or slow network throughputs), the file should be transferred to a temporary file then RENAME file extension to ".txt" or ".csv" (Configure Replicate monitor *.txt or *.csv files only), or reference file should be used to tell Replicate new files coming. These requires a local OS folder too.

Hope this helps.
Regards,

John.