tFileArchive: zip all files in directory into one single zip file

[Anonymous]

I have a few thousands files in a directory ( files are in different subfolders) and I want to zip all the files into a single .zip file.

Here is what I already did:

tFileList --- tFileCopy      (I copied all the files in the directory into a temporary folder)

|  OnSubjobOk

tFileArchive                     (I took all the files in that temporary folder and archived them into one single .zip file).

The whole process takes less than 1 minute and did what I want.

What I don't like is I have to create that extra temporary folder to hold all the files first.

So I was thinking of a different way to do it:

tFileList ----- iIterateToFlow ---- tFileArchive  

The process would run and created the .zip file but it seems to take forever to complete ( I killed the job before it completed).

Why this process is so so slow? Is it a correct way to do the same job?

Thanks!

TM

[Jesperrekuh]

Depends on the level of compressing, like fast or store. The latter is the one I often use because when moving / archiving / backing up they also use compression mechanism... so why wasting time in jobs
But It depends on your filetypes/structure, example jpg files don't compress well because they're already compressed.

The kind of limitations within the archive component I don't really know.
You could use tSystem and zip / rar / tar what files and how (store or fast) you want it.

Design an independent process to run at the end and doesnt interfere with other jobs.

[Anonymous]

Hi Dijke,

Within the three thousands source files, most of them are .xlsm, only a few are .xls files.

Talking about the 2nd method I tried, the connection between tIterateToFlow and tFileArchive is row(Main), but it looked like the job was looping through the files ONE BY ONE. It seems strange to me.

BTW, did you mean my 2nd method is also a correct one to go?

[Morpheus]

Are there other files in the directory that you don't want to include in the zip file?

[Anonymous]

No. Basically I want to create a .zip file for ALL files from the directory. So in both methods, "All Files" was checked in tFileArchive component.

[Anonymous]

Hi,

     You can zip the details in single run by selecting the sub directory option (if all the files are under single parent directory which you want to zip in one go). The description says,

Select this check box if you want to add the files in the subdirectories to the archive file.

This field is available only when zip is selected from the Archive format list.

    But please note that it will maintain the directory structure as many scenarios require the files to be zipped with directory structure intact (example, all log files for a week which are stored in daily directories are zipped and archived).

    In this situation, you can avoid the copy step and iteration step. Could you please try this also?

Warm Regards,

Nikhil Thampi

[Anonymous]

tFileList ----- iIterateToFlow ---- tFileArchive

Hi nikhilthampi,

In the above job, I did have Subdirectories checked on tFileArchive component.

Otherwise, the process would NOT create the .zip file in the target path at all (because my files are under the different subdirectories).

Thanks!

TM

[Jesperrekuh · Accepted solution]

@Moe,

I was talking about the tSystem... with zip command... Works better then the tArchive:
Example: https://sevenzip.osdn.jp/chm/cmdline/commands/update.htm
With the u update function you are able to add to an existing zip.

Your reasoning/principal/point of view I get it... you don't want to move to a temp, just unnecessary disk io
, why not? Honestly I have the same strategy, move first then archive and create a folder with a date and the job's process id. When needed (for whatever reason) Im able to exactly extract the correct data and trace it back to system logs. This is what is referred as cold storage.

From an infrastructural point of view ask for specific cold storage ... or move it to hadoop / hdfs...

Be schwifty )