Abstract question regarding scheduled automated transfer of files between a Linux device & Windows (for use as Qlik data source)

[evan_kurowski]

Hello Qlik Community,

We have a project that will perform the scheduled and automated movement of Excel spreadsheets from a Linux environment to a Windows folder.  I have some questions for the community about how to go about completing this.

What would be appropriate tools/software for performing this project?  We need files that will follow a naming convention placed into the same directory on a Linux device, to be moved without intervention onto a windows device.

What budget is necessary to set up such a project?

What personnel are necessary to fulfill this project?

What level of effort would you expect in terms of days/weeks/months should it take to complete this?

Thanks for your assistance, I know this isn't a specific QlikView question, but eventually the file that is transferred would be used as a source input for a Qlik application.

[#2]

Hi Evan,

It's difficult to answer this question without more details.  The question really needs to be answered in the context of your requirements and environment.  Areas that would need to be given consideration would include (but are not limited to):

• resources available for both implementation and support
• architectural guidelines / mandates within the organisation
• tooling is available within the company (e.g. enterprise scheduling capability)
• requirements in terms of security, performance, error handling for job failures / retries
• where does the job need to run from (this could be driven by any other requirements from above - e.g. who will be supporting the process, security, etc)
• how are the files exposed from the unix server (SMB share, FTP, HTTP, SFTP, etc)
• do you really need to move the files from the unix environment, or if you can just read them into QV on the fly would that be sufficient?

The most basic way to set this up would probably be using scp and a cron job on the unix box, or you could use sftp or some other mechanism.  What is the right choice will depend on your environment.  E.g. if the files are exposed from the unix server via an SMB share, you could just use robocopy to move them from the SMB share to the windows server.  If it's http, you can use CURL.  Depending on how the files are exposed, you could potentially even just load them directly into QV, or download them using a script in QV load script (shelling out). 

I would expect this to take hours or days, not months, but depends on the quality of the resource implementing it, their experience, and your requirements.

Regards,

Graeme

[evan_kurowski]

Hello Graeme,  Thank you for your response.  If I could maybe add some parameters, the security requirements internally are none.  If this could be automated by an individual user, that is sufficient (but Autosys might be a "formal avenue").  No error handling (it would be nice to have, but for now any means of automation is better than none).  The transport methods available is Connect::Direct?   It would initiate from the Unix side. 

• do you really need to move the files from the unix environment, or if you can just read them into QV on the fly would that be sufficient?

Reading the source files in place would be optimal, but how would that be achieved?  Does that require the SMB share option?  That may not be a permitted avenue in our environment, as far as I'm aware we have SFTP (which doesn't have automation avenues available), or we can use Connect::Direct which gets paired with Autosys?  Do any of these sound familiar?

[jmvilaplanap]

Hi Evan

I normally use a simple *.bat file (from windows) with some MS DOS commands (or PowerShell if is more complicated)

From time to time I used Talend (is an ETL software), is easy to use. https://www.talend.com/resource/etl/

Regards

[evan_kurowski]

Hello Jose, I guess the premise in that .bat file is there is a copy command which can reach a Linux device.  One of the issues here is that in our environment, we aren't allowed to mount a Linux drive in the context of a windows device.  For example, a copy command that went:  COPY LinuxPath WindowsPath.  I don't think we have that as an available option.

[jmvilaplanap]

Hi Evan

And from Linux is possible to do it?

Another solution I´m thinking is to use a synchronization tool like dropbox or something like this but with app for windows and linux.

Regards

[evan_kurowski]

Not sure.   Looks like without being able to mount any cross-platform directories, or use any of the cross-platform utilities, this is just a non-starter.

[jmvilaplanap]

At the end of this page, explain how to use rsync

http://lifehacker.com/372175/free-ways-to-synchronize-folders-between-computers

[evan_kurowski]

at some point, Microsoft solved this problem for Windows devi es.

while moving files around a windows network is a bunch of complicated steps, the steps presented to the end user are pretty simple.

you mount a folder, and then you can cut/paste or drag drop.

I'd like to believe there's a similar technical solution possible between *nix/Windows devices.  I've seen it advertised as easy as a mounted folder, but for whatever reasons sometimes the most convenient options are the ones least accessible.  Ironic!