Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
optiknerd
Contributor II
Contributor II

for each filelist next suddenly stops working - QS - in Legacy

Hi,

I built several scripts to build a QVDs using QS in legacy mode to do a for each next with filelist to load multiple files from a directory on my desktop.  This is the second time this has happened.

These scripts were working and all of a sudden stopped.

I have tried turning off my security software (all components antivirus, firewall etc) but that does not change things.

I have reinstalled the software, reinstalled the operating system (keeping files) and reinstalled the security software.

The first time that this happened the OS+Sec+QS reinstall worked (on that occasion the system would not let me uninstall QS so I had no other option.

This time I tried the same procedure but the problem does not go away.

I have tried running the same scripts with QV and it appears to have similar problems as it has a hiccup on the first file.

In both versions QV and QS, the system appears to successfully execute the AutoGenerate command to kickstart the table but the first table it tries to read indicates that it cannot find the field that i used in the AutoGenerate in the tables I am trying to load.

Let me repeat that these scripts were all working fine until suddenly they did not.

I am trying to convince my client to adopt QS for their analysis so this happening twice is really not helping my cause.

Does anyone have any ideas on how I can figure out what is tripping things up.

The script is obviously not the problem because it worked fine - I have the QVDs to prove it but something is blocking the access to the file and it does not appear to be the Security software.  My bet is that it is windows getting in the way.

I would log a ticket but sadly I am not a customer - just a fan trying to create a customer..

My System info is

Qlik Sense desktop version 12.16.2 (April)

  • .NET SDK 13.4.0
  • About Service 1.8.1
  • App Migration Service 4.8.1
  • Broker Service 5.11.2
  • Capability Service 1.4.1
  • Client 4.86.8
  • Data Preparation Service 2.32.0
  • Dependency Graph Service 1.0.6
  • Freya Client Service 2.8.0
  • Printing Service 13.15.1
  • QIX Engine 12.145.3
  • Qlik DataMarket Connector 2.8.0.1
  • Qlik Essbase Connector 1.1.0
  • Qlik ODBC Connector 4.8.0
  • Qlik REST Connector 1.4.0
  • Qlik SalesForce Connector 14.3.0
  • Qlik Sense Browser 7.2.0
  • Qlik Sense Hub 0.18.0
  • Qlik Web Connectors 0.36.0
  • Service Dispatcher 15.0.3
  • Web Extension Service 1.5.2
  • Web Extensions 1.4.1

Note to whoever at Qlik reads this - I suspect that many QS trials die on the vine because the anti virus blocks the app before it ever gets started.  There is something about the way the app is set up to be a local program but appears to operate more like a server type program seems to flag my security software were most other apps don't.  May be worth highlighting this risk so that potential customers don't lose interest before even opening the program.

8 Replies
marcus_sommer

It's not quite clear what didn't work - the access on certain folders/files or any preparing process-step, any autogenerate-load and the afterwards load of this table? The first indicates wrong paths, missing access rights and/or any file-blocking processes and the second that there is any (temporary) weaknesses in your script (please provide it and the error-messages).

- Marcus

evan_kurowski
Specialist
Specialist

 

Hello Kevin,

 

To add to discussion, without specifically knowing what went wrong in your setup...


If the script hasn't changed, maybe next thing to check is if inputs have changed.  Watch the input files for the following when using Qlik file iteration:

     

  • Programmatic characters in file path (sometimes an uneven quote, programmatic character, multiple extensions, or unicode can throw the iteration)

  • Symbolic links (self-referencing links can create these endless loops)

  • Imposter file extensions (the file name says .QVD but the contents turned out to be something else entirely)

 

For "industrial strength" file iteration, even when turning to something like Powershell (manufactured by Microsoft, the creator of the Windows operating system), even Get-ChildItem can still have issues when full file path length exceeds path length limit (and yet windows explorer has no trouble passing this limit, sooo...)

 

Totally agree on the potential of "sideways" impact from anti-virus as well.  Hard to make something work well, if at the same time something is trying to impede it.

 



optiknerd
Contributor II
Contributor II
Author

Marcus,

The script was working.  I did not change anything in the script!!!

Either something in my system stopped letting QS get access or something in QS stopped working for some unknown not script related.

I have set up BitDefender so it should have access to the files and it was letting them at the start.  

Here is the script.

Chimp_Load:

LOAD 0 as "Email Address" AutoGenerate 0;

for each file in filelist('lib://Chimp Raw Data/*members_export*.csv')

  Concatenate (Chimp_Load)

    load

   

           *,

       

// makes all characters in email address in lower case so we can use as a key field (ie avoid mismatch because of inconsistent caps)

        Lower("Email Address") as "Lowercase Email Address",

       

// Clean up garbage characters found in Lat Lon data and set temp field that is changed back next routine (can't duplicated field names)

        KeepChar(LATITUDE,'0123456789-.)')as Cleaned_Latitude,

        KeepChar(LONGITUDE,'0123456789-.)')as Cleaned_Longitude,

       

// Make GeoPoint that creates unique identifier for Qlik to use in making coordinates

        GeoMakePoint(KeepChar(LATITUDE,'0123456789-.'),KeepChar(LONGITUDE,'0123456789-.')) as Chimp_GeoPoint,

       

// extracts file name from the name of the spreadsheet

       

          FileName() as Filename

       

      

    from [$(file)](txt, utf8, embedded labels, delimiter is ',', msq);

next file

Drop Fields "Email Address",LATITUDE,LONGITUDE from Chimp_Load;

optiknerd
Contributor II
Contributor II
Author

Hi Evan,

I had thought that it was a change in the base files at first but then I went to an another app that I had got working and whose base files had not changed at it gets the same error.

I think that the imposter qvd files cannot be the case because I set up a test in QV and it pulled data from the same file source but stored the QVD in a temp directory specially set up for the test.

Could making the file length too long for one app and set of files screw up Qlik Sense for all of the apps even ones using completely different files?

Cheers,

Kevin

evan_kurowski
Specialist
Specialist

Kevin Hunt wrote:

Hi Evan,

I had thought that it was a change in the base files at first but then I went to an another app that I had got working and whose base files had not changed at it gets the same error.

I think that the imposter qvd files cannot be the case because I set up a test in QV and it pulled data from the same file source but stored the QVD in a temp directory specially set up for the test.

Could making the file length too long for one app and set of files screw up Qlik Sense for all of the apps even ones using completely different files?

Cheers,

Kevin

Hello Kevin,

Looks like you've done due-diligence examining impacts from the inputs side, and can rule out some potential root causes.

optiknerd
Contributor II
Contributor II
Author

Evan,

I have no idea how it happened but the files that I was so sure had not changed had but in the weirdest way.  One of 15 CSV files changed from UTF-8 to ANSI.  With that case resolved I went back to the app where I originally had the issue and discovered that I had an extra file with the same start to the name that was being picked up by the loop.

I have to conclude that I must have refreshed the data in the Chimp data set and forgotten about it.  Going to go back to Mail Chimp and see if I can reproduce the error.  Very weird if it does because it was only in one file and Chimp produces the list exports three at a time.

Anyway, many thanks for your help!

Cheers,

Kevin

optiknerd
Contributor II
Contributor II
Author

HI Marcus,

Evan gave me the tip that led me to my errors.  It appears that I had refreshed the data on the app that I thought was untouched.   With that cundrum solved I went back to my original problem and discovered the error there which was a simple naming one file too much like another so it got picked up by accident.

Thanks for your help!!

marcus_sommer

If you get the next time a similar error than enable the script-log (better this should be the default-setting) and use a trace-statement lile: trace '$(file)'; and you will find the breaking point quite easily and often it's then quite obvious that a wrong file was loaded or that the fileformat and/or datastructure has changed in any way (unfortunately this is a quite regular approach in my work because I don't control all the source-files).

- Marcus