26 Replies Latest reply: Jul 12, 2013 10:26 AM by Jeffrey Vermeire RSS

SR1 - SVN

James Gilpin

Has anyone downloaded QV11 SR1 and played with the SVN interface?

 

James

  • SR1 - SVN
    Edgar Kech

    As I want to integrate QV with SVN, too, I will test it next week. Qliktech said, they did some changes in the XML-structures (we had some problems with IR and lost settings when using XML-Files / -prj-Folder). Hope, they fixed this kind of problem and we get a real seemless integration...

    regards,

    Edgar

    • SR1 - SVN
      James Gilpin

      Here is what I found (so far)

       

      Pros:

       

      1)  management of the -prj (including creation) is handled by the desktop client

      2)  .qvw is NOT stored in SVN; when you checkout the working copy the .qvw is created from the -prj files

      3)  add project to SVN and get project from SVN work well.  however, you will need to setup the project folder structure (branches, tag, trunk) on your own.

       

      Cons:

       

      1) Requires an external merge tool.  There are some settings to point to the tool of your choice.  I am still trying to get kdiff3 and tortoise to accept the incoming file names.  If I drop out to windows explorer, the standard TortoiseSVN plugin will start up the merge tool correctly.  I prefer kdiff3 and it handles the required changes.  Sure would be nice to have visual tool that showed the difference instead of inferring the placement of objects.

      2) Documentation.  Or the lack thereof...

       

      Some recommended steps to use the interface:

       

      1) Always 'Get Latest Version' to avoid conflicts, prior to making changes.

      2) Be prepared to resolve conflicts externally.

       

      More to come, I suspect...

       

      James

      • Re: SR1 - SVN
        James Gilpin

        Update

         

        After extensive testing over the past two days, I am even more confused regarding the ability to use a version control system to track changes to QlikView documents.

         

        Base Assumption:

         

        Never store the .qvw in the repository.  You cannot view the changes to the binary and the resulting repository growth can be overwhelming.  Which means the QlikView desktop is require to get the latest version of the .qvw -prj files and rebuild the .qvw document.

         

        Findings:

         

        The QlikView desktop will successfully rebuild the .qvw based on the objects stored in the -prj folder and placed in the repository.  What is upsetting is that when the document is rebuilt, certain values stored in the .xml objects are CHANGED by simply rebuilding the document.  If you reload the data, other values are also changed.  The result:  the -prj folder now contains .xml files that differ from the repository.

         

        So what, you ask?  Is that not the point of the repository?

         

        Take the typical use of a source code repository...

         

        Dev creates the .qvw document and uses the QlikView desktop to 'Add Project to Source Control', which adds the -prj folder into the repository

        User 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document.

        Test Server 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document and the reloads the data.  (QVS, Access Point, auto build tool)

        User reports an issue and requests a modification

        Dev uses the QlikView desktop option to 'Get Latest Version' of the -prj folder (best practice to stay up to date, before starting new changes.)

        Dev makes changes and uses the 'Check in Pending Changes' into the repository

        User gets the latest changes, and finds that conflicts have occured (remember, the QlikView desktop has rebuilt the document and .xml files in the -prj folder used by User are now different than the repository)

        Test Server gets the latest working copy, finds that conflicts have occured....

         

        Question:

        Does this happen using the TFS interface?

         

        James

        • SR1 - SVN
          John Trigg

          Why are you placing the QVW in the repository?  The whole purpoe of the integration is to move the component files (found int he PRJ folder) into the repository and manage from this definition.  The goal has never been to version control the QVW given its potential size (with the data).

          • Re: SR1 - SVN
            James Gilpin

            Perhaps I was not clear, the base assumption I made is in total agreement with your statement.  I am NOT storing the .qvw document.  Only the files that are stored automatically by the QlikView desktop.

             

            It is these files that are in conflict between the different working copies.

          • SR1 - SVN
            Brian Garland

            I had made the assumption that the .qvw would be stored in the repository but with the data stripped out, sort of how Publisher opens the .qvw without data when it performs a reload. We're going to go ahead and test it anyway to see if it will work for us.

          • Re: SR1 - SVN
            James Gilpin

            John,

             

            I reread the above post and realized the wording did not reflect actual process.  It has been edited to more accurately reflect the actions taken by each individual in the process.

             

            thanks,

             

            James

          • SR1 - SVN

            Hi John,

            Echoing Brian Garland's comments - I would definitely like the ability to save the QVW binary (deflated) in the source code repository. I understand that theoretically that isn't really necessary. But I've found plenty of times where it's been a "last resort" to have a version-controlled binary tucked-away in a safe spot.

             

            A feature request I would look for is a toggle switch that allows "Include Binary In Repository." And naturally I would look for the feature to automatically reduce all data from the QVW.

             

            Tks,

            Bill

        • Re: SR1 - SVN
          John Trigg

          James

           

          I want to ask some questions of your steps (which I've nunbered for ease of reference). Can understand #2 & #3, but is the user going to test the document on the test server or locally? Lets say its tested locally by the user and posted on the test server.  So developer does some work and checks in changes.  User gets the latest and finds these changes; doesnt seem unreasonable that their version is going to be in conflict. Now one best practice maybe for the user to clear out their PRJ folder before doing get latest, if they are not doing development.  Same maybe true for the test server.

           

          I guess what I am looking for guidance on is what do you expect the user and test server to be faced with in steps 7 & 8?

          jmgilpin wrote:

           

          Take the typical use of a source code repository...

           

          1. Dev creates the .qvw document and uses the QlikView desktop to 'Add Project to Source Control', which adds the -prj folder into the repository
          2. User 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document.
          3. Test Server 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document and the reloads the data.  (QVS, Access Point, auto build tool)
          4. User reports an issue and requests a modification
          5. Dev uses the QlikView desktop option to 'Get Latest Version' of the -prj folder (best practice to stay up to date, before starting new changes.)
          6. Dev makes changes and uses the 'Check in Pending Changes' into the repository
          7. User gets the latest changes, and finds that conflicts have occured (remember, the QlikView desktop has rebuilt the document and .xml files in the -prj folder used by User are now different than the repository)
          8. Test Server gets the latest working copy, finds that conflicts have occured....

           

          Question:

          Does this happen using the TFS interface?

           

          James

          • Re: SR1 - SVN
            James Gilpin

            John,

             

            If neither the user nor the test server has made changes, no conflicts should occur.

             

            But it appears that the desktop is making changes to the .xml files based on the presence of data.  Sort criteria, present/not present, etc.

             

            James

          • Re: SR1 - SVN
            James Gilpin

            John,

             

            In step 2, the user gets a copy of the -prj as it sits in the repository.  As part of this process the desktop application recreates the .qvw document (as expected).  The process has the desktop application store a blank .qvw document and then applies the -prj xml files.  Once the -prj xml files have applied, then user is presented with the layout, script, etc. for review.  This restored copy of the .qvw is NOT saved until the User tells QlikView to save the document.

             

            IF the user SAVES the document at this point, the resulting -prj xml files are overwritten.  Even though the user has not made any changes some of the -prj xml files are different.  In my case, 9 out of 50 are different.  The next time the user issues the 'Get Latest' command a conflict is detected.

             

            I thought maybe it was due to platform, 32bit/64bit, or possibly different user preferences.  But this occurs between two different working copies of the -prj folder on the same machine.

             

            James

            • Re: SR1 - SVN
              John Trigg

              So have spent some time playing around with this today and cant quite replicate your conflict scenario, but put that down to user setup error on my part.  I suspect a date/time stamp is updating and forcing the conflict though I cannot yet confirm that.  Will do some more poking around and talk to the appropriate developers to get more insight.

               

              Cheers

              John

              • Re: SR1 - SVN
                James Gilpin

                John,

                 

                Thanks

                 

                Should you need an example document, give a shout and I will redact this one I have been playing with and post it....

                 

                James

                • SR1 - SVN
                  Rob Wunderlich

                  James,

                   

                  I want to confirm your scenario. The user only does:

                   

                  1.Open

                  2. Reload

                  3. Save

                   

                  And that creates changes in some of the object.xml files? I would expect a conflict in the AllProperties file, but no others. Any column sorting, object moving/restoring before save will cause change as well because those are layout changes.

                   

                  I'd be curious to know what files and properties you are getting conflict on.

                   

                  -Rob

                  • Re: SR1 - SVN
                    James Gilpin

                    Rob,

                     

                    It is not quite that simple.  This appears to be an issue with how the .qvw is rebuilt after getting the -prj from the svn repository.  When the desktop application reads the svn repository it must recreate the .qvw based on the files in the -prj folder.

                     

                    In my test case, three files are different...

                     

                    AllProperties.xml

                    CH01_391551278.xml

                    QlikViewProject.xml

                     

                    I created a 7zip archive of my test files to show these differences.  (yes, i included the .svn folder... bad developer...)

                     

                    James

  • SR1 - SVN
    Torben Seebach

    Has anyone managed to move code between servers without committing the .qvw?

     

    I looking for at method to maintain a central repository, and how this can be moved between sandbox, test and prodution. But I'm puzzled on how to best handle creating .qvw files if a new file is added to the repos.

     

    I'm disregarding the source control options in qlikview, since are running some very strange commands, and don't allow me to check out dependencies. As far I can tell the buildt in functions are very limited, and don't support a standardlized version control method.

     

    SVN is super scriptable, so I'm thing of making server hooks that perhaps adds a blank .qvw when a new -prj folder i found.

    • SR1 - SVN
      James Gilpin

      I have Jenkins in place performing builds of the .qvw by use of the 'svn revert', 'svn update', and 'qv.exe /r' commands.  But in my workflow I manually add a blank document for the initial setup.  It would be very interesting to see a script that detected a new -prj folder and copied a blank document into place.

       

      The next step for our group is to setup a code review process that requires an approved signature for changes to the .qvw document stored in the repository.  Note: this has no bearing on the users, only developers, ie. data source, joins, etc.

       

      James

    • SR1 - SVN
      Rob Wunderlich

      I build the qvw from -prj using VBScript:

       

      The CreateQvw() is enough to create the qvw that will then be populated when a user opens it. If you want it to be fully populated call the OpenClose() as well. The qvw name is determined in another process and passed to the routine. Hopefully this will give you a guide as to the steps to take after checkout. Because it's using QlikView, a GUI is required. I haven't figured out how to build the qvw in a headless environment.

       

      -Rob

       

      Function CreateQvw(qvwFullPath)

                Set Qv = CreateObject("QlikTech.QlikView")

                Set docObj = Qv.CreateDoc

                docObj.SaveAs qvwFullPath

                docObj.CloseDoc

                Set Qv = Nothing

                CreateQvw = qvwFullPath

      End Function

       

      Function OpenClose(qvwFullPath)

        Set Qv = CreateObject("QlikTech.QlikView")

        Set docObj = Qv.OpenDocEx (qvwFullPath,0,false)  ' Open the document

        docObj.Save

        docObj.CloseDoc

        Set Qv = Nothing

        OpenClose = qvwFullPath

      End Function

      • SR1 - SVN
        James Gilpin

        Rob,

         

        Do you find that you lose actions/triggers associated with fields by using this method?

         

        For example, when the blank document is created using the xml files in the -prj folder, QlikView is not aware of any fields until the document is reloaded.  The association of triggers/actions to those fields is lost (or appears to be) by creating a blank document.

         

        James

  • SR1 - SVN
    James Gilpin

    Update

     

    QV11 SR1 resolved a couple of minor issues, however there are more difficult issues which everyone should know.

     

    Issues:

     

    1) requires svn 1.6.17 (client)

     

    You cannot use version 1.7 of the svn client with the supplied QV SVN provider dll (source control settings).  Version 1.7 svn merge variables (%base, %mine, and %theirs) are handled incorrectly by the QV dll.  Downgrading to svn 1.6 (client) resolves this issue.  However, downgrading and trying to maintain working copies created under 1.7 OR using Tortoise 1.7 and svn client 1.6; NOT recommended.  Best to start over by uninstalling all the svn client software (including Tortoise, VisualSVN, etc.) and reinstalling the versions supported.

     

    2) Get Project from Source Control creates working copy, regardless of existing working copy

     

    I am being picky, but if the working copy structure already exists...  QV should recognize the working copy and use it instead of creating a new one inside the existing working copy.  (multiple .svn directories cause all kinds of grief).  You can force QV to recognize the structure by copying the SourceControlSettings.ini into the -prj folder within the existing working copy.

     

    3) Field Event Triggers can be lost... ARGH! (yes, i opened a support case)

     

    QV uses the .xml files in the -prj folder to recreate the .qvw document.  The .xml files represent the pieces of the .qvw.  When you 'get project from source control', QV rebuilds the .qvw document.  Triggers based on fields within load statements are not saved in the .xml files.  So when the .qvw document is recreated there is nothing to use to rebuild these triggers.

     

    James

    • Re: SR1 - SVN
      darrinm4

      James, Did you get a response about the Field Event Trigger issue with SVN? I found the same behavior and it's a show stopper for me to use version control.

       

      Is there a work around?

      • Re: SR1 - SVN
        James Gilpin

        No.

         

        We store the empty binary .qvw along with the export xml files

         

        James

        • Re: SR1 - SVN
          darrinm4

          How do you empty the binary .qvw before storing in svn? Is it a manual perging of data in QlikView, save, and syncing file manually to SVN outside QlikView?

          • Re: SR1 - SVN
            James Gilpin

            In the post commit script, the size of the file is checked and the commit is rejected if it is over 1 meg.  This is a basic size - the thought being that most documents would be over 1 meg (with data).

             

            The developer then has to use the reduce all function, save the empty file, and commit the .qvw document.

             

            Not ideal, but it works.

             

            James

    • Re: SR1 - SVN
      Jeffrey Vermeire

      I can confirm this is still an issue in Qv11.2 SR1.  Has anyone found a solution or is there any movement from QvSupport?  This is a pretty decent bug to deal with.  I have a developer ready to pull his hair out since every time he checks-in his changes, POOF!  Triggers are gone.

  • Re: SR1 - SVN
    Brian Vander Zanden

    James-

    I downloaded the latest version of VisualSVN, and TortoiseSVN and am having the merge issue.

     

    Did you have to use release 2.1.11 of VisualSVN Server? 

    I've found athe 1.6.16 version of TortoiseSVN, but not an earlier version of VisualSVN.

     

    Thx.

    Brian VZ