9 Replies Latest reply: Sep 1, 2016 11:55 AM by Alip Nath RSS

    development methodology in sense

    DANIEL CHOTZEN

      hi all,

      in old qlikview  we had solid methodology for the development to production cycle

      we usually work at customer's site so we have a development, test and production folder each with same structure

      folder for extract/staging stage where we put all qvd of data extracted from all the sources

       

      folder for transformation stage where we read the output of stage one and cleanse/transform/manipulate the data and produce another set of qvd files.

       

      folder for data-modeling where we read the output of stage 2 and create the data model

       

      and folder for layout where we binary read the 3rd stage and add security and layout objects.

       

      we use the development area to do the work,

      test it on the test area

      and upload it to the production area.

       

      all of the above can be done in one machine for a small customer and scale up to different machines for larger customers.

      we can combine resource management tools and roles to handle all the metadata and project management

      weather it is one developer or many.

       

      now comes along qliksense and all the cards are shuffled.

      we now have data-connections that hide the physical folders.

      we have apps and streams and rules and mash-ups.

       

      and we need to find a new way to handle the cycle.

      and although there is a lot of material on all things sense, from creating apps to installing servers to building mash-ups

      i  found  almost nothing on this topic.

      there is one nice post from

      but so far no one replied to it.

       

       

      we need answers and ideas

      please put anything you can think of here

      any ideas would be welcome

      maybe together we can form some methodology until qliktech comes up with another S.A.F.E document or something.

      (all this is made more difficult since sense is still in development and keeps springing changes and new concept on us)

      thanks in advance for any contribution

      Daniel

        • Re: development methodology in sense
          Gary Strader

          Daniel,

           

          I am really keen on developing a Sense methodology for the same reasons you are.  We have years of experience designing and managing multi-tier QlikView environments, but as you said a lot has changed with Sense.  I am working through a few issues right now, including:

           

          1) Sense server architecture is a lot more flexible and configurable than View. With that power comes more complexity.  The new documentation is a LOT better than old server/publisher documentation, but it still lacks best practice/methodology information.  I imagine that Qlik Expert Services is developing that as we speak.

           

          2) Given the new token-based licensing, does it even make sense to purchase a Sense test license?  You can add as many server nodes as you want.  What's the best way to set up dev/test/uat/prod nodes?  Via streams?  Custom properties?  Something else?

           

          3) Given a multi-tier environment with Sense, what's the best way to promote changes from dev/test/uat/prod?  Copy an app from the dev to test stream?  Export/import?  Something else?

           

          4) Does your ideal Sense architecture change when you have both QlikView and Qlik Sense in your environment?  e.g. QVDs can be shared between View and Sense but they must be created by View.  QVDs created in Sense are not backward compatible with View.

           

          Just some thoughts I wanted to jot down.  I will follow this thread to see if anyone has good ideas.  Perhaps the community can develop it over time together.

           

          Gary

          • Re: development methodology in sense
            Torben Seebach

            Looks like Sense is maturing enough for this discussion to become relevant. Great to hear the interest!

             

            In my initial post I on purpose asked more questions than I answered, because I wanted fresh eyes. Qlik has so far been very silent. And that's why I think the community needs to get together, so we can ensure long term success for Sense, in large deployments where governance and structure is key.

             

            A) On the ETL side I'm seeing to main methods emerging: QVD based and DW View based.

            The Data Connections have made the QVD layer less important. Which means that in a managed DW environment becomes a layer of unnecessary overhead. In general I load data directly from the DW to the Apps that need it.

            For Self Service Datamodelling, I'm exposing the Modelled DW layer or the Staging layer to the end users for them to connect directly to. The modelled interface is typically a View, because I need to ensure that field names are human readable, which the base tables are not.

             

            The QVD option is pretty much unchanged from QlikView, but I don't use % or KEY_ prefixes in fields (which some people still think is best practice), every field is very human readable named, including keys. I also a have Extract and a Transform Stream to hold ETL steps.

             

            B) Purchasing a testing server is still valid, if you need to have a large group of people testing. Then its cheaper and additional tokens.

             

            C) Code promotion from test to prod, is still a issue. The current release is very heavy on manuel process's and documentation. And its a mix of using the QMC, Windows Explorer and the Deployment Console.

              • Re: development methodology in sense
                Stephen Egerton

                Great insights Torben.

                 

                I'm in a situation where our environment is heavily invested in Qlikview. The ETL, QVD layer and Data Models are very mature and stable. Apps are 90% developed by IS. Users can create and share objects and bookmarks only. We use a product called Platform Manger from in4bi for code migrations between DEV-QAS-PRD.

                 

                Now I'm setting up a roadmap for Qlik Sense. Initially, it will be our Self Service and Mobile platform. Long term it could replace Qlikview and extract directly from our super fast Oracle Exadata ERP views or I'm considering using Qlikview long term as our back end ETL, QVD and Data Model tool and swapping Qlikview user cals for Qliksense tokens. This would get all users off of Qlikview and onto Qlik Sense. Our Qlik Sense apps would perform binary loads from our Qlikview Data Models in this option.

                 

                I'm attending Qlik Sense admin training next week and getting some 4 nice big engine servers the week after. 14core w/512g of ram each. I was going to put 2 engine servers in the PRD site and 2 in the QAS site. Central node(s) will be VM. If I create a DEV site, it will be all VM for the engines and central node. My immediate question is, do I really need the typical DEV-QAS-PRD servers/site set up for Qlik Sense when using Qlikview as the backend? I'm curious to see what you would do in my situation. Would you have all 4 engine servers on 1 site perhaps?

                 

                Thanks!

                Steve

                • Re: development methodology in sense
                  Madhavi Basawaraju

                  Great post Torben,

                  We are in middle of setting up the Qlik sense production and self service servers.

                  We are planning to have the Extract folders which are based on extracts from DW, Transformed Layer based on Extract from the first step.

                  In Qlikview as the above post mentioned we used to expose modeled application that could be used as a binary load for fourth layer.

                  SInce in Qlik all the modeled data will be in a QVF under Server repository. Is there a way to expose this layer of applications without creating a folder connection as this will also hold other managed applications from the BI team ,as the users need the ability to plugin their external data files and do reloads.

                   

                  Any insight you can provide would be helpful.

                   

                  Thanks,

                  Madhavi

                • Re: development methodology in sense
                  Boris Tyukin

                  hi Daniel, I am very curious what you ended up doing as i am facing the same questions/issues with our brand new sense install.

                   

                  Also wanted to link my question to your post

                  Development/Test vs. Production Data connections

                  • Re: development methodology in sense
                    Paul Scotchford

                    Development methodologies have many components.

                     

                    One I have developed and implemented here at my client site is Data Governance by a single repository for

                    calculation expressions used within 1 or more apps.

                     

                    This gives us control as to the instantiation of a business rule via an expression, i.e. just one version in the BD that is agreed and tested by the relevant stake holders. Single point of change and deployment (across UAT/PROD etc).

                     

                    Just email me here and I can send a tech spec of how this works.

                     

                    cheers

                     

                    Paul