2 Replies Latest reply: Jul 30, 2014 7:22 AM by Egor Kobylkin RSS

    Scalable interface to Teradata

    Egor Kobylkin

      I have just published this blog stub but was not able to pin it to this "place". If anywhere this should actually be pinned to scalability as it allows you to bring in data with tens and hundreds of MB/s.

      Qlikview quietly releases Teradata Parallel Transporter Connector

        • Re: Scalable interface to Teradata
          Egor Kobylkin

          Moving the content here too as it seems I can not create a blog post pinned to this group.

           

          Looking to upgrade to the most recent server software release I have found a TPT Connector in the list of available software. Intrigued I have downloaded and installed to see what that would be. Indeed this is a Teradata Parallel Transporter Connector for Qlikview! The main advantage of connecting to Teradata via TPT over ODBC is that it can leverage the native FastExport export technology of Teradata. Teradata is massively parallel database. Leveraging FastExport it can export data with the speed of tens when not hundreds of MegaByte per second in multiple parallel sessions. ODBC is limited to 500KB/s in my experience.

          I will be testing the TPT Connector and will post my experience with it here.

          • Re: Scalable interface to Teradata
            Egor Kobylkin

            Good news is that the connector works.

             

            I was not able to use the "select" feature as it never completed the connection part.

             

            The "connect" button though creates a connection string that works fine. Note that you will need not only the TPT and FastExport but also BTEQ installed.

             

            Another bummer for me was the inability for the TPT select to deal with a "select * from ..." form. It needs a list of comma separated column names and does not accept anything else. This is limiting the flexibility as now we need to rewrite the select statement every time we would change a source table. The beautiful ability of Qlikview to dynamically associate source files or tables is now lost with TPT. Basically most of my scripts is a list of load * from a, sql select * from b statements. When using this approach I have only one place to care about the column names - that is where the table is created. Now with TPT the Qlikview Script has to be synchronised as well. Is it too complicated for the Plugin to run a BTEQ with a "select trim(columnname)||',' from dbc.columns where tablename = 'x';" ? Well, may be the developers could implement that in the 1.1 version?

             

            As to the speed - current benchmark 50k rows per second. Will see how that varies across different source databases.