9 Replies Latest reply: Jun 6, 2013 1:42 PM by Jeffrey Vermeire RSS

    Publishing issues in 11.20.111718.0 (11.2 SR1)

    Dalton Ruer

      I'm having several problems with the publisher in 11.2 SR1 that we never experienced in 10.

       

      1. Documents sometimes take 17 seconds to reload and other times takes 3 minutes. Some take 56 minutes and then the next day they take 4 hours. (backend databases are fine problem is on QV server side)

      2. CPU goes to 100% on every CPU (32 of them) even when publishing a single document

      3. Jobs that are scheduled to repeat every 30 minutes, will reload every 30 minutes and then suddenly start reloading every 3 hours for a couple of times, then just stop scheduling themselves forcing me to restart. Once restarted they reload every 30 minutes and then continue again to have problems. Recreated the jobs multiple times to no avail.

       

      Are these known issues I couldn't find anything in search? Is the 11.0 with SR2 more reliable a platform that I should be on?

        • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
          Karthikeyan Subramaniam

          Delete the problem tasks and create them again. Now see the results.

          • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
            Nick Gan

            Hi,

             

            Just curious, is the 32 CPU on hyperthreading? or NUMA disabled?

             

            and what is your RAM size?

             

            Do you restart the QVDistributionService.exe on frequent basis? or never at all?

             

            Thanks,

              • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
                Dalton Ruer

                Thanks for the response. Wondering why you guessed 32 CPU's using hyperthreading. There are 32 "logical" cpu's. 16 CPU's with hyperthreading. Not sure how to check for sure if NUMA is disabled but I see 2 NUMA nodes in the resource monitor so guessing no. Is that a known issue that I should definitely check out?

                 

                256 GB of RAM.

                 

                Problem definitely stops (meaning applications publish faster) when I restart the distribution service then continues to degrade daily. I recently installed 11.2 SR2 and didn't want to be fooled into thinking it actually solved the problem when it is actually the restart solving the problem.

                 

                I NEVER restarted the server process or the distribution services when running version 10. Since upgrading I've had to make all kinds of changes.

              • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
                Jeffrey Vermeire

                We've run into similar issues.  I've found that if a document loads from qvds and has functions in the load statement, then the likelihood of the CPU spiking goes way up.  One of the things that we did to help the problem was to move any functions to the DB loads or after the initial load from qvd.

                  • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
                    Dalton Ruer

                    Thank you that helps alot. We definitely have lots of QVW's that make heavy use of functions.

                     

                    Didn't notice this under 10 but I never really had issues causing me to try to analyze performance. It's only been since I had the other issues that I started trying to look for causes for documents to take so long to load. I would have thought each document reload had it's own thread and that CPU might be 100% but wouldn't have expected all threads to spike.

                     

                    I will definitely research QVWs with no functions and see what happens compared to QVW's with lots of functions.

                      • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
                        Jeffrey Vermeire

                        I've done some more digging into this problem.  I have found that if I use a function nested inside another function in a load script statement then the CPU spikes to 100%.  The solution that I found is to split the functions across preceding loads.

                         

                        If I have a field like this:

                         

                        Date(Floor(datetime_field),'MM/D/YY') as new_date
                        

                         

                        The reload just destroys the CPU, but if I do something like this:

                         

                        LOAD
                             Date(new_date,'MM/D/YY') as new_date
                        ;
                        LOAD
                             Floor(datetime_field) as new_date
                        From
                             C:\mytable.qvd (qvd)
                        ;
                        

                         

                        then the reload runs fine with no issues.

                         

                        Is this related to Hyper-Threading or is this just an outright bug?

                    • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
                      Dalton Ruer

                      Updates to original post since working with QV Support.

                       

                      1. Time to complete a reload each day - This problem persisted even after installing 11.2 SR 2 per supports recommendations. When I manually bounced the distirbution services the times to complete went back to normal and continued to increase each day. Since SR 2 hasn't resolved the issue I've scheduled the restart of the distribution services daily and that keeps things processing at simillar, reasonable amounts of time daily.

                       

                      2. Nothing to date has stopped all of the processors from showing busy when even a single document is being reloaded.

                       

                      3. Jobs rescheduling issue was resolved by moving the QVPR from the default XML to a SQL Server. Did that after reviewing logs showing multiple errors occuring about files being busy and not able to be accessed.

                       

                      Under version 10 I was able to republish many documents at the same time. Limit was set to 20 and never had issue in version 10. QV support recommended reducing the max to 9. That didn't affect the time to republish at all. But has generated errors then when the 10'th document is due for refresh but an engine can't be granted to it.

                       

                      Have resorted to restarting my QV Server process and Distribution process on a nightly basis. Restarting QV Server Process throws away my named users each time which creates another headache but I've got to keep moving forward with 11.2 at this point.

                      • Re: Publishing issues in 11.20.111718.0 (11.2 SR1)
                        Bill Britt

                        Hi,

                         

                        Take a look at this

                         

                         

                        Bill.