Mastering the HyperCube: High-Volume Data Strategi... - Qlik Community

Ouadie · ‎2026-01-02

If you have been building custom web applications or mashups with Qlik Cloud, you have likely hit the "10K cells ceiling" when using Hypercubes to fetch data from Qlik.
(Read my previous posts about Hypercubes here and here)

You build a data-driven component, it works perfectly with low-volume test data, and then you connect it to production; and now suddenly, your list of 50,000+ customers cuts off halfway, or your export results look incomplete.

This happens because the Qlik Engine imposes a strict limit on data retrieval: a maximum of 10,000 cells per request. If you fetch 4 columns, you only get 2,500 rows (4 (columns) x 2500 = 10,000 (max cells)).

In this post, I’ll show you how to master high-volume data retrieval using the two strategies: Bulk Ingest and On-Demand Paging, using the @qlik/api library.

What is the 10k Limit and Why Does It Matter?

The Qlik Associative Engine is built for speed and can handle billions of rows in memory. However, transferring that much data to a web browser in one go would be inefficient. To protect both the server and the client-side experience, Qlik forces you to retrieve data in chunks.

Understanding how to manage these chunks is the difference between an app that lags and one that delivers a good user experience.

Step 1: Defining the Data Volume

To see these strategies in action, we need a "heavy" dataset. Copy this script into your Qlik Sense Data Load Editor to generate 250,000 rows of transactions (or download the QVF attached to this post):

// ============================================================
// DATASET GENERATOR: 250,000 rows (~1,000,000 cells)
// ============================================================

Transactions:
Load
    RecNo() as TransactionID,
    'Customer ' & Ceil(Rand() * 20000) as Customer,
    Pick(Ceil(Rand() * 5), 
        'Corporate', 
        'Consumer', 
        'Small Business', 
        'Home Office', 
        'Enterprise'
    ) as Segment,
    Money(Rand() * 1000, '$#,##0.00') as Sales,
    Date(Today() - Rand() * 365) as [Transaction Date]
AutoGenerate 250000;

Step 2: Choosing Your Strategy

There are two primary ways to handle this volume in a web app. The choice depends entirely on your specific use case.

1- Bulk Ingest (The High-Performance Pattern)

In this pattern, you fetch the entire dataset into the application's local memory in iterative chunks upon loading.

The Goal: Provide a "zero-latency" experience once the data is loaded.
Best For: Use cases where users need to perform instant client-side searches, complex local sorting, or full-dataset CSV exports without waiting for the Engine.

2- On-Demand (The "Virtual" Pattern)

In this pattern, you only fetch the specific slice of data the user is currently looking at.

The Goal: Provide a near-instant initial load time, regardless of whether the dataset has 10,000 or 10,000,000 rows as you only load a specific chunk of those rows at a time.
Best For: Massive datasets where the "cost" of loading everything into memory is too high, or when users only need to browse a few pages at a time.

Step 3: Implementing the Logic

While I'm using React and custom react hooks for the example I'm providing, these core Qlik concepts translate to any JavaScript framework (Vue, Angular, or Vanilla JS). The secret lies in how you interact with the HyperCube.

The Iterative Logic (Bulk Ingest):

The key is to use a loop that updates your local data buffer as chunks arrive.

To prevent the browser from freezing during this heavy network activity, we use setTimeout to allow the UI to paint the progress bar.

        qModel = await app.createSessionObject({ qInfo: { qType: 'bulk' }, ...properties });
        const layout = await qModel.getLayout();
        const totalRows = layout.qHyperCube.qSize.qcy;
        const pageSize = properties.qHyperCubeDef.qInitialDataFetch[0].qHeight;
        const width = properties.qHyperCubeDef.qInitialDataFetch[0].qWidth;
        const totalPages = Math.ceil(totalRows / pageSize);

        let accumulator = [];
        for (let i = 0; i < totalPages; i++) {
          if (!mountedRef.current || stopRequestedRef.current) break;

          const pages = await qModel.getHyperCubeData('/qHyperCubeDef', [{
            qTop: i * pageSize,
            qLeft: 0,
            qWidth: width,
            qHeight: pageSize
          }]);

          accumulator = accumulator.concat(pages[0].qMatrix);

          // Update state incrementally
          setData([...accumulator]);
          setProgress(Math.round(((i + 1) / totalPages) * 100));

          // Yield thread to prevent UI locking
          await new Promise(r => setTimeout(r, 1));

The Slicing Logic (On-Demand)

In this mode, the application logic simply calculates the qTop coordinate based on the user's current page index and makes a single request for that specific window of data (rowsPerPage).

        const width = properties.qHyperCubeDef.qInitialDataFetch[0].qWidth;
        const qTop = (page - 1) * rowsPerPage;

        const pages = await qModelRef.current.getHyperCubeData('/qHyperCubeDef', [{
          qTop,
          qLeft: 0,
          qWidth: width,
          qHeight: rowsPerPage
        }]);

        if (mountedRef.current) {
          setData(pages[0].qMatrix);
        }

I placed these two methods in custom hooks (useQlikBulkIngest & useQlikOnDemand) so they can be easily re-used in different components as well as other apps.

Best Practices

Regardless of which pattern you choose, always follow these three Qlik Engine best practices:

Engine Hygiene (Cleanup): Always call app.destroySessionObject(qModel.id) when your component or view unmounts.
Cell Math: Always make sure your qWidth x qHeight is strictly < 10,000. For instance, if you have a wide table (20 columns), your max height is only 500 rows per chunk.
UI Performance: Even if you use the "Bulk" method and have 250,000 rows in JavaScript memory, do not render them all to the DOM at once. Use UI-level pagination or virtual scrolling to keep the browser responsive.

Choosing between Bulk and On-Demand is a trade-off between Initial Load Time and Interactive Speed. By mastering iterative fetching with the @qlik/api library, you can ensure your web apps remain robust, no matter how much data is coming in from Qlik.

Attached is the QVF and here is the GitHub repository containing the full example in React so you can try it in locally - Instructions are provided in the README file.

(P.S: Make sure you create the OAuth client in your tenant and fill in the qlik-config.js file in the project with your tenant-specific config).

Thank you for reading!

Mastering the HyperCube: High-Volume Data Strategies with @qlik/api