About data loading

[#1]

Hi All,

I want to ask about data loding, Can i load data only from one load scenario, ok i explain, i have 5 tab of loading scenario: Main, Customer, Sales etc, and when i modificate data in database, i want to load only customer updated data from customer load scenario, not of all, becouse it toks very long time.

[jagan]

Hi,

Try Partial Reload check Help file.

Regards,

Jagan.

[#3]

Hello,

You can have a variable used as a flag. The variable will be populated with customized value which identifies your different scenarios. This variable can then be used in the script using IF statement.

else

You can try Partial reload OR modify the queries for each tab in script to return result only when correct parameters are passed.

Thanks,

Angad

[#4]

thanks a lot

[#5]

if the answer was helpful, please mark the question as answered or helpful.

Thanks

[magavi_framsteg]

Hi Sigitas.

If you search the forums for "incremental load" or "incremental reload" you will find dozens of posts about it.

This particular post shows quite a few diffrent techniques for achieving what you want: Incremental reload

Kind regards

Magnus Åvitsland

BI Architect Consultant

Framsteg Business Intelligence Corp.

[sspe]

I actually asked the same question for a few weeks ago, and there is a little more detailed suggestion in that thread - http://community.qlik.com/thread/101459

Regards

Steen

[Peter_Cammaert]

Just to provide you with some context:

What you're asking for is usually solved by employing a technique called INCREMENTAL LOADING e.g. loading only those records that are new or have changed since your last check. Other names are Delta Loads, Diffs etc.

Incremental loading is fairly easy to implement if you have some field in your database that marks changes to some record set, e.g. a Sales Order date in your Sales table, or a Last Changed field in other tables. You see that for some tables it will be awfully difficult to figure out which records have changed. MasterData tables aka Dimensions are infamous for this, but on average they aren't too big. It's another story with Facts tables where record counts easily run in the millions and the situation might get even worse if you plan to reload often with short intervals.

QlikView has some interesting features to help you with incremental loading, like the Partial Reload-mode, the BUFFER prefix, QVDs as temporary containers to check against etc. Check them out.

Best,

Peter

[#9]

Hi,

Please find the below ex,

Incremental Data Loads

In organizations where data volumes are large or batch windows are small, there will be a requirement to

Implement incremental loads therefore only load data that has changed or added since your last load. These

Types of data loads can only be implemented if QVD files are used.

Various papers are available that detail the Incremental load process further.

Large Data Loads

QVD files can also be used to massively improve the loading of historical data in to your QlikView application.

QVD files can load millions of rows in seconds.

Large Data Example

In an organization that has masses of transactional data going back 5 years, you don’t want to have to load all

5 years including the current years (growing) data every night. One solution to this is below:

Create QVD Files for each year you wish to analyze. These can be created dynamically based on how many

Years you wish to look at. The script below show how to create a simple QVD file based on a moving date

clause.

'Year-5':

Load Id,

StoreName,

StoreLocation,

Date,

Product,

Sales

FROM C:\Data.xls (biff, embedded labels, table is [Core Data$])

Where Year(Date) = YEAR(TODAY())-5;

Store 'Year-5' into C:\Year-5.qvd;

Drop Table 'Year-5';

Creating multiple instances of this script above and simply changing the Where clause will enable you to

create a set of yearly QVD files based on the number of years of analysis you require, along with a current

year QVD file ( Where Year(Date) = YEAR(TODAY()) ).

Once all of the necessary QVD files are created, you can then source your end user application on these files.