Skip to main content
Announcements
See what Drew Clarke has to say about the Qlik Talend Cloud launch! READ THE BLOG
cancel
Showing results for 
Search instead for 
Did you mean: 
asgardd2
Creator III
Creator III

Drop data after Binary Load (application size is not reduced)

Hello!

     After reloading via the binary load (second application), I droped unwanted data (tables, fields). But the size of the application is practically not reduced compared to the original application.

Note!! If I do not have current selections in second application and choose the option to reduce the data size -  the application is substantially reduced. (i expect to see it after drop)

     How i can fix it? I have one solution to fix it, but i think this is not right way:

     1. Store Transaction table, then original application is reloading. (this table is very big)

     2. After "Binary load" in second application i will drop this table and then load this table from qvd file.

Best regards,

Anton Aleksandrov.

1 Solution

Accepted Solutions
marcus_sommer

I think the reason of this behaviour is that no processing happened by the binary load then the datamodel will be simply transferred into the RAM and by dropping some tables/fields afterwards no real processing is triggered else only the metadata of the tables/fields are removed or maybe even only invisible.

Therefore the processing must be triggered in any way. Maybe it could be happen with some dummy load-statement like:

dummy: load 0 as dummy autogenerate 0;

drop table dummy;

or with a direct accessing a loaded table by loading or adding something and removing it again.

- Marcus

View solution in original post

8 Replies
Gysbert_Wassenaar

Do you keep the file open after reloading or do you save the document, close it and only then look at the file size?


talk is cheap, supply exceeds demand
asgardd2
Creator III
Creator III
Author

Yes, i save + close the document and then reopen it.

Gysbert_Wassenaar

Ok, try this:

Just do a binary load and don't drop any tables. Then check the sizes of all the tables in the Document Properties. Specifically of the tables you want to drop. How significant is the size of those tables compared to the total of all tables?


talk is cheap, supply exceeds demand
asgardd2
Creator III
Creator III
Author

I would like drop second table :

And drop some fields from first table:

(all values from export memory statistics)

     Only now I do not understand why only after reducing the data (although none of the filters is not selected) application size decreases.

Gysbert_Wassenaar

Well, no idea really. My guess is that the Reduce data action creates a new defragmentated file whereas a simple save after reload and dropping some tables doesn't do any defragmentation.


talk is cheap, supply exceeds demand
asgardd2
Creator III
Creator III
Author

     Perhaps this is true, I will have to use qvd file(main transaction table) in script.

Thank you!

marcus_sommer

I think the reason of this behaviour is that no processing happened by the binary load then the datamodel will be simply transferred into the RAM and by dropping some tables/fields afterwards no real processing is triggered else only the metadata of the tables/fields are removed or maybe even only invisible.

Therefore the processing must be triggered in any way. Maybe it could be happen with some dummy load-statement like:

dummy: load 0 as dummy autogenerate 0;

drop table dummy;

or with a direct accessing a loaded table by loading or adding something and removing it again.

- Marcus

asgardd2
Creator III
Creator III
Author

I used variant with the addition of a dummy field to the table, and then delete it.


Concatenate(Transactions)

load

    0 as dummy

Autogenerate 1;

DROP FIELD dummy;


After that, the application size has decreased.
Thank you!