Skip to main content
Announcements
Join us at Qlik Connect for 3 magical days of learning, networking,and inspiration! REGISTER TODAY and save!
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Count in Canonical Date doesn't give correct values

Hi guys,

I feel more and more comfortable with Qlik Sense but some parts are still confusing. For example, today I got the canonical dates working...Well...halfway.

There is a vessel table which has 39,691 records; gate table with 60,214 records and moves with 71,765 records. The concatenated DateLink table has 171,670 records.

But then, if I use =count({<Flag={'Date_gate'}, Flag2={'Container_Gates'}>} Container) or

=count({<Flag={'Date_moves'}, Flag2={'Container_Moves'}>} Container) and select a day using the date from the filter panel '=Date(Floor(Date(TimeStamp,'MM/DD/YYYY hh:mm:ss.ffff')),'DD/MM/YYYY')'

I am not getting the correct values. They are usually higher than they should be. If I don't select any date, the count gives lower values than it should be i.e. Count using  Container_Moves, Container_Vessel and Container_Gate.

What am I doing wrong in my query?

Background: the vessel table contains information on the vessel visit i.e. one vessel visit could have many containers (container id) but one departure date (EST_DPTR_D).

The move table contains information on the container moves where [TET] is the hour of the move.

The gate table contains information on the time ([TRACTOR_DEPART_DATE]) a container left the port.

The requirement is to have one calendar on the front-end which will allow a user to pick a date and see the # of containers which were moved within the port (move table) and left the port (gate table). In theory, the number of moves in the vessel table should match with the number of moves in the moves table, so I use both tables interchangeable. For this reason, I named all three fields i.e. TET, EST_DPRT_D and TRACTOR_DEPART_DATE as TimeStamp. It was all good but I need to do some calculation which requires usage of columns from the vessel table and move table.

So, I looked into the canonical date feature. It looks good i.e. I can perform calculation using cols from both tables, but now I have an issue of bizzare count.

Vessel:

LOAD

[idvessl],

    `EST_DPTR_D` ,

    `CNTR_NUM` as Container

FROM [lib://Desktop/vesel_load.xlsx]

(ooxml, embedded labels, table is Sheet1);


Moves:

LOAD

[idmove],

[CONTAINER] as Container,

[TET]

FROM [lib://Desktop/move_vessel_upload.xlsx]

(ooxml, embedded labels, table is Sheet1);


Gate:

LOAD

[idgate],

[CNTR] as Container,

[TRACTOR_DEPART_DATE]

FROM [lib://Desktop/gate_10.xlsx]

(ooxml, embedded labels, table is Sheet1);


DateLink:

LOAD

Container

,'Container_Moves' as Flag2 

,Timestamp([TET],'DD/MM/YYYY hh:mm:ss tt') as TimeStamp   

  ,'Date_moves' as Flag  

RESIDENT Moves

;

Concatenate

LOAD

Container

,'Container_Gates' as Flag2

,Timestamp([TRACTOR_DEPART_DATE],'DD/MM/YYYY hh:mm:ss tt') as TimeStamp

,'Date_gate'  as Flag   

RESIDENT Gate

;

Concatenate

LOAD

Container

    ,'Container_Vessels' as Flag2

,Timestamp(`EST_DPTR_D`,'DD/MM/YYYY hh:mm:ss tt') as TimeStamp       

,'Date_vessel'  as Flag   

RESIDENT Vessel

;





//Creating a calendar

SUB CalendarFromField(_field, _calendar, _prefix)

[$(_calendar)]:


LOAD

Date(Floor(Date([$(_field)],'MM/DD/YYYY hh:mm:ss.ffff')),'DD/MM/YYYY') AS [$(_prefix)Date],

[$(_field)]

,year([$(_field)]) as [$(_prefix)Year]

,month([$(_field)]) as [$(_prefix)Month]

,day([$(_field)]) as [$(_prefix)Day]

,weekday([$(_field)]) as [$(_prefix)Weekday]

    ,hour([$(_field)]) as [$(_prefix)Hour]

    ,minute([$(_field)]) as [$(_prefix)Minute]   

;


LOAD

date(DateMin + IterNo()) as [$(_field)]

WHILE DateMin + IterNo() <= DateMax

;


LOAD

min(datefield)-1 as DateMin

,max(datefield) as DateMax

;


LOAD

FieldValue('$(_field)', RecNo()) as datefield

AutoGenerate FieldValueCount('$(_field)');


END SUB


CALL CalendarFromField('TimeStamp', 'CommonCalendar', '');

28 Replies
robert99
Specialist III
Specialist III

Can you post the Data model viewer

robert99
Specialist III
Specialist III

I think you must use applymap

and only link the finest grain table to the link table. Using a unique number. Either the ids if it is unique. Or a  row() as UniqueField in the finest grain table

I used

CallFSR_LINK,

in the example above but often use row() now as I can be sure it is unique

But load the  'Data model viewer'   and then I can help some more

Anonymous
Not applicable
Author

Here you go.

Like I said, 1 container could have several dates in Gate, Moves and Vessel table. We are interested in TET, EST_DPRT_D, and TRACTOR_DEPART_DATE.

The requirement is to have one calendar where a person could select a date range and see the # of containers - I am simplifying here - that left the port by vessel (dates in the EST_DPRT_D col), by a track (dates in the TRACTOR_DEPART_DATE col) and # of moves within the port (dates in the TET col).

It was working fine w/o using any concatenation until I realized that I need to show in one table some columns from the Vessel table and some calculations based on the Move table. As there is a bunch of filters - mostly built on the vessel table - I decided to built a canonical date view.

dv.png

Anonymous
Not applicable
Author

Let me read through.

robert99
Specialist III
Specialist III

Hi Ekaterina

I'm almost certain this will not work. as one Container includes many dates.

You need to do it as explained above.

  • Link the LinkTable to the table with the finest grain.There needs to be a one to one link with this table
  • And this one table must be many (or one) to one (not many) with the other 2 tables
  • The other two table. use mapping load when setting up the link table
  • You need to true date (not timestamp) to link with the calendar
  • The finest grain table must include all containers in the other two tables
Anonymous
Not applicable
Author

Robert, about '

  • And this one table must be many (or one) to one (not many) with the other 2 tables'

I don't think it will work because like I said, one container could have multiple dates in a corresponding date column, e.g. a container X could have several dates in a Est_departure date because it was, for example, brought up by one vessel by picked up by another; several dates in the gate table.

swuehl
MVP
MVP

Ekaterina,

first of all, don't COUNT(KeyField). The answer will most probably not a correct one.

See

getting wrong count result when using key field | Qlik Community

Do you want a COUNT(DISTINCT KeyField) instead?

Then, a solution to your link table issue might be to create a link table that essentially uses the unique record IDs of each table to link to each table, i.e. the link table will use the IDs as keys to the tables instead of the Container field.

Since you want Container to be a common filter, move the field to the link table, too.

Something like

LinkTable:

LOAD

     Container,

     VesselD,

     Dayname(VesselDate) as CanonicalDate,

     'Vessel' as Type

Resident Vessel;

Concatenate (LinkTable)

LOAD

     Container,

     MoveID,

     Dayname(MoveDate) as CanonicalDate,

     'Move'  as Type

Resident Moves;

// more fact tables

DROP FIELD Container FROM Vessel, Moves; // other fact tables

Not sure if this the best data model, but it should work.

robert99
Specialist III
Specialist III

Hi

OK then (I believe) you have an issue then. It must be a one to many join not many to many. With the LINKTABLE joined to the many table not the one.

Try Stefan's approach. If it works ignore the below. If not

When I have this situation I concatenate the tables (but you said above that you don't want to do this. But unsure why not) Lining up Container and the key dates. And also lining up a LINK field based on ['name' plus row()]  or the renamed ID keys (so they all have the same name). Although if you then have all the dates required in the one date column you don't really need a canonical date. If not use the canonical date option

Or another option. if you do have a one to many table (ie one table has only one row per container and the other two have more than one) just concatenate the two 'many tables' and join the one 'one table'.

I really like canonical date and I will accept the extra work required when I concatenate tables (and often its not that much) compared to joining. But I always join rather than concatenate if possible. But somethings concatenate give a better outcome.

Although in this example I would seriously consider concatenating tables anyway.

With each table giving a  set analysis name

'Move' as Tabletype (add to Movetable)

'Gate' as Tabletype  (add to Gatetable)

etc

Anonymous
Not applicable
Author

Hi Stefan,

Following Robert's suggestion, I created the id for each table using RowNo(). Following your suggestion, I used it RowNo for my LinkTable. I think I am getting the results, but let me test more. The high-level test, at least, was fine.

Question, though. Why did you use Dayname in the table and why if I use my format i.e. the timestamp format, the results are not correct? I guess you are doing the aggregation in this way. And finally, how can I get the timestamp if I follow your format? As I need to do some calculation based on the hour interval.

Sorry for the basic questions.

LinkTable:

LOAD

RowNo() as m_row_id,

    visit,

    Container

,Dayname([TET]) as TimeStamp   

  ,'Move' as Type

Resident Moves

;

Concatenate(LinkTable)

LOAD

RowNo() as v_row_id,

    visit,

    Container

,Dayname(`EST_DPTR_D`) as TimeStamp       

,'Vessel'  as Type

Resident Vessel

;

Concatenate(LinkTable)

LOAD

RowNo() as g_row_id,

    Container

,Dayname([TRACTOR_DEPART_DATE]) as TimeStamp       

,'Gate'  as Type

Resident Gate

;