Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi guys,
I feel more and more comfortable with Qlik Sense but some parts are still confusing. For example, today I got the canonical dates working...Well...halfway.
There is a vessel table which has 39,691 records; gate table with 60,214 records and moves with 71,765 records. The concatenated DateLink table has 171,670 records.
But then, if I use =count({<Flag={'Date_gate'}, Flag2={'Container_Gates'}>} Container) or
=count({<Flag={'Date_moves'}, Flag2={'Container_Moves'}>} Container) and select a day using the date from the filter panel '=Date(Floor(Date(TimeStamp,'MM/DD/YYYY hh:mm:ss.ffff')),'DD/MM/YYYY')'
I am not getting the correct values. They are usually higher than they should be. If I don't select any date, the count gives lower values than it should be i.e. Count using Container_Moves, Container_Vessel and Container_Gate.
What am I doing wrong in my query?
Background: the vessel table contains information on the vessel visit i.e. one vessel visit could have many containers (container id) but one departure date (EST_DPTR_D).
The move table contains information on the container moves where [TET] is the hour of the move.
The gate table contains information on the time ([TRACTOR_DEPART_DATE]) a container left the port.
The requirement is to have one calendar on the front-end which will allow a user to pick a date and see the # of containers which were moved within the port (move table) and left the port (gate table). In theory, the number of moves in the vessel table should match with the number of moves in the moves table, so I use both tables interchangeable. For this reason, I named all three fields i.e. TET, EST_DPRT_D and TRACTOR_DEPART_DATE as TimeStamp. It was all good but I need to do some calculation which requires usage of columns from the vessel table and move table.
So, I looked into the canonical date feature. It looks good i.e. I can perform calculation using cols from both tables, but now I have an issue of bizzare count.
Vessel:
LOAD
[idvessl],
`EST_DPTR_D` ,
`CNTR_NUM` as Container
FROM [lib://Desktop/vesel_load.xlsx]
(ooxml, embedded labels, table is Sheet1);
Moves:
LOAD
[idmove],
[CONTAINER] as Container,
[TET]
FROM [lib://Desktop/move_vessel_upload.xlsx]
(ooxml, embedded labels, table is Sheet1);
Gate:
LOAD
[idgate],
[CNTR] as Container,
[TRACTOR_DEPART_DATE]
FROM [lib://Desktop/gate_10.xlsx]
(ooxml, embedded labels, table is Sheet1);
DateLink:
LOAD
Container
,'Container_Moves' as Flag2
,Timestamp([TET],'DD/MM/YYYY hh:mm:ss tt') as TimeStamp
,'Date_moves' as Flag
RESIDENT Moves
;
Concatenate
LOAD
Container
,'Container_Gates' as Flag2
,Timestamp([TRACTOR_DEPART_DATE],'DD/MM/YYYY hh:mm:ss tt') as TimeStamp
,'Date_gate' as Flag
RESIDENT Gate
;
Concatenate
LOAD
Container
,'Container_Vessels' as Flag2
,Timestamp(`EST_DPTR_D`,'DD/MM/YYYY hh:mm:ss tt') as TimeStamp
,'Date_vessel' as Flag
RESIDENT Vessel
;
//Creating a calendar
SUB CalendarFromField(_field, _calendar, _prefix)
[$(_calendar)]:
LOAD
Date(Floor(Date([$(_field)],'MM/DD/YYYY hh:mm:ss.ffff')),'DD/MM/YYYY') AS [$(_prefix)Date],
[$(_field)]
,year([$(_field)]) as [$(_prefix)Year]
,month([$(_field)]) as [$(_prefix)Month]
,day([$(_field)]) as [$(_prefix)Day]
,weekday([$(_field)]) as [$(_prefix)Weekday]
,hour([$(_field)]) as [$(_prefix)Hour]
,minute([$(_field)]) as [$(_prefix)Minute]
;
LOAD
date(DateMin + IterNo()) as [$(_field)]
WHILE DateMin + IterNo() <= DateMax
;
LOAD
min(datefield)-1 as DateMin
,max(datefield) as DateMax
;
LOAD
FieldValue('$(_field)', RecNo()) as datefield
AutoGenerate FieldValueCount('$(_field)');
END SUB
CALL CalendarFromField('TimeStamp', 'CommonCalendar', '');
Can you post the Data model viewer
I think you must use applymap
and only link the finest grain table to the link table. Using a unique number. Either the ids if it is unique. Or a row() as UniqueField in the finest grain table
I used
CallFSR_LINK,
in the example above but often use row() now as I can be sure it is unique
But load the 'Data model viewer' and then I can help some more
Here you go.
Like I said, 1 container could have several dates in Gate, Moves and Vessel table. We are interested in TET, EST_DPRT_D, and TRACTOR_DEPART_DATE.
The requirement is to have one calendar where a person could select a date range and see the # of containers - I am simplifying here - that left the port by vessel (dates in the EST_DPRT_D col), by a track (dates in the TRACTOR_DEPART_DATE col) and # of moves within the port (dates in the TET col).
It was working fine w/o using any concatenation until I realized that I need to show in one table some columns from the Vessel table and some calculations based on the Move table. As there is a bunch of filters - mostly built on the vessel table - I decided to built a canonical date view.
Let me read through.
Hi Ekaterina
I'm almost certain this will not work. as one Container includes many dates.
You need to do it as explained above.
Robert, about '
I don't think it will work because like I said, one container could have multiple dates in a corresponding date column, e.g. a container X could have several dates in a Est_departure date because it was, for example, brought up by one vessel by picked up by another; several dates in the gate table.
Ekaterina,
first of all, don't COUNT(KeyField). The answer will most probably not a correct one.
See
getting wrong count result when using key field | Qlik Community
Do you want a COUNT(DISTINCT KeyField) instead?
Then, a solution to your link table issue might be to create a link table that essentially uses the unique record IDs of each table to link to each table, i.e. the link table will use the IDs as keys to the tables instead of the Container field.
Since you want Container to be a common filter, move the field to the link table, too.
Something like
LinkTable:
LOAD
Container,
VesselD,
Dayname(VesselDate) as CanonicalDate,
'Vessel' as Type
Resident Vessel;
Concatenate (LinkTable)
LOAD
Container,
MoveID,
Dayname(MoveDate) as CanonicalDate,
'Move' as Type
Resident Moves;
// more fact tables
DROP FIELD Container FROM Vessel, Moves; // other fact tables
Not sure if this the best data model, but it should work.
Hi
OK then (I believe) you have an issue then. It must be a one to many join not many to many. With the LINKTABLE joined to the many table not the one.
Try Stefan's approach. If it works ignore the below. If not
When I have this situation I concatenate the tables (but you said above that you don't want to do this. But unsure why not) Lining up Container and the key dates. And also lining up a LINK field based on ['name' plus row()] or the renamed ID keys (so they all have the same name). Although if you then have all the dates required in the one date column you don't really need a canonical date. If not use the canonical date option
Or another option. if you do have a one to many table (ie one table has only one row per container and the other two have more than one) just concatenate the two 'many tables' and join the one 'one table'.
I really like canonical date and I will accept the extra work required when I concatenate tables (and often its not that much) compared to joining. But I always join rather than concatenate if possible. But somethings concatenate give a better outcome.
Although in this example I would seriously consider concatenating tables anyway.
With each table giving a set analysis name
'Move' as Tabletype (add to Movetable)
'Gate' as Tabletype (add to Gatetable)
etc
Hi Stefan,
Following Robert's suggestion, I created the id for each table using RowNo(). Following your suggestion, I used it RowNo for my LinkTable. I think I am getting the results, but let me test more. The high-level test, at least, was fine.
Question, though. Why did you use Dayname in the table and why if I use my format i.e. the timestamp format, the results are not correct? I guess you are doing the aggregation in this way. And finally, how can I get the timestamp if I follow your format? As I need to do some calculation based on the hour interval.
Sorry for the basic questions.
LinkTable:
LOAD
RowNo() as m_row_id,
visit,
Container
,Dayname([TET]) as TimeStamp
,'Move' as Type
Resident Moves
;
Concatenate(LinkTable)
LOAD
RowNo() as v_row_id,
visit,
Container
,Dayname(`EST_DPTR_D`) as TimeStamp
,'Vessel' as Type
Resident Vessel
;
Concatenate(LinkTable)
LOAD
RowNo() as g_row_id,
Container
,Dayname([TRACTOR_DEPART_DATE]) as TimeStamp
,'Gate' as Type
Resident Gate
;