There is however also a second case where you want to create reference dates, but the data model is quite different. It is when you have a list of contracts with validity periods: Rental contracts, Insurances policies, Healthcare commitments, etc.
Each contract has a begin day and an end day. The analysts of an insurance company would probably want to ask the question: “How many valid insurance policies did we have on this specific day?” In other words, he wants to click on a reference date to see the count of policies that are associated with this date – even though this date doesn't exist in the source data.
The solution is to first load all policies in one table. Then load a second table that has one record per policy and date (Policies_x_Dates). This second table can in turn link to a master calendar.
But the middle table does not exist in the source database. Instead, you must generate it using a While loop that loops over each record in the Policies table, i.e. each source record will be loaded several times. This way, you can load not only the “From” date and the “To” date, but also all dates in between:
Policies: LoadPolicyID, BirthDate, PolicyAmount, FromDate, ToDate, OtherPolicyAttribute From Policies;
Policies_x_Dates:
LoadPolicyID, Age( FromDate + IterNo() – 1, BirthDate ) as Age, PolicyAmount / (ToDate - FromDate + 1) as DailyAmount, Date( FromDate + IterNo() – 1 ) as ReferenceDate Resident Policies While IterNo() <= ToDate - FromDate + 1 ;
Note that the Policies table has exactly one record per insurance policy, and the newly created Policies_x_Dates table has exactly one record per combination of policy and date. Note also that there are other fields that should be put in the Policies_x_Dates table, e.g., the age of the insured person, since this depends on the reference date. Further, it is possible to break up a cost or an income into daily amounts, which is useful when you want to show the correct amount distributed over the year.
The While loop is a very useful tool whenever you need to create additional tables in the data model. It is often a better option than the IntervalMatch.
i need your help for a similar problem i have a contract table with start and enddate and an invoice table with invoice date using Hendric's approach to generate reference date for my contract table is working nice
but i have a problem with linking the invoice date from invoice Table with the master calendar since loading invoice dates into my linktable resindent to invoice Table is duplicating my data in invoice Table and sum of Amount from invoice table is giving back the same amount regarless of Month.
ps: sum for value from contract table is working perfect for day , month, year...
@patlechat237 I am not sure I understand how you are linking the Linktable to the Invoice table - should not be particularly related to the Contracts table.
I have a complex issue to deal with, I have two tables, where each record represent an event in a system.
The first table has all events that represent a new entry for detailing information of a field (cod) used in the other table, these events have multiple fields that later on are used as criteria for calculation.
The second table has other events that use the field (cod) to associate the values of this table with the details that are in the other table, so far so good, but, the first table can have multiple entries for the same cod, and also there is a field for start date and end period of validity of the events in the first table.
What I need to do is, I need to search the first table based on some criteria:
1. events in the first table must have been included before the events was created in the second table, so here is the first datetime to compare between the tables (datetime the event was created).
2. period of the events in the second table must be between the start and end date of the event in the first table (period is a YYYY-MM format)
3. the end period in the first table can be null, so there are multiple scenarios where more than one event is applicable, in those cases I need to retrieve the latest one considering criteria 1
I have tried using IntervalMatch, but I endup with duplicates when joining the two tables, because of the events that get attached to more than one cod.
@fnfr Hey! I realize this will be a pretty generic answer, but this is a tough issue that will take some time. This is going to require multiple steps of transformation and prioritization of that transformation.
As a start, you'll need to clean table one to either purge the duplicate rows or house the unique data per unique cod in a separate table.
You'll also need to replace Null() end dates with today() or a similar representation of current time.
Lastly, both dates may need to be joined to calendars if you have heavy complications in comparison criteria. This is usually avoided via interval match but IM does not appear usable here.
Another comparison you can use is simple if()s on dates - though this is heavy and inefficient, it may also help to remove some confusion initially.
@CJ_Bauder I hear you, I have tried multiple ways of achieving this, I got close, some cases I got 99% precision, but than there are others where I got 72%.
It is really tricky depending on how are the entries in the first table, I guess I will just try another tool or programming language.