Skip to main content
Announcements
See what Drew Clarke has to say about the Qlik Talend Cloud launch! READ THE BLOG
cancel
Showing results for 
Search instead for 
Did you mean: 
kcrouse
Contributor II
Contributor II

Interval and time explanation for creating a new SchemaEvent

Hi folks, 

I'm looking to automate the creation and management of schema event tasks to reload external programs and apps on a schedule, but I can't find a definition of the interval fields to handle this. 

tl;dr, how does one set  schemaFilterDescription and incrementDescription to define repetitive events?  They're not described in the API reference documentation.

I know that:

"incrementOption" is an enum as follows:
  •  0: once
  • 1: hourly
  • 2: daily
  • 3: weekly
  • 4: monthly

And I think, between schemaFilterDescription, incrementDescription, and incrementOption, it controls the specifics, but these aren't defined anywhere I can see. schemaFilterDescription (default "['* * - * * * * *']" ) looks similar to a cron entry, but a cron line doesn't have a dash and only has 6 elements... and when I tried to guess at it led to some crazy next reload times.  

For example, I created a schema event with the following fields, intending it to execute at 8pm daily:

Post to /qrs/schemaevent with data:

 

{
    ...
    'incrementOption': 2,  # daily
    'incrementDescription': '0 0 1 0', # ??
    'schemaFilterDescription': ['* * - * 20 * * *'], # 8pm? Doesn't appear so.
}

 

 

 

And it set the next execution time for for July. I see a few posts in which people ask a question and someone responds with the correct a line to get the desired results, but what do the fields mean and how do we set them to do what we want in general?

Labels (5)
5 Replies
Levi_Turner
Employee
Employee

Here's a writeup that I've found useful before:

SchemaFilterDescription
Default: "* * - * * * *"
This is a list of SchemaFilterDescriptions used to filter when a SchemaEvent is allowed to trigger. An important thing to remember is that what is set as SchemaFilterDescriptions are when it is allowed to trigger and not the opposite.

For Minute, Hour, WeekDay (using numeric values), DayOfMonth and Month you can use syntax with hyphen to state "from - to". For the others you must state each character to be used in the SchemaFilterDescription, this can of course also be used by Minute, Hour, WeekDay, DayOfMonth and Month as well. Example: You only want to allow the first 15 minutes each hour you can either put ‘1-15’ or ‘1,2,3,4,5,6,7,8,9,10,11,12,13,14,15’ in the first position. There should be no spaces in the SchemaFilterDescription other than between each position, the space character is used as a delimiter between the positions so inserting one would cause the SchemaFilterDescription to, at best, not work as intended or most likely to cause an error.


Starting from left to right (first position is 0) each position is explained below.

Position 0 - Minute
Legend: 1-60. '*' = all

Which minute of an hour (0 - 60) when the SchemaEvent is allowed to trigger.

Position 1 - Hour
Legend: 1-24. '*' = all

Which hour of a day (1-24) when the SchemaEvent is allowed to trigger.

Position 2 - WeekDayPrefix
Legend: 1-4. '¤' = last. '-' = none

WeekDayPrefix works together with WeekDay by adding a prefix. By means of this you can state that only the last (¤) friday in a given month or 1st (1) saturday in a given month is allowed.

Position 3 - WeekDay
Legend: 0-6 (where sunday is 0)

Which weekday the SchemaEvent is allowed to trigger. See also WeekDayPrefix for further information.

Position 4 - WeeklyInterval
Legend: An integer. '*' = all

The SchemaEvent is allowed to trigger every n:th week, where n is the number set in this position.

Position 5 - DayOfMonth
Legend: 1-31. '*' = all. '¤' = last

Which day within a month when the SchemaEvent is allowed to trigger. Using last (¤) the Scheduler will only allow the SchemaEvent to trigger on the last day in a month which is checked dynamically depending on month (and leap year).

Position 6 - Month
Legend: 1-12. '*' = all

Which month of the year when the SchemaEvent is allowed to trigger.



Practically I've not built out full work-flows across all the options, but here's an example in PowerShell (https://gist.github.com/levi-turner/f4a431b306c399c7c5caf0406a0a8be9#file-qs-qrs-telemetry_task_fix-...) which does weekly or daily for reference.

kcrouse
Contributor II
Contributor II
Author

Hi Levi,

Thanks for this information. It's helpful and seems to mostly work.  You only defined 7 positions (0-6) but the string has 8 positions.  I did some further brute force testing, and it looks like Position 7 is MonthlyInterval (n number of months).

So, to summarize SchemaFilterDescription by position:

  1. Minute (1-60)
  2. Hour (1-24)
  3. WeekdayPrefix/Week-of-Month, '-' means no prefix, i.e. no restriction, which is interesting
  4. Weekday (0-6)
  5. Weekly Interval, i.e. Every N weeks
  6. DayofMonth(1-31)
  7. Month (1-12)
  8. Monthly Interval, i.e. Every N Weeks

IncrementOption

I am not sure if IncrementOption does anything. I can set it to 0, 1, 2, 3 and it doesn't seem to interfere with scheduling.  Any thoughts on that?

IncrementDescription

In contrast, IncrementDescription seems to be important, mysterious, and prone to unexpected outcomes. It *appears* that incrementDescription operates like a bitmask and needs to be in sync with the Schema Filter Description, whereas the first value is linked to minutes and the second value is linked to hours. Presumably, the third is linked to days and the fourth either weeks or months.   

If IncrementDescription is zeroed out '0 0 0 0', the event will not be scheduled, regardless what your schema filter Description is.

If the minutes are set, IncrementDescription must have the first parameter set to 1, or no events will be scheduled:

 

 

# This is correct:
{
    "incrementDescription": "1 0 0 0",
    "schemaFilterDescription": ["15 14 - * * * * *"],
}

# This combination will not be scheduled.
{
    "incrementDescription": "0 1 0 0",
    "schemaFilterDescription": ["15 14 - * * * * *"],
}

 

 

If minutes is '*' and hours are set, the required formats are different:

 

 

# This works fine:
{
    "incrementDescription": "0 1 0 0",
    "schemaFilterDescription": ["* 14 - * * * * *"],
}

# This does not:
{
    "incrementDescription": "0 0 1 0",
    "schemaFilterDescription": ["* 14 - * * * * *"],
}

 

 

Levi_Turner
Employee
Employee

RE IncrementOption, it doesn't impact things. Nominally it has enum values but ultimately it's the schemaFilterDescription which fuels the actual schedule (as well as IncrementDescription based on your testing)

RE IncrementDescription, I honestly have not delved into things to this degree to exhaustively diff test different configurations.

simonaubert
Partner - Specialist II
Partner - Specialist II

Hi guys,

just a big thank for the explanations. It helped me a lot to build by monitoring app.

Best regards,

Simon

Bi Consultant (Dataviz & Dataprep) @ Business & Decision
k21
Contributor II
Contributor II

Hi All,  I think I need ur help, 

I am trying to create some tasks based on a specific frequency ( Daily in a specific hour, weekly on a specific day and monthly in a specific DayOfMonth .. But this is not working (The task is created fine but it doesnt have a trigger event .. ), any ideas what's the issue ? 

Thanks in advance .. 

        if [ "$TaskFrequence" == "Daily" ]; then
            schemaFilterDescription="* $TaskHour - * * * * *"
            incrementDescription="0 1 0 0"
            incrementOption="1"
        elif  [ "$TaskFrequence" == "Weekly" ]; then
            schemaFilterDescription="* * - $WeekDay 1 * * *"
            incrementDescription="0 0 1 0"
            incrementOption="1"
        elif  [ "$TaskFrequence" == "Monthly" ]; then
            schemaFilterDescription="* * - * * $DayOfMonth * *"
            incrementDescription="0 0 1 0"
            incrementOption="4"