loading data from excel/csv auto into my qlik sense app

[124rooski]

I'm working on some COVID data that the WHO publishes out daily with the confirmed death counts, cases, etc. What is the best way to append the data from excel/csv into my app without having to import this constantly through the data manager? I was thinking of setting a where clause so if column DATE is >= Today() then import it in. 

[JustinDallas]

Do you have a sample of the data?  Hopefully, your data lends itself to incremental loading.

[124rooski]

I go to this website https://ourworldindata.org/coronavirus-source-data to grab the data (excel/csv) under "Full dataset". It looks like they update it daily and I'd ideally want my app to just append those new rows since the file name, etc will be the same.

[JustinDallas]

Does it absolutely have to be an incremental load?  There are only 5K+ rows, and the size of the CSV is only 160KB.  You can easily make a Web Connection that will pull down the file and then go about your way.

Generally, when I use incremental loads, it's because a full load is expensive resource-wise.

Because you don't control how much source data you read (you can't partially download the file) you have to do a full data load anyway.

Sorry I couldn't be more useful.

[124rooski]

No not at all thanks for your help. I am going to give this a shot. Does this pretty much override whats already in my app with the updated data? Do I need to set up a task in the QMC so that it refreshes and retrieves that URL in order to update my data?

[JustinDallas]

I'm not sure what your setup looks like, but I would generally have a task that would fire off, and pull down the file into a Repository folder.

The task could be as regular as you need it to be.  And yes, this will steamroll any data that your app already has in it concerning this file.

[124rooski]

I'm getting hit with this error here. Any idea how to overcome this? I noticed in your URL you had "ecdc" and wondering how that came about? Is there an initial set up I need to do before I can use the web file connection?