mhmmd_srf
Creator II
2023-07-30
09:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Which is faster for 1M records per day CSV or Rest API?
Hi All,
I have incremental load to create QVD. It takes last one days data and append. But, the volume of last one day data is nearly 1M and this is taking nearly 10 hours to just read the data. My source is Service Now and I am connecting to Service Now through rest API.
If I get the data dump in cvs format, will be faster than fetching data directly from Service Now?
Thanks,
Sarif
350 Views
1 Reply
deepanshuSh
Creator III
2023-07-30
12:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, Yes Excel file would definitely boost the loading time, but do consider that there is a limitation of the excel of maximum of 104876 ( approx 1M) so the data might get lost in the process if volume is high and records are more than the set limit.
Trial and error is the key to get unexpected results.
335 Views