Skip to main content
Announcements
Qlik Introduces a New Era of Visualization! READ ALL ABOUT IT
cancel
Showing results for 
Search instead for 
Did you mean: 
Kartik2
Partner - Creator
Partner - Creator

Select Only the latest file from the Azure storage data connection

I want to automatically fetch only the first file into qlik sense from this path. How can do that? There is a date column in that excel file. 

Kartik2_0-1710217007831.png

 

Labels (5)
1 Solution

Accepted Solutions
Levi_Turner
Employee
Employee

Example logic (we need to use the Azure Storage Metadata Connector):

LIB CONNECT TO 'Azure_Storage_Metadata';
// Get a list of files, you will likely want to filter on prefix
[list_of_files]:
LOAD 
	Name as [file_name], 
	[Creation-Time] as [file_time_created], 
	[Last-Modified] as [file_time_modified];

SELECT Name,
	[Creation-Time],
	[Last-Modified]
FROM ListBlobs
WITH PROPERTIES (
prefix='',
includeUncommitted='false',
maxResults=''
);

// Get most recent file, we are ordering by the file_time_created field in order to ensure sort order for the later variable
[most_recent_file_list]:
NoConcatenate
LOAD
	[file_name],
    [file_time_created] AS [file_time_created],
    [file_time_modified] AS [file_time_modified]
RESIDENT [list_of_files]
ORDER BY [file_time_created] DESC;

// Drop initial load
DROP TABLE [list_of_files];

// Select first record in the sorted file list
LET v_file = Peek('file_name',0,'most_recent_file_list');

// Load the file, your load logic will need to be adjusted to reflect the schema of your file
[data]:
LOAD
    *
FROM [lib://Azure_Storage/$(v_file)] (txt, codepage is 28591, embedded labels, delimiter is ',', msq);

DROP TABLE [most_recent_file_list];

View solution in original post

1 Reply
Levi_Turner
Employee
Employee

Example logic (we need to use the Azure Storage Metadata Connector):

LIB CONNECT TO 'Azure_Storage_Metadata';
// Get a list of files, you will likely want to filter on prefix
[list_of_files]:
LOAD 
	Name as [file_name], 
	[Creation-Time] as [file_time_created], 
	[Last-Modified] as [file_time_modified];

SELECT Name,
	[Creation-Time],
	[Last-Modified]
FROM ListBlobs
WITH PROPERTIES (
prefix='',
includeUncommitted='false',
maxResults=''
);

// Get most recent file, we are ordering by the file_time_created field in order to ensure sort order for the later variable
[most_recent_file_list]:
NoConcatenate
LOAD
	[file_name],
    [file_time_created] AS [file_time_created],
    [file_time_modified] AS [file_time_modified]
RESIDENT [list_of_files]
ORDER BY [file_time_created] DESC;

// Drop initial load
DROP TABLE [list_of_files];

// Select first record in the sorted file list
LET v_file = Peek('file_name',0,'most_recent_file_list');

// Load the file, your load logic will need to be adjusted to reflect the schema of your file
[data]:
LOAD
    *
FROM [lib://Azure_Storage/$(v_file)] (txt, codepage is 28591, embedded labels, delimiter is ',', msq);

DROP TABLE [most_recent_file_list];