Python SSE: how to go through same RequestIterator multiple times

[adamlopuch]

Hi Qlik SSE gurus,

The Python SSE is working like a charm, however in one call I'd like to go through the *same* request payload *twice*... is this possible?

My goal:

1. FIRST ITERATION: go through each row and extract all parameters
2. Parse all distinct parameter values in entire request
3. SECOND ITERATION: go through each row AGAIN and yield result with distinct parameters

My issue:

I suspect that the (simple) approach I am using fully consumes the request iterator the first time I go through it. (Step 1).  When I go through it a second time (Step 3), there is nothing to iterate.

Example code:

# STEP 1: Go through each row and extract all parameters
all_params = []
for bundled_rows in request:
    for row in bundled_rows.rows:
        all_params.append([d.strData for d in row.duals])

# STEP 2: Get all values for parameters across entire request
distinct_params = something_with(all_params)

# STEP 3: go through each row again and use the distinct parameters
for bundled_rows in request:
    response_rows = []
    for row in bundled_rows.rows:
        duals = iter([SSE.Dual(numData=something_with(distinct_params)])
        response_rows.append(SSE.Row(duals=duals))
    yield SSE.BundledRows(rows=response_rows)

[Nabeel_Asif · Accepted solution]

Hi @adamlopuch ,

Your suspicion is correct; the request is a generator object which can only be consumed once in Python. One way around it is to first store the request in a different data structure such as a list that you can step through multiple times.

# Get a list from the generator object so that it can be iterated over multiple times
request_list = [request_rows for request_rows in request]

In my SSE I use a function that can then take this list and convert it to a pandas dataframe which is usually a more convenient structure for Python:

def request_df(request_list, row_template, col_headers):
    """
    This function takes in a SSE request as a list together with a row template and column headers as lists of strings.
    Returns a Data Frame for the request.
    e.g. request_df(request_list, ['strData', 'numData', 'strData'], ['dim1', 'measure', 'kwargs'])
    """
    
    rows = [row for request_rows in request_list for row in request_rows.rows]
    outer = []
    
    for i in range(len(rows)):
        inner = []
        
        for j in range(len(row_template)):
            inner.append(getattr(rows[i].duals[j], row_template[j]))
        
        outer.append(inner)
    
    return pd.DataFrame(outer, columns=col_headers)

Finally, at the end of your function you'll need to structure the output as SSE.Duals inside SSE.Rows inside SSE.BundledRows 😂.