How can I pull the results of a query in aws into Talend?

[pthomas]

I have a bucket with multiple parquet files.  I need to get from all of the files within a folder, the unique ids.  I need to have that in Talend so I can loop through the next steps.

In terminal, using spark, I can run the statement below and get the ids that I need.  Is there a way to run that in a tSystem or some other component that will return the df list?

df = spark.read.parquet("s3a://talend/bronze/books/").select("bookId").distinct()

df.show(false)

[Shicong_Hong · Accepted solution]

Hello 

You can run PySpark commands in a tSystem component like you did in terminal. Refer to these topics to learn how to execute a python script file using tSystem component.

https://community.qlik.com/t5/Talend-Studio/how-to-execute-a-python-script-file-with-an-argument-usi...

https://community.qlik.com/t5/Talend-Studio/using-tsystem-in-studio-in-windows-to-call-python-script...

Before running the PySpark commands, you must have a Spark environment set up.

Regards

Shicong