Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hello Guys,
We have been tasked to demonstrate streaming using Talend. We have seen that Talend Data Fabric 6.3.1 has Big Data Streaming where you can create streaming jobs similar to integration jobs. We have chosen to create a Spark streaming job. We tried to follow the simple spark streaming job from the given talend documentation where we will generate data from tRowGenerator and load it into a given HDFS folder in AVRO format. Below is the job and the corresponding spark configuration we used:
When the job was run we have received this error:
We also tried to run it using local mode in spark but it still produced the same error. We are not sure where we went wrong with this one. We also tried to use Kafka into the job as shown below, by using a tKafkaInput to extract the words we inputted in a kafka producer and load them using a tLogRow into the console, but we still have the same issue.
We dont have any significant experience in using Spark Streaming and are not sure if the errors that occurred are because of wrong spark configurations or lacking.
Any help would be gladly appreciated
Locke
We have been tasked to demonstrate streaming using Talend. We have seen that Talend Data Fabric 6.3.1 has Big Data Streaming where you can create streaming jobs similar to integration jobs. We have chosen to create a Spark streaming job. We tried to follow the simple spark streaming job from the given talend documentation where we will generate data from tRowGenerator and load it into a given HDFS folder in AVRO format. Below is the job and the corresponding spark configuration we used:
When the job was run we have received this error:
We also tried to run it using local mode in spark but it still produced the same error. We are not sure where we went wrong with this one. We also tried to use Kafka into the job as shown below, by using a tKafkaInput to extract the words we inputted in a kafka producer and load them using a tLogRow into the console, but we still have the same issue.
We dont have any significant experience in using Spark Streaming and are not sure if the errors that occurred are because of wrong spark configurations or lacking.
Any help would be gladly appreciated
Locke
Hi Talend Team,
We are still having this issue on our end. It seems that even if we are using the Local Mode of the Spark configuration, our job is still not able to load into HDFS. Do you have a sample job that executes properly using Spark Streaming? We are using Talend Data Fabric 6.3.1
Any help would be gladly appreciated,
Regards,
Locke
only You can guess - where You have this settings
as I understand - You have space there
Hi Talend Team,
We are still having this issue on our end. It seems that even if we are using the Local Mode of the Spark configuration, our job is still not able to load into HDFS. Do you have a sample job that executes properly using Spark Streaming? We are using Talend Data Fabric 6.3.1
Any help would be gladly appreciated,
Regards,
Locke
only You can guess - where You have this settings
https://www.talendforge.org/forum/img/members/403111/mini_Screen_Shot_2017-03-15_at_11.02.13_PM.png....
as I understand - You have space there
Thanks Vapukov, for that. I've now changed the folder name to where my Talend Data Fabric is located by removing the spaces. After encountering issues in opening the studio, I was able to run the job again, and I'm still encountering this nullpointer exception below:
I am not sure why the tRowGenerator cant generate data(Original Job Below)
Any help would be gladly appreciated.
Regards,
Locke
