Hi
I need to analyze the data from the Industrial Control System (PLC, SCADA)
These systems have OPC UA interface.
Amount of data in the
Industrial Control System - Big Data
Hi Victor,
How you are planning to move data from your hardware systems to file system? If you can get data into the file system, then Talend could help to process them as required. Just few questions out of curiosity..
What is the Volume and Velocity of your Big Data?
Do you want to process the data in real time?
What is the frequency by which you will be offloading data from hardware to file system?
What you want to do with that data?
Thanks
Vaibhav
There are 16 computers, each generates about 50 megabyte per day. Offloading data plan in the database. Tell me, what a database is best suited for BigData? We want to process the data in real time. Based on the data necessary build graphs
Victor, Putting the OPC UA data into a database is not a good idea. Instead, write an OPC UA client that converts UA calls to XML messages that you can access from Talend applications. I am very interested in this work and can help you if you want. John Gillerman
Hi,
There are 16 computers, each generates about 50 megabyte per day.
>> Only 50MB per day is not a Big Data... Data could be much bigger... GB/TB/PB of data with very high velocity (per minute/hour/day) could come in the perview of Big Data
Offloading data plan in the database. Tell me, what a database is best suited for BigData?
>> depends upon what and how you want to play with the data
We want to process the data in real time.
>> Real time process of data is not in the scope of ETL, it is in the scope of Dashboard development. There are many vendors who work on real time data display. Concept is dashboard queries database in realtime every second and changes data which is new. In order to support real time data display, what is the frequency of data offloading to database from your systems. This can't be real time as you are off loading with particular frequency.
Based on the data necessary build graphs
This is again in the scope of Report and Dashboard development. Few graph components are available which can draw graphs for you, but that is limited to line and bar charts etc.
I hope you got an idea. Let me know if you have any further questions...
Victor,
I believe that bring UA data on to a Talend integration bus is a very valid use case. Importantly, Talend supports the notion of information models which is also the case of UA of course. IMHO, it makes sense to write a UA client that is used to set up UA subscriptions to the backend UA server. Once the UA client starts receiving UA DataChange events, it just publishes them on to the Talend Bus. Talend Applications can then subscribe to Talend messages on the Talend Bus. The limitation of this of course is that the monitored item group is set up in the UA client and shared by all Talend message consumers. You could set up multiple subscriptions in the client and publish them using different message ids on the Talend bus. But this requires coordination of configuration between the Talend applications and the UA client. I recommend that you start with this. Over the longer term, you can also get CreateMonitoredItems and Subscribe messages on to the Talend bus, I would not start out doing this.
John
With regard to using a database and rest, this adds a very large layer between UA and Talend message consumers. The database should either be a UA client (preferred) or a Talend message subscriber. Feel free to contact me directly at
john@gridcloudsys.com.
