topic Re: How to extract data from ethereum using Talend in Talend Studio

How to extract data from ethereum using Talend

YLi1644627156 — Fri, 15 Nov 2024 23:04:28 GMT

Very new to Talend. My very goal is to use Talend to do the following:

```

from web3 import Web3

w3 = Web3(Web3.HTTPProvider('https://mainnet.infura.io/v3/XXXXXXXXXXXXXXXXXXX'))

w3.eth.get_block('latest').items()

```

Wonder which components I need and how to achieve it.

Open to java implementation as well, if it is easier with tJava.

Thanks

Re: How to extract data from ethereum using Talend

Anonymous — Wed, 30 Mar 2022 12:07:55 GMT

Could you give us a bit more information about this please? It looks like you will just need to use a tRestClient component for this, but information about the API would be useful.

Re: How to extract data from ethereum using Talend

YLi1644627156 — Wed, 30 Mar 2022 14:34:05 GMT

So in this python example, a function call need to be made to extract the data. Will RestClient allow us to make a direct call like this?

```

from web3 import Web3

w3 = Web3(Web3.HTTPProvider('https://mainnet.infura.io/v3/XXXXXXXXXXXXXXXXXXX'))

w3.eth.get_block('latest')

```

Alternatively, I have been testing out tSystem that calls Python. I can get the output "to console", but if I use "normal" and connect to "tLogRow", I am not getting any outputs. Am I using tSystem correctly?

Re: How to extract data from ethereum using Talend

Anonymous — Wed, 30 Mar 2022 17:52:07 GMT

This is why I asked for information on the API. I presume that it will require some sort of authentication and presume it will return JSON....although it could XML. The tRestClient can handle that, but without the API specification document, it is very hard to suggest how to use it. You certainly don't need to use a tSystem to call Python.

I have tried to call the URL with the XXXXs and by substituting them with an Ethereum wallet address, but that returned nothing in a browser. So I presume that the XXXXs represent something other than an Ethereum wallet address.

Re: How to extract data from ethereum using Talend

YLi1644627156 — Wed, 30 Mar 2022 18:41:17 GMT

Thanks for testing. We actually use an internal API, which does not require authentication.

The XXXX in the snippet are not wallet, but rather keys from https://infura.io/. You may sign up and get a free one if you wish to.

The returns are in JSON.

I only see tRest, not tRestClient on our side, but we should have ESB license. So please describe how to find tRestClient.

Under tRestClient, where should I input something like "w3.eth.get_block('latest')"? If you can post some pictures, it would be great.

Much appreciated!

Re: How to extract data from ethereum using Talend

Anonymous — Wed, 30 Mar 2022 20:55:09 GMT

OK, I have created an account and tested it. This is how it works. First, to get the tRestClient component, just type...tRestClient ....on the design window. It will appear.

Then create a job looking like this.....

The tJavaFlex is coded like this.....

The code that is important is.....

row1.string = "{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}";

The rest is just commented out code. Note that the JSON here is escaped for Java. See the backslashes before quotes. This JSON is the body and will hold the parameters for your query.

The tRestClient is configured like this.....

Note the HTTP Method and the Content Type. I have blocked out my code in the URL.

The response to this is shown by the tLogRow which needs no configuration. It looked like this for me....

{"jsonrpc":"2.0","id":1,"result":"0xdd1895"}

I presume that the "latest" referenced in your code is actually a JSON element that is returned when you have data. You should be able to collect this using a tExtractJSONFields component. You will need to use JSONPath for this.

Re: How to extract data from ethereum using Talend

YLi1644627156 — Wed, 30 Mar 2022 23:14:52 GMT

This is great. Thanks rhall!

Somehow I don't have access to tRestClient, although I have ESB. But tRest seems to work as well.

One extra question, how should I parse out rawContract.value in tExtractJSONFields? I got the "$.result.transfers[*]" part handled, but under mapping, it is no longer a Jsonpath query, but a json query, and "$.rawContract.value" does not work.

{"id":1,"result":{"transfers":

[{"blockNum":"0xa97ae2","hash":"0x56a00d5fac0dcacfe1cd950c49506712ec5bf319e246701321e8a92643da349d","from":"0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be","to":"0xfd5d6417fdcf02f0627aafa9e594a67d4485642d","value":5.73621309,"erc721TokenId":null,"erc1155Metadata":null,"tokenId":null,"asset":"ETH","category":"external","rawContract":{"value":"0x4f9b1f69e1655400","address":null,"decimal":"0x12"}}]}

Re: How to extract data from ethereum using Talend

Anonymous — Thu, 31 Mar 2022 09:49:13 GMT

In this scenario you need to use the Loop JsonPath Query and your other paths have to be in relation to that. Here is a screenshot of how I have set this up to work.....

The output from this can be seen below. Notice that I retrieved elements from different levels in the JSON....

.------------------------------------------------------------------------------------------.

| #1. tLogRow_2 |

+---------------------+--------------------------------------------------------------------+

| key | value |

+---------------------+--------------------------------------------------------------------+

| blockNum | 0xa97ae2 |

| hash | 0x56a00d5fac0dcacfe1cd950c49506712ec5bf319e246701321e8a92643da349d |

| rawContract_value | 0x4f9b1f69e1655400 |

| rawContract_decimal | 0x12 |

+---------------------+--------------------------------------------------------------------+

Re: How to extract data from ethereum using Talend

YLi1644627156 — Thu, 31 Mar 2022 19:49:34 GMT

This great. One last thing. Our team members have been comparing Talend again Alteryx.

It looks like Alteryx can automatically parse json and filled in all the column names. Is this functionality available in any of Talend components?

Re: How to extract data from ethereum using Talend

Anonymous — Thu, 31 Mar 2022 21:26:02 GMT

In the Enterprise Edition there is the Talend Data Mapper. It will parse the JSON (and other flat file structures like XML, etc) and produce a schema you can work with. I believe you are using the Open Studio version which is limited somewhat.

Re: How to extract data from ethereum using Talend

YLi1644627156 — Thu, 31 Mar 2022 22:37:32 GMT

what is the data mapper component called? I have enterprise license.

Again I am not seeing tRestClient, how should I enable it?

Re: How to extract data from ethereum using Talend

YLi1644627156 — Thu, 31 Mar 2022 22:37:53 GMT

I am using Talend Studio

Re: How to extract data from ethereum using Talend

Anonymous — Thu, 31 Mar 2022 23:08:25 GMT

The tRestClient component is standard in all Talend Studio products. It doesn't need to be enabled. If you type t r e s t c l i e n t into the design window you should see it. If you don't see it, you should contact support.

The Data Mapper is more than a single component. The component is called tHMap. But there are several configuration stages for this component. You should look at the documentation for this.

https://help.talend.com/r/en-US/8.0/data-mapper-functions-reference-guide/introduction-data-mapper-functions