How to extract data from ethereum using Talend

[YLi1644627156]

Very new to Talend. My very goal is to use Talend to do the following:

```

from web3 import Web3

w3 = Web3(Web3.HTTPProvider('https://mainnet.infura.io/v3/XXXXXXXXXXXXXXXXXXX'))

w3.eth.get_block('latest').items()

```

Wonder which components I need and how to achieve it.

Open to java implementation as well, if it is easier with tJava. 

Thanks

[Anonymous]

Could you give us a bit more information about this please? It looks like you will just need to use a tRestClient component for this, but information about the API would be useful.

[YLi1644627156]

So in this python example, a function call need to be made to extract the data. Will RestClient allow us to make a direct call like this?

```

from web3 import Web3

w3 = Web3(Web3.HTTPProvider('https://mainnet.infura.io/v3/XXXXXXXXXXXXXXXXXXX'))

w3.eth.get_block('latest')

```

Alternatively, I have been testing out tSystem that calls Python. I can get the output "to console", but if I use "normal" and connect to "tLogRow", I am not getting any outputs. Am I using tSystem correctly?

[Anonymous]

This is why I asked for information on the API. I presume that it will require some sort of authentication and presume it will return JSON....although it could XML. The tRestClient can handle that, but without the API specification document, it is very hard to suggest how to use it. You certainly don't need to use a tSystem to call Python.

I have tried to call the URL with the XXXXs and by substituting them with an Ethereum wallet address, but that returned nothing in a browser. So I presume that the XXXXs represent something other than an Ethereum wallet address.

[YLi1644627156]

Thanks for testing. We actually use an internal API, which does not require authentication.

The XXXX in the snippet are not wallet, but rather keys from https://infura.io/. You may sign up and get a free one if you wish to.

The returns are in JSON.

I only see tRest, not tRestClient on our side, but we should have ESB license. So please describe how to find tRestClient.

Under tRestClient, where should I input something like "w3.eth.get_block('latest')"? If you can post some pictures, it would be great.

Much appreciated!

[Anonymous · Accepted solution]

OK, I have created an account and tested it. This is how it works. First, to get the tRestClient component, just type...tRestClient ....on the design window. It will appear.

Then create a job looking like this.....

 

The tJavaFlex is coded like this.....

The code that is important is.....

row1.string = "{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}";

The rest is just commented out code. Note that the JSON here is escaped for Java. See the backslashes before quotes. This JSON is the body and will hold the parameters for your query.

The tRestClient is configured like this.....

 

Note the HTTP Method and the Content Type. I have blocked out my code in the URL.

The response to this is shown by the tLogRow which needs no configuration. It looked like this for me....

{"jsonrpc":"2.0","id":1,"result":"0xdd1895"}

I presume that the "latest" referenced in your code is actually a JSON element that is returned when you have data. You should be able to collect this using a tExtractJSONFields component. You will need to use JSONPath for this.

[YLi1644627156]

This is great. Thanks rhall!

Somehow I don't have access to tRestClient, although I have ESB. But tRest seems to work as well.

One extra question, how should I parse out rawContract.value in tExtractJSONFields? I got the "$.result.transfers[*]" part handled, but under mapping, it is no longer a Jsonpath query, but a json query, and "$.rawContract.value" does not work.

{"id":1,"result":{"transfers":

[{"blockNum":"0xa97ae2","hash":"0x56a00d5fac0dcacfe1cd950c49506712ec5bf319e246701321e8a92643da349d","from":"0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be","to":"0xfd5d6417fdcf02f0627aafa9e594a67d4485642d","value":5.73621309,"erc721TokenId":null,"erc1155Metadata":null,"tokenId":null,"asset":"ETH","category":"external","rawContract":{"value":"0x4f9b1f69e1655400","address":null,"decimal":"0x12"}}]}

[Anonymous]

In this scenario you need to use the Loop JsonPath Query and your other paths have to be in relation to that. Here is a screenshot of how I have set this up to work.....

 

The output from this can be seen below. Notice that I retrieved elements from different levels in the JSON....

.------------------------------------------------------------------------------------------.

|                   #1. tLogRow_2                    |

+---------------------+--------------------------------------------------------------------+

| key         | value                               |

+---------------------+--------------------------------------------------------------------+

| blockNum      | 0xa97ae2                              |

| hash        | 0x56a00d5fac0dcacfe1cd950c49506712ec5bf319e246701321e8a92643da349d |

| rawContract_value  | 0x4f9b1f69e1655400                         |

| rawContract_decimal | 0x12                                |

+---------------------+--------------------------------------------------------------------+

[YLi1644627156]

This great. One last thing. Our team members have been comparing Talend again Alteryx.

It looks like Alteryx can automatically parse json and filled in all the column names. Is this functionality available in any of Talend components?

[Anonymous]

In the Enterprise Edition there is the Talend Data Mapper. It will parse the JSON (and other flat file structures like XML, etc) and produce a schema you can work with. I believe you are using the Open Studio version which is limited somewhat.