topic Re: Logging Integration with Sentry,io in Talend Studio

Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 09:56:06 GMT

Can anyone help with a challenge I have. We are trying to integrate and stream errors captured by tLogCatcher to stream via tRESTClient to our account in sentry.io.

The job is running and it seems to be packaging the JSON on but nothing ever arrives in sentry and doesn't fail. Do we need to install something within the job to support sentry or are we missing a trick somewhere. Surely - we're not the first to have done this...

I'm added images to this post to help understand the setup.

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 12:43:37 GMT

Add a tLogRow to the Response and Error outputs of the tRestClient. I do not know sentry.io, but looking at what you have done my guess is that your JSON is not in a suitable schema.

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 21:43:21 GMT

@rhall wrote:

Add a tLogRow to the Response and Error outputs of the tRestClient. I do not know sentry.io, but looking at what you have done my guess is that your JSON is not in a suitable schema.

I did that and then the JSON was checked and I'm able to send it directly via a PHP script outside of Talend up to the sentry, so we know the string is fine.

Sentry has integration with Java but I can't work out how to set it up - https://docs.sentry.io/clients/java/?platform=javascript.

Tried to copy the logj41.x XML into the job logj4 configuration but still no joy. I'm sure we're missing something somewhere and configuring this way is unchartered territory right now so any points would be great.

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 21:45:51 GMT

oh - and I've also downloaded and copied into C:/program/ the latest Apache Maven 3.5.4 http://maven.apache.org/download.cgi and added this to the Maven configuration within Talend

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 22:17:29 GMT

If you can send your data via REST Service, you do not need to do anything special in Talend. However, REST services (any services to be honest) can be a bit tricky to configure. If you got it to work with a PHP script, would you mind sending the script that worked and also your tRESTClient configuration. I should be able to see what is wrong by comparing the two.

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 22:27:48 GMT

Also, did you get ANY response from the Response tLogRow or the Error tLogRow? What input column were you passing the data to (your JSON)?

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 23:14:13 GMT

@rhall wrote:

Also, did you get ANY response from the Response tLogRow or the Error tLogRow? What input column were you passing the data to (your JSON)?

Yes, sorry I tested with HTTP and HTTPS...

HTTP gives me the error:

308|<html>

<head><title>308 Permanent Redirect</title></head>

<center><h1>308 Permanent Redirect</h1></center>

<hr><center>nginx/1.13.12</center>

</body>

</html>

HTTPS gives me:

301|HTTP 301 Moved Permanently

Screen Shot 2018-09-21 at 23.12.31.png

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 23:15:19 GMT

screen shot in the last post shows setup for tmap2 which is the output from the BuildJSONString

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 23:39:01 GMT

@rhall wrote:

If you can send your data via REST Service, you do not need to do anything special in Talend. However, REST services (any services to be honest) can be a bit tricky to configure. If you got it to work with a PHP script, would you mind sending the script that worked and also your tRESTClient configuration. I should be able to see what is wrong by comparing the two.

This is the send_message php that is called and gets the message through...

class sendSQSMessage

{

/**

* sendSQSMessage constructor.

* @rchinta string $commandName

* @rchinta string $output

public function __construct(string $commandName, string $output)

{

$client = new Raven_Client('https://XXXX@sentry.XXXXX.net/11');

$client->setEnvironment('testing'); //production or development or testing

$client->setRelease('0.01');

$error_handler = new Raven_ErrorHandler($client);

$error_handler->registerExceptionHandler();

$error_handler->registerErrorHandler();

$error_handler->registerShutdownFunction();

$sqs_client = new \Aws\Sqs\SqsClient(unserialize(AWS_CREDENTIALS));

$result = $sqs_client->getQueueUrl(['QueueName' => QUEUE_NAME]);

$queue_url = $result->get('QueueUrl');

@/** @var $uuid Uuid */

$uuid = Uuid::uuid4();

/**

* The message we will be sending

$our_message = [

'id' => $uuid,

'dateFinished' => new DateTime(),

'commandName' => $commandName,

'workerName' => gethostname(),

'output' => $output

];

/**

* Send the message

$sqs_client->sendMessage([

'QueueUrl' => $queue_url,

'MessageBody' => json_encode($our_message)

]);

}

Attached is the config of the tRESTClient...

Screen Shot 2018-09-21 at 23.37.45.png
Screen Shot 2018-09-21 at 23.37.34.png

Re: Logging Integration with Sentry,io

Anonymous — Fri, 21 Sep 2018 23:52:13 GMT

OK, this is a difficult one to solve while not having access to the service. But I can help. It looks like your service endpoint is redirecting. This redirect endpoint will be returned as a Header in the response message. The header should be called "Location". Below is some code which you can use to find this redirect URL.

java.util.Iterator<String> it =   ((java.util.Map<String,java.util.List<String>>)globalMap.get("tRESTClient_1_HEADERS")).keySet().iterator();

while(it.hasNext()){
	System.out.println(it.next());
}

I've assumed your tRestClient is called tRESTClient_1. This will iterate through the response Header Keys. Run it and look for something like "Location". Then, amend the code to this...

java.util.Iterator<String> it =   ((java.util.Map<String,java.util.List<String>>)globalMap.get("tRESTClient_1_HEADERS")).keySet().iterator();

//while(it.hasNext()){
//	System.out.println(it.next());
//}


System.out.println(((java.util.Map<String,java.util.List<String>>)globalMap.get("tRESTClient_1_HEADERS")).get("Location"));

I've used "Location" as the Key above, change it if required.....or you could just try running the second code straight away. The URL returned is the redirect location. Set that in your tRESTClient URL.

That should work for you

Re: Logging Integration with Sentry,io

Anonymous — Sat, 22 Sep 2018 00:37:22 GMT

I'm an idiot (but it is late here and that code may be useful to you in the future), there is a MUCH easier way of solving this. Just tick the "Follow redirects" option on the Advanced Settings of the tRESTClient.

Re: Logging Integration with Sentry,io

Anonymous — Sat, 22 Sep 2018 10:32:26 GMT

Thanks for the pointers. Seems like we're getting somewhere now. Attached is a console output after putting your code in there. It seems from one of the messages in the console that we are missing a 'Referer header' in the sentry call.

I'll have our developer who set up sentry check this out but thought I'd share this in case you can spot anything else.

Thanks for the help - really appreciated.

error.txt.zip

Re: Logging Integration with Sentry,io

Anonymous — Sat, 22 Sep 2018 12:55:07 GMT

Think we now have it talking to sentry. Suspect there are now some authentication issues within sentry that need sorting so we're investigating. I'll keep you posted as it will be good to wrap this one up so we have a sentry integration solution on the community....

If reviewing the text file you see something else then do let me know...

Re: Logging Integration with Sentry,io

Anonymous — Sat, 22 Sep 2018 13:50:18 GMT

It looks like your error was caused by the file your tFileOutputDelimited_1 component is writing to already existing. You will need to enable it to append or to overwrite.

if authentication is required, look at the "Use Authentication" tick box on the tRESTClient.

Re: Logging Integration with Sentry,io

Anonymous — Sat, 22 Sep 2018 13:59:47 GMT

yeah - the delimited file error is forced to fire the error message to check it goes to sentry.

We are looking at the authentication. I tried using the authentication method but that still failed - its an issue at the sentry end it seems so my team are looking into that now...

Will come back when we have a solution...

Thanks

dave

Re: Logging Integration with Sentry,io

Anonymous — Mon, 24 Sep 2018 14:21:17 GMT

Hey @rhall

ok - we've made a bit more progress. Think we're now close. I have downloaded and installed the sentry jar and had this loaded up with a tLibraryLoad. I then have a java class I need to install so I can then call that from the job but not sure how to load it into Talend. See images attached for what I mean...

Once this is installed then we are led to believe we should be off and running.

Any ideas?

Screen Shot 2018-09-24 at 14.18.01.png
Screen Shot 2018-09-24 at 14.17.50.png
Screen Shot 2018-09-24 at 14.17.09.png

Re: Logging Integration with Sentry,io

Anonymous — Mon, 24 Sep 2018 14:52:45 GMT

I'm confused. If you were able to call the REST service from a PHP script, then I don't quite understand the need for the Sentry Jar to be added to the job. A web service can be called from ANY source so long as the message and call are formatted properly. It sounded like it was just a security issue from the last message.

However, using Java class to make the call will also work with Talend. First of all you need to create a Routine. A Routine is essentially just a Java class. You should be able to take the code you have shown me and add it to a Routine. Having said that, the code I have seen looks very basic and may need some tweaking to make it suit your requirements. For example, it has a "static void main" method which implies it is intended to be run from the command-line. If you have any Java guys there, ask them to tweak it so that you can call a "send" method as a static method. The send method will need to have at least a String parameter to hold your JSON.

Once you have your routine, you will need to add your Jar to its list of libraries. To do this, right click on the routine and select "Edit Routine Libraries". Add it here. This will mean you do not need the tLibraryLoad in your job.

The you can use your "send" method anywhere in your job. If you routine is called "MyRoutine" and your send method is called "send", you can call it using code like this.....

routines.MyRoutine.send(row1.errorJSON)

Re: Logging Integration with Sentry,io

Anonymous — Tue, 25 Sep 2018 20:48:18 GMT

Thanks, you've been really helpful!

Our Java fella's have now sorted it. None of us can understand why the API wasn't working when we could run via PHP but that became too much to sort when we could simply install the libraries and change the Java to send to sentry. Must be something funny with Sentry...

After loading the 4 JARs via the tLibraryLoad component:

jackson-core-2.9.7.jar
sentry-1.7.5.jar
javax.servlet-api-3.0.1.jar
slf4j-api-1.8.0-beta2.jar

...and creating a class to call in a Code repository we were then able to call from the tJavaRow and we started to receive the messages in Sentry.

So, all good now and thanks again for your help...

Re: Logging Integration with Sentry,io

Anonymous — Tue, 25 Sep 2018 20:54:18 GMT

Glad you got it sorted. These struggles with Talend tend to benefit you in the long run as you usually end up learning something new about the tool while experimenting 🙂

It sounds like you have added your code to a Routine, which is great. If this is the case, try adding your libraries to the Routine (as I described in my last post) as this will remove the requirement for the tLibraryLoad components in your job. It will also mean that your code will work in ANY job without having to add the tLibraryLoad components.

Re: Logging Integration with Sentry,io

Anonymous — Tue, 25 Sep 2018 21:34:18 GMT

Oh great - love that idea. Thanks - will make sure I give that a go. Missed that in the last post.

Thanks

Dave