Solved: Logging Integration with Sentry,io - Qlik Community

Anonymous · ‎2018-09-21

Hi

Can anyone help with a challenge I have. We are trying to integrate and stream errors captured by tLogCatcher to stream via tRESTClient to our account in sentry.io.

The job is running and it seems to be packaging the JSON on but nothing ever arrives in sentry and doesn't fail. Do we need to install something within the job to support sentry or are we missing a trick somewhere. Surely - we're not the first to have done this...

I'm added images to this post to help understand the setup.

Anonymous · ‎2018-09-24

I'm confused. If you were able to call the REST service from a PHP script, then I don't quite understand the need for the Sentry Jar to be added to the job. A web service can be called from ANY source so long as the message and call are formatted properly. It sounded like it was just a security issue from the last message.

However, using Java class to make the call will also work with Talend. First of all you need to create a Routine. A Routine is essentially just a Java class. You should be able to take the code you have shown me and add it to a Routine. Having said that, the code I have seen looks very basic and may need some tweaking to make it suit your requirements. For example, it has a "static void main" method which implies it is intended to be run from the command-line. If you have any Java guys there, ask them to tweak it so that you can call a "send" method as a static method. The send method will need to have at least a String parameter to hold your JSON.

Once you have your routine, you will need to add your Jar to its list of libraries. To do this, right click on the routine and select "Edit Routine Libraries". Add it here. This will mean you do not need the tLibraryLoad in your job.

The you can use your "send" method anywhere in your job. If you routine is called "MyRoutine" and your send method is called "send", you can call it using code like this.....

routines.MyRoutine.send(row1.errorJSON)

View solution in original post

Anonymous · ‎2018-09-21

Add a tLogRow to the Response and Error outputs of the tRestClient. I do not know sentry.io, but looking at what you have done my guess is that your JSON is not in a suitable schema.

Anonymous · ‎2018-09-21

@rhall wrote:

Add a tLogRow to the Response and Error outputs of the tRestClient. I do not know sentry.io, but looking at what you have done my guess is that your JSON is not in a suitable schema.

I did that and then the JSON was checked and I'm able to send it directly via a PHP script outside of Talend up to the sentry, so we know the string is fine.

Sentry has integration with Java but I can't work out how to set it up - https://docs.sentry.io/clients/java/?platform=javascript.

Tried to copy the logj41.x XML into the job logj4 configuration but still no joy. I'm sure we're missing something somewhere and configuring this way is unchartered territory right now so any points would be great.

Anonymous · ‎2018-09-21

oh - and I've also downloaded and copied into C:/program/ the latest Apache Maven 3.5.4 http://maven.apache.org/download.cgi and added this to the Maven configuration within Talend

Anonymous · ‎2018-09-21

If you can send your data via REST Service, you do not need to do anything special in Talend. However, REST services (any services to be honest) can be a bit tricky to configure. If you got it to work with a PHP script, would you mind sending the script that worked and also your tRESTClient configuration. I should be able to see what is wrong by comparing the two.

Anonymous · ‎2018-09-21

Also, did you get ANY response from the Response tLogRow or the Error tLogRow? What input column were you passing the data to (your JSON)?

Anonymous · ‎2018-09-21

@rhall wrote:

Also, did you get ANY response from the Response tLogRow or the Error tLogRow? What input column were you passing the data to (your JSON)?

Yes, sorry I tested with HTTP and HTTPS...

HTTP gives me the error:

308|<html>

<head><title>308 Permanent Redirect</title></head>

<center><h1>308 Permanent Redirect</h1></center>

<hr><center>nginx/1.13.12</center>

</body>

</html>

HTTPS gives me:

301|HTTP 301 Moved Permanently

Screen Shot 2018-09-21 at 23.12.31.png

Anonymous · ‎2018-09-21

screen shot in the last post shows setup for tmap2 which is the output from the BuildJSONString

Anonymous · ‎2018-09-21

@rhall wrote:

If you can send your data via REST Service, you do not need to do anything special in Talend. However, REST services (any services to be honest) can be a bit tricky to configure. If you got it to work with a PHP script, would you mind sending the script that worked and also your tRESTClient configuration. I should be able to see what is wrong by comparing the two.

This is the send_message php that is called and gets the message through...

class sendSQSMessage

{

/**

* sendSQSMessage constructor.

*

* @rchinta string $commandName

* @rchinta string $output

*/

public function __construct(string $commandName, string $output)

{

$client = new Raven_Client('https://XXXX@sentry.XXXXX.net/11');

$client->setEnvironment('testing'); //production or development or testing

$client->setRelease('0.01');

$error_handler = new Raven_ErrorHandler($client);

$error_handler->registerExceptionHandler();

$error_handler->registerErrorHandler();

$error_handler->registerShutdownFunction();

$sqs_client = new \Aws\Sqs\SqsClient(unserialize(AWS_CREDENTIALS));

$result = $sqs_client->getQueueUrl(['QueueName' => QUEUE_NAME]);

$queue_url = $result->get('QueueUrl');

@/** @var $uuid Uuid */

$uuid = Uuid::uuid4();

/**

* The message we will be sending

*/

$our_message = [

'id' => $uuid,

'dateFinished' => new DateTime(),

'commandName' => $commandName,

'workerName' => gethostname(),

'output' => $output

];

/**

* Send the message

*/

$sqs_client->sendMessage([

'QueueUrl' => $queue_url,

'MessageBody' => json_encode($our_message)

]);

}

Attached is the config of the tRESTClient...

Screen Shot 2018-09-21 at 23.37.45.png
Screen Shot 2018-09-21 at 23.37.34.png

Anonymous · ‎2018-09-21

OK, this is a difficult one to solve while not having access to the service. But I can help. It looks like your service endpoint is redirecting. This redirect endpoint will be returned as a Header in the response message. The header should be called "Location". Below is some code which you can use to find this redirect URL.

java.util.Iterator<String> it =   ((java.util.Map<String,java.util.List<String>>)globalMap.get("tRESTClient_1_HEADERS")).keySet().iterator();

while(it.hasNext()){
	System.out.println(it.next());
}

I've assumed your tRestClient is called tRESTClient_1. This will iterate through the response Header Keys. Run it and look for something like "Location". Then, amend the code to this...

java.util.Iterator<String> it =   ((java.util.Map<String,java.util.List<String>>)globalMap.get("tRESTClient_1_HEADERS")).keySet().iterator();

//while(it.hasNext()){
//	System.out.println(it.next());
//}


System.out.println(((java.util.Map<String,java.util.List<String>>)globalMap.get("tRESTClient_1_HEADERS")).get("Location"));

I've used "Location" as the Key above, change it if required.....or you could just try running the second code straight away. The URL returned is the redirect location. Set that in your tRESTClient URL.

That should work for you

Logging Integration with Sentry,io

JSON

REST

Talend Data Integration

v7.x