Skip to main content
Announcements
Qlik Introduces a New Era of Visualization! READ ALL ABOUT IT
cancel
Showing results for 
Search instead for 
Did you mean: 
MDAtl
Contributor
Contributor

Call an API from Replicate

Hello:

We have a need to transform as we replicate using a 3rd party transformer that provides an API. Is there a way to invoke an API for each row as it is being replicated?

Thanks,

Manas.

Labels (4)
2 Solutions

Accepted Solutions
john_wang
Support
Support

Hello @MDAtl ,

Welcome to Qlik Community forum and thanks for reaching out here!

Qlik Replicate supports rich built-in transformations, and we also support User-defined transformations to extend calling tailor-made user modules, the UDTF should be written in C.

After the UDTF is compiled and setup done, the transformation can be seen in "User Defined" functions list, it take effect for each row in the replication task.

Hope this helps.

John.

 

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!

View solution in original post

Heinvandenheuvel
Specialist III
Specialist III

@john_wang >> After the UDTF is compiled and setup done, the transformation can be seen in "User Defined" functions list, it take effect for each row in the replication task.

Nit-picking:  such UDT is called once for each column it is associated with for each row being processed.

The argument passed to the UDT is a pointer to the input column text. The return value is a pointer to the a string buffer with transformed text to return. There are pointers to other data types available, but last time I looked specifically the CLOB data type was NOT supported.

The Replicate installation comes with an an example. For Windows you can find that under C:\Program Files\Attunity\Replicate\addons . Brave souls should be able to figure out how to set this up (I did :-). Mere mortals may want to engage Qlik Professional Services to make sure to create a robust solution and to be aware of the latest  restrictions and features.

Hein.

 

View solution in original post

8 Replies
john_wang
Support
Support

Hello @MDAtl ,

Welcome to Qlik Community forum and thanks for reaching out here!

Qlik Replicate supports rich built-in transformations, and we also support User-defined transformations to extend calling tailor-made user modules, the UDTF should be written in C.

After the UDTF is compiled and setup done, the transformation can be seen in "User Defined" functions list, it take effect for each row in the replication task.

Hope this helps.

John.

 

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
Heinvandenheuvel
Specialist III
Specialist III

@john_wang >> After the UDTF is compiled and setup done, the transformation can be seen in "User Defined" functions list, it take effect for each row in the replication task.

Nit-picking:  such UDT is called once for each column it is associated with for each row being processed.

The argument passed to the UDT is a pointer to the input column text. The return value is a pointer to the a string buffer with transformed text to return. There are pointers to other data types available, but last time I looked specifically the CLOB data type was NOT supported.

The Replicate installation comes with an an example. For Windows you can find that under C:\Program Files\Attunity\Replicate\addons . Brave souls should be able to figure out how to set this up (I did :-). Mere mortals may want to engage Qlik Professional Services to make sure to create a robust solution and to be aware of the latest  restrictions and features.

Hein.

 

MDAtl
Contributor
Contributor
Author

Thank you @john_wang  and @Heinvandenheuvel . I now have enough info to get started.

john_wang
Support
Support

Thank you so much for your great support @MDAtl !

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
noobDE
Contributor II
Contributor II

Hi, mere mortal here 🙂

Could you maybe give some high-level pointers for this?

I have the same scenario where I use curl to call the API. I have been trying but have still had no luck.

For example, when creating the .dll file, do we need to link the curl library statically or dynamically?

john_wang
Support
Support

Hello @noobDE ,

You don't need to link the curl library in the C project since the UDTF operates within the Replicate framework. The 'curl' program and Replicate APIs function as independent modules.

Hope this helps.

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
noobDE
Contributor II
Contributor II

Hi @john_wang , thank you so much for responding.

 

Just to clarify, the API I'd like to call is an external endpoint (let's say we have our company's internal API and we want to send POST request to it from Replicate), and here we've created the program using curl.h on our main function.

So this was the part that I would appreciate some pointers. We've managed to compile the sample code MyTransformation but still no luck when it comes to the calling API part.

If you can give list of factors/components as the checklist that we need to make sure that the final .dll can be loaded in Replicate.

Again, thank you for your help.

Heinvandenheuvel
Specialist III
Specialist III

First off, I think you need to create your own topic, with good, clear subject line.

Second, I still don't understand the main process flow. You wrote "I have the same scenario where I use curl to call the API."

The basic Replicate flow is 

  1. for each source DML event
  2. check transformation
  3. if UDT(s) present call UDT(s) with param
  4. update target column with transformed value
  5. apply to target.

I suspect the in step 3 the you want the UDT code that is belng called by Replicate to activate a curl command (which is a program!) to poke another server and return a value to the udt. Yikes. That's a whole lot of overhead in a restricted context calling environment. Hopefully you mean that you calling LIBCURL that's one layer easier.

Free advice

  • Carefully study the Replicate User Guide - K Replicate add-ons API
  • Carefully study the Replicate User Guide - K Replicate add-ons API 5 more times
  • build a test UDT which returns "Hello World"
  • build a test UDT which returns "Hello {first-passed-parameter} World {second passed parameter}"
  • Try call LIBCURL - login successful? 
  • Try call LIBCURL - login, post, result. Works?
  • Think about how to retain login context (memory pool?) - which might NOT be possible

If this fails - create fresh topic clearly stating where it all failed, maybe as soon as while linking, what error messages.

Personally I suspect this is one  level too complex for a UDT, but I'm aware of UDT calling out for encryption methods in libraries and other clever stuff. You may need Qlik Profession Services to help.

Hein

 

  •  
  •