tSOAP - access the response header

[Anonymous]

Hello,

i have been searching around now for quite a time, I want to communicate with a SOAP Webservice where the process is the following:

1. connect with the Webservice via Login Credentials sent in the body part
2. receive a Cookie from the header of the response to the login message
3. use the received cookie for further requests on the webservice

This is a quite common scenario.

Component options which i have tested:

- tSOAP component - no chance to get the HTTP header information out of that

- tWebservice component - same.

- tHttpRequest - no chance to get the HTTP header out of the response, only the body part - WTF is this component else for instead of raw HTTP requests?!?

- tESBConsumer component with hacks to get to the HTTP header - does somehow not work for me (see https://community.talend.com/t5/Design-and-Development/resolved-SOAP-header-in-tJavaRow/m-p/78503#M3...)

Can someone suggest a solution? Reading out a cookie from a HTTP response Header is not that uncommon and i think Talend should support this kind of stuff out of the box, tell me when i'm not right. My alternative currently is to just use Java Code for building a component which should already exist?!?

[Anonymous]

The code looks fine. Can you try moving the tJava to connect to the tESBConsumer_3 using the OnComponentOK link. This is just a test. Unfortunately I cannot try anything out as I don't have your service available to me.

Regarding Talend not being able to build the schema from the WSDL, that is odd but can be caused by WSDLs that refer to XSDs. 

[Anonymous]

I tried it with OnComponentOk but i'm still getting a NullpointerException

[Anonymous]

This is down to a bug in the documentation. I should have known as I have actually hit this before. The globalMap variable is actually tESBConsumer_3_HTTP_HEADERS. Replace the code with the code below....

System.out.println("Get Cookie from Header");
try {
	java.util.Map<String,java.util.List<String>> headers =((java.util.Map<String,java.util.List<String>>)globalMap.get("tESBConsumer_3_HTTP_HEADERS"));

java.util.Iterator<String> it = headers.keySet().iterator();
	while(it.hasNext()){
		System.out.println(it.next());
		// todo: cookie processing
		// check if line has the cookie in it
		// if yes, write the cookie to a context variable
	} 
} catch (Exception nullPointerException) {
	// TODO: handle exception
	System.out.println("nullPointerException");
}

[Anonymous]

Hello rhall_2_0,

i just found out the same; the autocomplete gives me _HEADERS while the documentation says _HTTP_HEADERS.

I opened an issue in the bug tracker for that, https://jira.talendforge.org/browse/TESB-23531

thank you very very much for your help so far,

best regards,

Finn

[Anonymous]

We were unable to get help from Talend support about how to use SFAPI to extract data from SuccessFactors, so we came up with a proof of concept how to do that.

Although we are still perfecting our approach, here is what we did as a proof-of-concept.

We did not use tSOAP to extract data from SFAPI as it was not at all obvious how to get that component to create a cookie via login(), and use that cookie in subsequent components within the same job.

The POC job looks like this:

The POC uses the SFAPI login method to generate a cookie file in tSystem_1, uses that cookie file in tSystem_2 to execute the SFAPI query method, saving the result in an XML file. It then processes that file in tFileInputXML_1 using generated metadata (discussed further below), finally logging the results of the query in tLogRow_1.

Before attempting to run this job, you must generate File xml metadata discussed further below.

Below is a description of the components of the above job.

tSystem_1:

• This component uses curl to login to SFAPI and generates a cookie file
• Use common home directory across all components
• The command is "bash login.sh" (you can use sh rather than bash if you prefer)
• login.sh has this: curl -c cookie.txt -H "Content-type: text/xml" -d @login.xml -X POST "https://api15.sapsf.cn/sfapi/v1/soap"
• File login.xml contains a  payload for the SFAPI login method.

tSystem_1:

• This component uses curl to query SFAPI, using the cookie file generated by the prior component
• Use the same common home directory as above
• The command is "bash query.sh" (you can use sh rather than bash if you prefer)
• query.sh has this: curl -s -b cookie.txt -o queryOutput.xml -H "Content-type: text/xml" -d @query.xml -X POST "https://api15.sapsf.cn/sfapi/v1/soap"
  
• File query.xml contains a payload for the SFAPI query method; this POC uses queryString: select user_id, local_job_title from EmpJob
• See GENERATING QUERY RESULTS METADATA below for discussion of schema

tFileInputXML_1:

• The File name/Stream  value is the full path to the queryOutput.xml file generated above (we will change this to use a Stream)
• The Loop XPath query value is based on the metadata
• See GENERATING QUERY RESULTS METADATA below for discussion of schema

tLogRow_1:

• The components logs the rows from the prior component
• We use Mode "Table (print values in cells of a table)"

RUNNING CURL COMMANDS UNDER WINDOWS

If you run Talend Studio under Linux, you may skip this section.

In order to run the curl command under windows, we install either Git Bash or Cygwin on the machine where Talend Studio runs. To make the bash/sh and curl commands available to components in the above job, you can put the directory containing those commands at the end of your Windows PATH environment variable.

If you are using Git Bash, that directory may be C:\Program Files\Git\usr\bin. If you are using Cygwin, that directory may be %CYGWIN_HOME%\bin, where %CYGWIN_HOME% is your Cygwin installation directory.

IMPORTANT:

If you are running under Windows, you may need to use the dos2unix command to convert your script files (login.sh and query.sh) to linux format before using those scripts.

GENERATING QUERY RESULTS METADATA

To make this job work properly, you must generate File xml metadata based on the results of the query.sh script, and use that metadata in the tSystem_2 and tFileInputXML_1 components. Generating this metadata is actually very simple:

1. Open a Git Bash or Cygwin shell
2. Change directory to the common home directory for the components in the job
3. Execute this: sh login.sh
4. Execute this: sh query.sh
5. The above generates file queryOutput.xml, which you will use to create the File xml metadata
6. Right click on Metadata --> File xml and select Create file xml
7. Give the metadata a name and click Next >
8. Select Input XML and click Next >
9. In File Settings use the queryOutput.xml generated above for the XML file and click Next >
10. Drag sfobject to the Xpath loop expression and change Loop limit to an appropriate value, if desired
11. Drag user_id and local_job_title to Fields to extract and click Next >
12. Click Refresh Preview to show a preview, if desired
13. Click Finish
14. In tSystem_2, specify Schema: Repository, and select the above File xml metadata created above
15. In tFileInputXML_1, specify Property Type: Repository, and select the File xml schema created above
16. In tFileInputXML_1, specify Schema: Repository, and select the File xml metadata create above

I hope this helps; I will try to answer any questions you may post.

We came up with this approach as a Proof of Concept for extracting data from SFAPI after weeks of trying other methods and after pleading with Talend support unsuccessfully for help.

I thank @Dijke for his hint to use curl; without that hint, we would have been hopelessly lost.

Merry Christmas 2018!

Sincerely,

BGI Talend Team

[Anonymous]

I apologize for my typo. My article was about extracting data from SuccessFactors, NOT Salesforce!