SAP SQL Connector RfcCommunicationException RETURN CODE: 28

[andy]

Hi,

After running since 2017 with no major problems we since 4 days are getting these kind of error messages.

It happens to different tables after different amount of time. One table that failed can work next time.

20230206T132647.691+0100 0179 SQL SELECT
20230206T132647.691+0100 0180 *
20230206T132647.691+0100 0181 FROM CE10700
20230206T132731.265+0100 167 fields found: .....
20230206T144948.256+0100 Error: RfcCommunicationException:no SAP ErrInfo available
20230206T144948.256+0100 RETURN CODE: 28
20230206T144948.275+0100 Execution Failed

There was a network change and the network guys could see a lot of packet losses so they fixed it, for us it just seems to mean that the same error takes longer to show up.

For what it is worth here is our connector string

CUSTOM CONNECT TO "provider=QvSAPSqlConnector.exe;servertype=0;ASHOST=172.xx.x.xx;CLIENT=070;SYSNR=30;Timeout=3600;SNC_MODE=false;SNC_QOP=9;advancedProperties=true;ConvRoutine=false;CheckSeparator=false;Nulldate=true;RemoveAllBlanks=true;ReplaceNullvalue=true;Trace=false;MixedMode=false;NulldateLegacy=false;FLTPLegacy=false;TIMSLegacy=false;DataTypeLegacyMode=false;PacketSize=20000;FetchBuffers=99999999;BufferPercentage=10;TimeOutBatch=600;TimeOutFetch=1200;TimeOutStartBatch=2400;BatchJobName=/QTQVC/READ_DATA;SocketBufferSize=32768;"

Below are some statistics from todays trials.

We are kind of desperate since the organization relies on SAP data in Qlik and it is getting rather old.

Running on QSoW Nov 2022 SR2 

Connector id:SQL version:7.4.0.0 InstallationPackageVersion:Qlik SAP Connector Package v8.0.2

Same error when fetching data from both SAP ERP and SAP BW

Any hints on where to look or break this down would be highly appreciated.

Attached is a log-file from C:\ProgramData\Qlik\Custom Data\QvSAPSqlConnector

[andy]

Found something interesting in one SAP SQL connector log

2023-02-06 19:10:40.935 [INF] [SAP] SapSqlConnection.OpenConnection() - Connection opened, ISOLanguage:EN Language:E
2023-02-06 19:10:46.148 [WRN] [SAP] InvokeFunction failed for '/QTQVC/FETCH_STREAM': RfcAbapException:TIMEOUT_READ_MEMORY
Timeout when trying to read shared buffer
SAP.Middleware.Connector.RfcAbapException: TIMEOUT_READ_MEMORY
at SAP.Middleware.Connector.RfcConnection.ThrowRfcErrorMsg()
at SAP.Middleware.Connector.RfcConnection.RfcReceive(RfcFunction function)
at SAP.Middleware.Connector.RfcFunction.RfcDeserialize(RfcConnection conn, IRfcIOStream stream)
at SAP.Middleware.Connector.RfcFunction.RfcCallReceive(RfcConnection conn, RFCID rid)
at SAP.Middleware.Connector.RfcFunction.Invoke(RfcDestination destination)
at SapRfcWrapper.SapNCoWrapper.InvokeFunction(String functionName, IDictionary`2 inputParameters, IEnumerable`1 activeParameters)
2023-02-06 19:10:46.148 [WRN] [SAP] OpenFetchStream.FetchStream() - /QTQVC/FETCH_STREAM threw an exception, RfcErrorKey:TIMEOUT_READ_MEMORY elapsed:00:00:05.1973933
2023-02-06 19:10:46.148 [WRN] [SAP] OpenFetchStream.FetchStream() - TIMEOUT_READ_MEMORY received, retrying
2023-02-06 19:11:31.051 [INF] [SAP] Invoked function '/QTQVC/FETCH_STREAM' with 2 input parameters took 44900ms

[andy]

This is the SQL connector -log for the error that actually breaks the load. The RfcCommunicationException exception

2023-02-06 14:48:16.413 [INF] [SAP] OpenFetchStream.FetchStream() - read 2806 data rows, total rows:597678 elapsed:01:21:23.7822772
2023-02-06 14:49:48.241 [WRN] [SAP] InvokeFunction failed for '/QTQVC/FETCH_STREAM': RfcCommunicationException:no SAP ErrInfo available
RETURN CODE: 28
SAP.Middleware.Connector.RfcCommunicationException: no SAP ErrInfo available
RETURN CODE: 28
at SAP.Middleware.Connector.CpicConnection.CpicReceive(Int32 timeout)
at SAP.Middleware.Connector.CpicConnection.Read(Byte* buffer, Int32 offset, Int32 count)
at SAP.Middleware.Connector.CpicConnection.Read(Byte[] buffer, Int32 offset, Int32 count)
at SAP.Middleware.Connector.CpicDriver.Read(Byte* buffer, Int32 count)
at SAP.Middleware.Connector.RfcConnection.ReadBytes(Byte* buffer, Int32 count)
at SAP.Middleware.Connector.RfcConnection.ReadRfcIDEnd(RFCID expectedRid, RfcParameter& param, RfcTable& table)
at SAP.Middleware.Connector.RfcConnection.ReadUpTo(RFCGET readState, RfcFunction function, RFCID toRid)
at SAP.Middleware.Connector.RfcConnection.RfcReceive(RfcFunction function)
at SAP.Middleware.Connector.RfcFunction.RfcDeserialize(RfcConnection conn, IRfcIOStream stream)
at SAP.Middleware.Connector.RfcFunction.RfcCallReceive(RfcConnection conn, RFCID rid)
at SAP.Middleware.Connector.RfcFunction.Invoke(RfcDestination destination)
at SapRfcWrapper.SapNCoWrapper.InvokeFunction(String functionName, IDictionary`2 inputParameters, IEnumerable`1 activeParameters)
2023-02-06 14:49:48.241 [WRN] [SAP] OpenFetchStream.FetchStream() - /QTQVC/FETCH_STREAM threw an exception, RfcErrorKey: elapsed:01:22:55.5837261
2023-02-06 14:49:48.241 [ERR] [SAP] OpenFetchStream.FetchStream() - Threw an exception, RfcErrorKey:
System.Exception: RfcCommunicationException:no SAP ErrInfo available
RETURN CODE: 28
at SapRfcWrapper.SapNCoWrapper.InvokeFunction(String functionName, IDictionary`2 inputParameters, IEnumerable`1 activeParameters)
at SapRfcWrapper.Helper.OpenFetchStream.FetchStream[T](ISapRfcWrapper wrapper, String functionName, String sapJobNumber, String sapResultTableName, BlockingCollection`1 sourceDataRows, Func`2 processRowFunc, Boolean isCancelJobSupported, CancellationToken cancellationToken, Int32 timeoutStartBatch, Int32 timeoutFetch, Nullable`1 timeoutCheckJob, Action`1 setFieldsFunc)
2023-02-06 14:49:48.241 [ERR] [] SQLConnectorDataReader.Read() - 

[Tomas_Sommergyll]

Hi Andy,

It seems that the SQL Connector is trying to access the Read Buffer in SAP but get errors in that instance. As this happens intermittently, it could be that the Read Buffer space in SAP is running low for some reason. 

In another thread:

https://community.qlik.com/t5/QlikView-Connectivity/Suddently-unstable-SAP-connector-General-advices...

they did a reboot of SAP to solve their issue.

It may be unrelated but could still be something to consider and try if possible.

Kind regards,

Tomas

[andy]

Hi Tomas, 

Thanks for you spending time on this. Last night we restarted SAP with doubled size of the buffer according to the URL below but the error still persist.

https://community.qlik.com/t5/QlikView-Connectivity/SAP-SQL-Connector-RfcCommunicationException-RETU...

[andy]

We found something strange. In the Qlik connector we pass TimeOutFetch= 1200 but in SAP when looking at the connection it is set to 5 seconds. Changing it to 600 doesn't matter it is stuck at 5 seconds no matter what we pass to it. But changing for example BufferPercentage from 10 to 15 can be seen in SAP. What's wrong with the TimeOutFetch parameter?

CUSTOM CONNECT TO "provider=QvSAPSqlConnector.exe;servertype=0;ASHOST=172.xx.x.xx;CLIENT=070;SYSNR=20;Timeout=3600;SNC_MODE=false;SNC_QOP=9;advancedProperties=true;ConvRoutine=false;CheckSeparator=false;Nulldate=true;RemoveAllBlanks=false;ReplaceNullvalue=false;Trace=false;MixedMode=true;NulldateLegacy=false;FLTPLegacy=false;TIMSLegacy=false;DataTypeLegacyMode=false;PacketSize=20000;FetchBuffers=99999999;BufferPercentage=15;TimeOutBatch=600;TimeOutFetch=600;TimeOutStartBatch=2400;BatchJobName=/QTQVC/READ_DATA;SocketBufferSize=32768;"

[Hakan_Ronningberg]

Hi,

Unfortunately there are two parameters with similar names. The one you refer to is internal and should always be 5 seconds. The connection string property you can change (TimeOutFetch) is used by the Windows part of the connector:
"The number of seconds spent trying to fetch from SAP without getting any records back."
You can read more about the connection string properties here:
https://help.qlik.com/en-US/connectors/Subsystems/SAP_Connectors_Help/Content/Connectors_SAP/Clients...

Have you checked the job log for job /QTQVC/READ_DATA in transaction SM37?

Regards,
Håkan

[andy]

Thanks for that explanation of the timeout-parameter.

We see this in SAP

07.02.2023 05:21:25 Job started                                                                               00           516          S
07.02.2023 05:21:25 Step 001 started (program /QTQVC/READ_DATA, variant &0000001399862, user ID QLIKSAP)      00           550          S
07.02.2023 06:02:20 Timeout reached while waiting for clearance of shared buffer.                             00           001          E
07.02.2023 06:02:20 Job canceled after system exception ERROR_MESSAGE                                         00           564          A

[Tomas_Sommergyll]

Hi, 

The log from your SAP System shows that the batch job is working fine and that it is waiting for the Windows part to fetch the next piece of data in the buffer. 

As we have seen from the Windows log, the data fetch is not happening for some reason. 
The next part to investigate is now the network connection: 

• Does the issue happen when using different clients (e.g. try from another computer)?
• Does it happen on a certain time of the day or at random times?
• Is the client connected using wi-fi or ethernet cable? (Try to swap if possible).
• If possible, try using a network scanner program to see the exchange of packages and possible disruptions.

/Tomas

[andy · Accepted solution]

Hi,

We have found the root cause.

There are redundant paths for reverse traffic from the core network to the data-center, where the Qlik nodes are installed.

One of these paths has a degraded optic fiber cable which causes intermittent errors.

These errors prevented the traffic to reach the Qlik-machine which closed the TCP-session due to timeout. Since the other path was working correct the error showed up randomly for us.

If anyone end up reading this in the future these are good commands to run.

On the Qlik-nodes:

netstat -e

On the switch (the yellow figures shouldn't be there):

"show interfaces counters errors"
"show interfaces eth1/34 counters errors"

Thank you @Hakan_Ronningberg and  @Tomas_Sommergyll  for responding and pointing in the right direction.