Multiple record layouts in a single EBCDIC file

[wlawton_2019]

We have a single EBCDIC file that contains multiple record layouts. That is, a header record followed by one to four different detail records. I am attempting to use the data mapper to read and convert the EBCDIC file to an XML file. Data mapper works perfectly when the source file has a single record layout and a single copybook. I see the multiple record types support for flat files, but this is not really a flat file. This is a COBOL EBCDIC file. I see in the web a suggestion to create a merged copybook where each individual copybook is turned into a redefines for the entire record. I am testing that approach. 

Is there a recommended best practice for loading multiple COBOL copybooks into a single structure/data map?

Thank you for any suggestions

[Anonymous]

Hello,

tFileInputRaw reads all the content of a file and sends it to a single output column.

Could you please try to use tFileInputRaw along with tHmap to read data your cobol copybooks?

Best regards

Sabrina

[wlawton_2019 · Accepted solution]

This was an interesting problem to solve, but we eventually found a solution. First, the support in Talend Data Mapper (TDM) for Cobol copybooks is still maturing. I am sure that future releases will have more complete support for some of the things in this post. There are several issues we had to work around.

Issue 1: TDM does not support the notion of reading multiple copybooks into a single structure. This is required in order to properly parse and map a multi-record formatted data file.

Solution: Merge the multiple copybooks into a single copybook and run the import. This was a start but was not a complete solution. 

Issue 2: TDM does not currently support multiple 01 records in a copybook. TDM was not able to properly setup the expected references for each of the multiple structures.

Solution: Create a new O1 record called something like "root" or any meaningful name that you like. Push all the previous 01 rows to 02, 02 to 03, 03 to 04, and so on. This allowed TDM to import into a single structure, but is still not a complete solution.

Issue 3: TDM attempted to map each record type into the first record definition, and threw mapping errors for all except the first record type.

Solution: In our case, there are five record types in the multi-record file. We chose the header record ("HDR-HSE") to be the primary record definition, and added "REDEFINES HDR-HSE" to record types 2, 3, 4, and 5. This brought us closer but is still not a complete solution.

Issue 4: The second field of each record is a record type identifier. We followed the instructions in the TDM documentation for setting up a "type" and ran the mapping job. TDM basically ignored the "type" values.

Solution: After a few hours of trying different alternatives based on the TDM documentation, I sat down with a beer and asked myself how it should work instead of what the TDM doumentation said. The solution was to shift the first two fields of the 02 records up to the 01 root. Then in the map, add a validation function to the 02 level of each record type in the copybook. Specifically, added this to each record definition in the map:

Equal

     First value

          /root/Record/Segment-Names

     Second value

          Constant 'xx'

where 'xx' was replaced with the record type for that record.

We now have a working solution for converting Mainframe Cobol data files with a multi-record layout into XML for downstream processing.

Questions?

Have a pleasant day

Wayne

[Anonymous]

Still I am not getting the way to use "redefine" and "depending on" feature in talend.