Talend Data Mapper - Finding the structure of a EBCDIC file without the copy book
Hi Team,
I need to create a data map for an EBCDIC data file. However, we do not have the copybook for the data file. Also we are not sure of the exact encoding. However, we do have the metadata from a different DI tool (DataStage). Is it possible for us to create the structure and do the mapping without having the copybook?
There is a way to do it using what is called an Importer structure. However, it's not easy. I'm checking to see if I have it documented somewhere. In the meantime, can you do a copy/paste of the datastage metadata and attach it?
Thanks @thoff I just received the datastage columns screenshots and started working on it. I will try with IBM037 encoding as it seems to be the default. Please do let me know if you get more information on the importer structure. I saw that on user guide. But it was very minimal details as shown below.
Using a Map to Import Definitions
You can use a map to create a structure definition from any input. For example, suppose you have a positional structure that is described by a spreadsheet. The spreadsheet contains a list of element names, sizes, start columns, and further description. You can export this spreadsheet as a CSV file and create a map that maps the contents of the spreadsheet to the Importer structure.
TheImporterstructure is a predefined structure in theBuiltinproject (Builtin/Structures/Executable Structures/Importer). When it is executed in a map, theImporterstructure can create one or more structures whose elements are defined by the mappings. To use it, create a standard map, and specify theImporterstructure as the output structure. You can specify any input structure you like and map the elements to produce the desired elements in the structure. You can build and test your map in the studio as usual, producing the output of the map in XML for testing purposes. TheTest Runmenu will provide an additional option calledTest Run to Importer Structure, which will actually create the structures. You can re-run the map as many times as you like, which creates the structures each time and replaces their contents.
How many columns are there? The thing about the copybooks is that they can contain packed fields (COMP, COMP-3) which will need to be set up correctly.
Thanks for sharing the recording @thoff. For me to even take a view at the EBCDIC datafile without the copybook, I may have to convert the data file to ASCII, right? After that at least I should be able to see the initiators for the records and based on that, we can dig deeper as per to the process mentioned in the video, right?
How did you create the csv file with columns? Which utility helped you to create that? As per to the Video, seems like you created it manually. In our situation, we have only some datastage columns tab screenshots. However, we cannot view the contents of the data file. I am tempted to convert the file to ascii.
Can you ask them to copy/paste the datastage info to you? It can probably be parsed by TDM and then you can create your csv file from that. I've done a similar thing before. No, shouldn't have to convert it to ASCII first. That's what TDM is for.
I finally received the copybook details from them. Though the email contained something like below. If I copy the content between the top and bottom markers, save it as UTF8 windows text file and import the copybook, data mapper is throwing many errors. However, if I just copy the contents without the line numbers and import it, it is getting imported fine. Is the second method, the correct way of importing the copy book? After importing the copybook, I am still facing challenges on viewing the data from the sample file received. Hence I am doubting my approach. Once you confirm the approach I will post the details of the other errors?
