Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
1 Solution
Accepted Solutions
Yes you can do all these things with Talend, but some are more difficult than others.
You have to ask yourself why some of these conditions are occurring. If the spreadsheet is being created from an automated process, why would (1) the worksheet be missing or (2) some of the columns be missing. Is there something you can do outside of Talend to remove those types of errors. You may need to ask the people providing the excel data to do a better job
If a sheet is missing you will get an error, so you can handle with On Error for the Component or Subjob.
If a column is missing or out of place that causes your schema not to match, you can check with schema validation.
You can create a log table in a database, but you may have to write custom java with the tJava component to put into the file the exact information you want to capture. It all depends on what you need to log. You will have to investigate each Talend component to see what data values they provide you. Each component provides an error message which is very handy. Talend has some built in logging if you look in the documentation. There are three files they write to if you set your jobs up correctly; a Stats table, a Logs table and a Meter table. Find those in the documentation and use them if they fit your needs.
tMap has an expression editor to let you do almost anything you need to do to put data into an output field, so review all the code they provide in that editor.
Trimming can be done automatically with many components, like the tFileInputxxxx conponents. And the expression editor in tMap has code that will let you trim.
Coalesce can be done with Java equivalents like this syntax (expression true or false) ? do this if true : do this if false where you test if some field is null. You can nest these as well.
Hope this gives you some good ideas.
Thanks,
dg
You have to ask yourself why some of these conditions are occurring. If the spreadsheet is being created from an automated process, why would (1) the worksheet be missing or (2) some of the columns be missing. Is there something you can do outside of Talend to remove those types of errors. You may need to ask the people providing the excel data to do a better job
If a sheet is missing you will get an error, so you can handle with On Error for the Component or Subjob.
If a column is missing or out of place that causes your schema not to match, you can check with schema validation.
You can create a log table in a database, but you may have to write custom java with the tJava component to put into the file the exact information you want to capture. It all depends on what you need to log. You will have to investigate each Talend component to see what data values they provide you. Each component provides an error message which is very handy. Talend has some built in logging if you look in the documentation. There are three files they write to if you set your jobs up correctly; a Stats table, a Logs table and a Meter table. Find those in the documentation and use them if they fit your needs.
tMap has an expression editor to let you do almost anything you need to do to put data into an output field, so review all the code they provide in that editor.
Trimming can be done automatically with many components, like the tFileInputxxxx conponents. And the expression editor in tMap has code that will let you trim.
Coalesce can be done with Java equivalents like this syntax (expression true or false) ? do this if true : do this if false where you test if some field is null. You can nest these as well.
Hope this gives you some good ideas.
Thanks,
dg
This may give you some ideas. You could use tFileList to pick up the filename, input the file into tMap along with a lookup of the distinct BatchIDs of your database, filter your output to only the records that do not match the lookups (see second picture below), then move the file with tFileCopy to the archive if the insert record count was greater than zero or to the error folder if the record count equal to zero.
Yes you can do all these things with Talend, but some are more difficult than others.
You have to ask yourself why some of these conditions are occurring. If the spreadsheet is being created from an automated process, why would (1) the worksheet be missing or (2) some of the columns be missing. Is there something you can do outside of Talend to remove those types of errors. You may need to ask the people providing the excel data to do a better job
If a sheet is missing you will get an error, so you can handle with On Error for the Component or Subjob.
If a column is missing or out of place that causes your schema not to match, you can check with schema validation.
You can create a log table in a database, but you may have to write custom java with the tJava component to put into the file the exact information you want to capture. It all depends on what you need to log. You will have to investigate each Talend component to see what data values they provide you. Each component provides an error message which is very handy. Talend has some built in logging if you look in the documentation. There are three files they write to if you set your jobs up correctly; a Stats table, a Logs table and a Meter table. Find those in the documentation and use them if they fit your needs.
tMap has an expression editor to let you do almost anything you need to do to put data into an output field, so review all the code they provide in that editor.
Trimming can be done automatically with many components, like the tFileInputxxxx conponents. And the expression editor in tMap has code that will let you trim.
Coalesce can be done with Java equivalents like this syntax (expression true or false) ? do this if true : do this if false where you test if some field is null. You can nest these as well.
Hope this gives you some good ideas.
Thanks,
dg
You have to ask yourself why some of these conditions are occurring. If the spreadsheet is being created from an automated process, why would (1) the worksheet be missing or (2) some of the columns be missing. Is there something you can do outside of Talend to remove those types of errors. You may need to ask the people providing the excel data to do a better job
If a sheet is missing you will get an error, so you can handle with On Error for the Component or Subjob.
If a column is missing or out of place that causes your schema not to match, you can check with schema validation.
You can create a log table in a database, but you may have to write custom java with the tJava component to put into the file the exact information you want to capture. It all depends on what you need to log. You will have to investigate each Talend component to see what data values they provide you. Each component provides an error message which is very handy. Talend has some built in logging if you look in the documentation. There are three files they write to if you set your jobs up correctly; a Stats table, a Logs table and a Meter table. Find those in the documentation and use them if they fit your needs.
tMap has an expression editor to let you do almost anything you need to do to put data into an output field, so review all the code they provide in that editor.
Trimming can be done automatically with many components, like the tFileInputxxxx conponents. And the expression editor in tMap has code that will let you trim.
Coalesce can be done with Java equivalents like this syntax (expression true or false) ? do this if true : do this if false where you test if some field is null. You can nest these as well.
Hope this gives you some good ideas.
Thanks,
dg