topic Re: Extract strings from log file in Talend Studio

Extract strings from log file

Parikhharshal — Sat, 16 Nov 2024 03:10:26 GMT

Hi experts

I have got log file like below from which I want to extract Applicant ID, Agent Applicant ID and Error:

OES Admission Application Process Log at 2020-02-18 09:12:34.733

***** START OF LOGS *****

Application Name:ERWGERG Application ID: A-5562 Cal Applicant ID: 431234 CRM Applicant ID: C-4357623
[INFO] - New Applicant ID 431234 created for Agent Applicant ID 0035Q000002hzDmQAL.
Preference 1 for application 123 has been successfully loaded into Applicant Portal.
No document has been inserted as no information has been provided.
Applicant Application data has been inserted, even though error(s) did occur. The data that caused the error has been ignored or the error has been ignored.

[ERROR] - Field <SCNDRY><COMPLETION_YR> that has been defined as mandatory by the institution has not been supplied.
Secondary data supplied has been matched to data that already exists for the applicant therefore the data has been ignored.
Tertiary data supplied has been matched to data that already exists for the applicant therefore the data has been ignored.
Either Planned Date or Date Achieved must be entered.

What's the best way to extract and if regex needs to be written, what it should be?

Thanks

Harshal.

Re: Extract strings from log file

Anonymous — Wed, 26 Feb 2020 06:42:15 GMT

Hi
The format of log messages are always the same? and what Error string do you want to extract?

Regards
Shong

Re: Extract strings from log file

Parikhharshal — Wed, 26 Feb 2020 06:55:46 GMT

@shong The format will remain same but I will have multiple messages in same log file like below:

Admission Application Process Log at 2020-02-26 16:34:06.124

***** START OF LOGS *****

Application Name: FRT450as TEST Application ID: A-445664 Callista Applicant ID: 1234567 CRM Applicant ID: C-453234
[INFO] - New Applicant ID1234567 created for Agent Applicant ID 0035P000002lpE3QAI.
Preference 1 for application 123 has been successfully loaded into Applicant Portal.
No document has been inserted as no information has been provided.

[ERROR] - Field <SCNDRY><COMPLETION_YR> that has been defined as mandatory by the institution has not been supplied.
Only one of Institution Code and Institution name is required.
Your application is incomplete. You must provide at least one qualification for secondary education, tertiary education, other qualification or work experience before submitting
Applicant Application data has not been inserted as an error has occurred.

Application Name: FREF23 TEST Application ID: A-345656 Callista Applicant ID: 2345678 CRM Applicant ID: C-A0534546
[INFO] - New Applicant2345678 created for Agent Applicant ID 0035O000002lowAWAQ.
Preference 1 for application 123 has been successfully loaded into Applicant Portal.
No document has been inserted as no information has been provided.

So I want to capture anything after [ERROR] and capture it.

Re: Extract strings from log file

Parikhharshal — Wed, 26 Feb 2020 20:14:14 GMT

Hi @rhall @shong Could you please reply?

Re: Extract strings from log file

Parikhharshal — Thu, 27 Feb 2020 00:36:40 GMT

@rhall Could you pls respond to this?

Re: Extract strings from log file

Anonymous — Thu, 27 Feb 2020 16:58:31 GMT

The easiest way is to standard Java String methods (https://docs.oracle.com/javase/7/docs/api/java/lang/String.html). You know the words that precede the data you want. So you need to find their positions in the String (using something like "indexOf()") then you will need to look for the first space character after your String has started. The easiest way to do this will be to use code similar to this (assuming the text is in row1.body)....

String startOfApplicantId = row1.body.subString(row1.body.indexOf("Application ID:"));
String ApplicantId = startOfApplicantId.trim().subString(0,startOfApplicantId.trim().indexOf(" "));

The above code was written off the top of my head, so may need tweaking. But that is essentially what you need to do.

Re: Extract strings from log file

Parikhharshal — Sun, 01 Mar 2020 20:48:20 GMT

Hi @rhall

I wrote below code and achieved it.

String str = row18.line;

String strFind = "Callista Applicant ID:";

String strFind1= "for Agent Applicant ID";

String strFind2= "[ERROR] -";

String strFind3= "Application Name:";

int count = 0, fromIndex = 0;

int count1 =0, fromIndex1= 0;

int count2 =0, fromIndex2= 0;

while ((fromIndex = str.indexOf(strFind, fromIndex)) != -1 ){

// System.out.println("Found at index: " + fromIndex);

String str1 = str.substring(strFind.length() + fromIndex, strFind.length() + fromIndex + 9);

str1 = str1.trim();

System.out.println("Callista applicant ID " + str1);

count++;

fromIndex++;

}

while ((fromIndex1 = str.indexOf(strFind1, fromIndex1)) != -1 ){

// System.out.println("Found at index: " + fromIndex1);

String str2 = str.substring(strFind1.length() + fromIndex1, strFind1.length() + fromIndex1 + 19);

str2 = str2.trim();

System.out.println("Account ID " + str2);

count1++;

fromIndex1++;

}

However, I want to store value of str1 and str2 in db and do further processing. How cna I do that?

Re: Extract strings from log file

Anonymous — Mon, 02 Mar 2020 03:43:03 GMT

Do you write these code on tJavaRow or tJavaFlex component? We usually define context variables and store the value of str1 and str2, and then these context variable can be used later on other components.

Regards
Shong

Re: Extract strings from log file

Parikhharshal — Mon, 02 Mar 2020 04:00:24 GMT

@shong I know you can store in context variable.

I have got multiple values (multiple applicant ID, multiple Account Id) to be captured from log file. Then how do I capture in tJava like this below

Applicant Id, Account ID

123,345

234,677

456,466

and push it into DB?

Re: Extract strings from log file

Anonymous — Mon, 02 Mar 2020 06:03:20 GMT

As I mentioned, I usually use context variable to store the value. Then, i will use tFixedFlowInput to generate the data flow and insert them into DB.
What is the value of str1 and str2 you can get now? Show me the value.

Re: Extract strings from log file

Parikhharshal — Mon, 02 Mar 2020 06:22:12 GMT

@shong

The values I am getting are like this from tjavarow using println (using arraylist to output):

Applicant_id: [2084506, 2084507, 2084508, 2084509, 2084510]

Account_Id: [0035O000002lpB3QAI, 0035O000002lowPQAQ, 0035O000002lpBrQAI, 0035O000002lpCaQAI, 0035O000002lpD9QAI]

Re: Extract strings from log file

Anonymous — Mon, 02 Mar 2020 10:54:04 GMT

If you are storing these values in ArrayLists, you are almost there. I assume that you have used a tJavaFlex. I imagine you created the ArrayLists in the Start Code section and populated them in the Main Code section. To release these values to a row so that you can update your DB, you can either alter your code to release these values in Main Code section. Simply create the columns you want to output and assign the values to those columns instead of adding them to ArrayLists. Alternatively, if you need to assign the values to ArrayLists, you can read those ArrayLists using another tJavaFlex. Consider the Start Code as the area to open your loop. Maybe like this....

//Assuming your ArrayList is called myArrayList
for(int i = 0; i<myArrayList.size(); i++){

Then in the Main Code extract the values in the myArrayList using the value of "i" and assign them to output columns.

You have to remember that you need to close your for loop with the following code in the End Code section....

Re: Extract strings from log file

Parikhharshal — Mon, 02 Mar 2020 18:30:40 GMT

@rhall Could you please share job design or something which can show me how would you accept it as column and store? That would be really helpful because that's where I am stuck and not able to proceed.

Re: Extract strings from log file

Parikhharshal — Mon, 02 Mar 2020 18:33:12 GMT

@rhall I am suing tjavarow and below is my code:

String str = row18.line;

String strFind = "Callista Applicant ID:";

String strFind1= "for Agent Applicant ID";

String strFind2= "[ERROR] -";

String strFind3= "Application Name:";

int count = 0, fromIndex = 0;

int count1 =0, fromIndex1= 0;

int count2 =0, fromIndex2= 0;

List<String> results1 = new java.util.ArrayList<String>();

List<String> results2 = new java.util.ArrayList<String>();

while ((fromIndex = str.indexOf(strFind, fromIndex)) != -1 ){

// System.out.println("Found at index: " + fromIndex);

String str1 = str.substring(strFind.length() + fromIndex, strFind.length() + fromIndex + 9);

str1 = str1.trim();

//System.out.println("Callista applicant ID " + str1);

//globalMap.put("Applicant_Id",str1);

//System.out.println("Applicant_Id: "+((String)globalMap.get("Applicant_Id")));

results1.add(str1);

//results[count] = globalMap.get("Applicant_Id");

count++;

fromIndex++;

}

while ((fromIndex1 = str.indexOf(strFind1, fromIndex1)) != -1 ){

// System.out.println("Found at index: " + fromIndex1);

String str2 = str.substring(strFind1.length() + fromIndex1, strFind1.length() + fromIndex1 + 19);

str2 = str2.trim();

results2.add(str2);

//System.out.println("Account ID " + str2);

//globalMap.put("Account_Id",str2);

//System.out.println("Account_Id: "+((String)globalMap.get("Account_Id")));

//results[count1] = ","+globalMap.get("Account_Id");

count1++;

fromIndex1++;

}

//globalMap.put("Applicant_id",results1);

//globalMap.put("Account_id",results2);

String[] arr1 = new String[results1.size()];

arr1 = results1.toArray(arr1);

//globalMap.put("App_id",arr1);

//System.out.println("App_Id: "+((String)globalMap.get("App_id")));

System.out.println("Applicant_id: "+ java.util.Arrays.toString(arr1));

String[] arr2 = new String[results2.size()];

arr2 = results2.toArray(arr2);

// globalMap.put("Acc_id",arr2);

//System.out.println("Acc_id: "+((String)globalMap.get("Acc_id")));

System.out.println("Account_Id: "+java.util.Arrays.toString(arr2));

Re: Extract strings from log file

Anonymous — Mon, 02 Mar 2020 21:37:43 GMT

Look at this post (https://community.talend.com/t5/Design-and-Development/Dynamic-column-export-with-order/m-p/205749/highlight/true#M112597). The tJavaFlex_2 section covers what you want to do.

Re: Extract strings from log file

Parikhharshal — Tue, 03 Mar 2020 03:27:37 GMT

@rhall I am using tjavarow. Does it mean I will have to change my code to accomodate according to tJavaflex and end both the loops in end code?

Re: Extract strings from log file

Anonymous — Tue, 03 Mar 2020 09:32:22 GMT

When you are reading the data, you can do that in a tJavaFlex without the Loop starting in the start code and ending in the end code. The "loop" is controlled by the rows of data coming in (your log file). You will need to save the data (the ArrayLists) using the globalMap in the end code though. When reading the data in, it should work exactly the same way as the example I sent you. However, you will probably want to use a for loop instead of an iterator with a while loop. This is because you will want to retrieve data from two ArrayLists. I'm assuming that they will have the same number of elements.

Re: Extract strings from log file

Parikhharshal — Tue, 03 Mar 2020 09:41:52 GMT

Yes that's right both will have same number of elements.

But how do I write for loop for both together? What I don't get is how do you create two columns in first component and assign the values there using for loop? If you could help here, would really appreciate. I am literally stuck here since many days.

Re: Extract strings from log file

Parikhharshal — Tue, 03 Mar 2020 10:02:26 GMT

In my case I must use while or some loop to find multiple occurrences of same attribute. If you see my pasted code you will know why I’m using loop.

Re: Extract strings from log file

Anonymous — Tue, 03 Mar 2020 10:12:32 GMT

Yes, I see that in your input code (populating the ArrayLists), but in your output code (getting the data from the ArrayLists) it will probably be better to use a for loop