topic Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB in Talend Studio

tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Sat, 16 Nov 2024 04:14:01 GMT

I have an odd problem, basically I have a parent job setup that takes in an excel sheet with 38 different context groups. The parent job runs taking in 1 context group at a time, sends it to a child job where it runs some API calls does some calculations and pushes the results to the DB waits 3 seconds then starts the parent job again iterating through 38 times. I've been seeing some weird issues where there are 0's being sent to the DB when they should not be. I've been doing some testing, and at the last step of the child job I have a tmap to tpostgresqloutput. I've added a tlog row as a separate output from the tmap to see what's happening. When I compare the tlog row to what is pushed to the DB sometimes there is a "double row" I'll call it. I see lets say 40 rows in the tlog outputs, for example, when the DB only shows 38 (which is the correct amount). It seems one of the "double rows" has the incorrect data and sometimes is being pushed into the DB rather than the correct row. I'm confused to why were getting double rows, and I've attached screenshots of the two jobs. If anyone has any ideas on what's going on I would love to hear it!

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 15:21:06 GMT

You have a tMap before your db component and tLogRow. What is the configuration of that component?

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 15:23:51 GMT

ey again @rhall ! I've attached a screenshot of the configuration to this reply.

tMap.PNG

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 15:26:21 GMT

OK, that looks OK. I take it you have no hidden filter logic on the outputs?

Have you tried running just the child job? This might reveal (looking at the number of records passed through rows) something about the flow.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 15:29:31 GMT

No filters. I could run the child job by itself, but the parent job is what allows me to run all 38 at once which I think has something to do with the issue I'm having. If I ran the child job alone, I would need to specify the context and only be able to run with 1 context group.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 15:32:08 GMT

That's the issue with running child jobs. They are incredibly useful, but lead to issues with tracking problems like this. Could you maybe make a change to enable the a couple of contexts to be tested within this job. I think it is the best way of identifying this.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 16:37:30 GMT

The reason I originally created the parent/child jobs was due to the fact I could not find a way to loop through and get the context parameters without this setup. I will try to see if I can get this created to be able to do some testing as you mentioned.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 17:38:52 GMT

Hi @rhall , so I was able to create 1 job (no parent/child relationship) and have some screenshots of a good vs bad output. I think it may have something to do with the taggregate row grouping by date? If you could take a look and let me know what you think that would be much appreciated, thanks again!

outputs.png
tAggRow.png
tMap.PNG

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 17:53:00 GMT

It looks like the same number of rows are hitting your tLogRow as your db component. I recall from your previous screenshot of your tMap that the tLogRow has slightly different records to your db component. There was a single link that went to one of the outputs that didn't appear to go to the other.

Are you getting any errors from your db? You said you were getting 40 rows to your tLogRow and only 38 to your db. This *could* be caused by the an error at the database end.

Regarding your aggregate, this could be caused by your date. I believe you are using the current date which would include time. It is entirely possible (likely in fact) that the difference in milliseconds or even seconds or minutes would cause this.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 18:11:00 GMT

On the tPostgresqlOutput it's set to insert or update based on the report_date and product_id fields so I think even though 2 are coming into that component only 1 is sent to the DB. I thought the date may be the issue, but wasn't aware it was date time. I will reformat this and do a little more testing and hopefully this change will fix the issue.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 18:25:24 GMT

Date objects are not dependent on format. If you want a date that is limited to just yyyy-MM-dd you will need to ensure that the hours, minutes, seconds and milliseconds are set to 00:00:00.000. You can write a routine to do that which would be the easiest way to ensure it.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 18:39:21 GMT

I just created the date in the tMap with "TalendDate.parseDate("yyyy-MM-dd",TalendDate.formatDate("yyyy-MM-dd",TalendDate.getCurrentDate())) ". I'm assuming this would work as well, but if you see nay issues please let me know. I've ran it a few times with no issues, but sometimes there's 10+ runs that look ok so I will continue testing, but think this may have been the cause.

Re: tLog row displaying different # of outputs than tPostgresqlOutput sends to DB

Anonymous — Tue, 29 Oct 2019 18:53:05 GMT

I believe that would work. All you need to do to test it is to format the date with hours, minutes and seconds to see whether they are all 0s.