tLog row displaying different # of outputs than tPostgresqlOutput sends to DB
I have an odd problem, basically I have a parent job setup that takes in an excel sheet with 38 different context groups. The parent job runs taking in 1 context group at a time, sends it to a child job where it runs some API calls does some calculations and pushes the results to the DB waits 3 seconds then starts the parent job again iterating through 38 times. I've been seeing some weird issues where there are 0's being sent to the DB when they should not be. I've been doing some testing, and at the last step of the child job I have a tmap to tpostgresqloutput. I've added a tlog row as a separate output from the tmap to see what's happening. When I compare the tlog row to what is pushed to the DB sometimes there is a "double row" I'll call it. I see lets say 40 rows in the tlog outputs, for example, when the DB only shows 38 (which is the correct amount). It seems one of the "double rows" has the incorrect data and sometimes is being pushed into the DB rather than the correct row. I'm confused to why were getting double rows, and I've attached screenshots of the two jobs. If anyone has any ideas on what's going on I would love to hear it!
No filters. I could run the child job by itself, but the parent job is what allows me to run all 38 at once which I think has something to do with the issue I'm having. If I ran the child job alone, I would need to specify the context and only be able to run with 1 context group.
That's the issue with running child jobs. They are incredibly useful, but lead to issues with tracking problems like this. Could you maybe make a change to enable the a couple of contexts to be tested within this job. I think it is the best way of identifying this.
The reason I originally created the parent/child jobs was due to the fact I could not find a way to loop through and get the context parameters without this setup. I will try to see if I can get this created to be able to do some testing as you mentioned.
Hi @rhall , so I was able to create 1 job (no parent/child relationship) and have some screenshots of a good vs bad output. I think it may have something to do with the taggregate row grouping by date? If you could take a look and let me know what you think that would be much appreciated, thanks again!
It looks like the same number of rows are hitting your tLogRow as your db component. I recall from your previous screenshot of your tMap that the tLogRow has slightly different records to your db component. There was a single link that went to one of the outputs that didn't appear to go to the other.
Are you getting any errors from your db? You said you were getting 40 rows to your tLogRow and only 38 to your db. This *could* be caused by the an error at the database end.
Regarding your aggregate, this could be caused by your date. I believe you are using the current date which would include time. It is entirely possible (likely in fact) that the difference in milliseconds or even seconds or minutes would cause this.
On the tPostgresqlOutput it's set to insert or update based on the report_date and product_id fields so I think even though 2 are coming into that component only 1 is sent to the DB. I thought the date may be the issue, but wasn't aware it was date time. I will reformat this and do a little more testing and hopefully this change will fix the issue.
