Rule based logic and/or detailed logging of calculations
I have a process built in TOS which transforms a file of expenses into various buckets. The heavy lifting of the logic is done in various tMap and tAggr components and often involves grouping multiple expenses into a single lump amount. Of course this then obscures the original raw values, and so then someone will ask - how did we end up with $199.95 in 'bucket A'? Where did that come from? That usually involves looking at the rule document (or the code), and the raw data, to prove that $199.95 is really correct and was comprised of $50 item1, $60 item2, and $89.95 item3. Sometimes the rule is more complex if there is a negative amount, put it somewhere else, etc. I need to get out of the business of providing this info.
One thought is have all the rules in some sort of rule table and log which rules were applied. Not sure if this is possible in TOS or even really does what I need. Another thought is that I need a way of a single tMap calculation doing two different things: providing an output value, but also some logging of what the input values were? I see no way to do that without duplicating the rule in two different places, which compromises maintainability of the code.
The ultimate goal would be to allow a web interface to see the resultant data but also be able to drill down to the underlying data. That could be just two database tables - a summary and a detail. Then any business person could go audit on their own.
Clearly I could build the whole thing in brute force Java, but I'm hoping to not abandon the productivity of Talend.
sum FeeIncAmount where FeeName in ('Qual', 'Mid', 'Non', 'ERR'). If result is < 0, set to zero.
Desired 'logging': FeeIncAmount was comprised of $100 Qual + $90 Mid
If the < 0 rule had fired, log that the amount was set to 0 because the component values were < 0.
Similar logic for NetAmount
The input file might or might not have all entries possible. It might only have Qual, and none of the others. For audit purposes the business wants to 'see the logic'. We can open up the code and the input/output data and look at a case by case basis... but the input file is around 50,000 fee records and the output is thousands of records, so the time required to manually examine any substantial number for accuracy is significant.
Oh, I should add, the output file has 200+ fields, each with its own calculation. I've greatly simplified the problem by only showing a 3 field output file.
