Why not create a running total table by product and whatever other dimensions you want to support in the script? would be a lot faster than set analysis.
monthstart(masterdate) as date,
sum(qty) as inventoryCount,
you can create a composite key to rid of the synthetic that may result and do an AUTONUMBER on the composite key to gain in performance AUTONUMBER(monthstart(masterdate) & '_' & productid) AS key and have the same key in the Fact table
assuming that you might have a lot of transactions in your inventory file, the idea to pre-calculate a running balance is certainly the best, in terms of performance.
To answer the specific question, and to clarify Jagan's answer - in Set Analysis, when comparing dates, you need to spcify exactly the same date format as used for the date field in question. On the other hand, the function max() is always returning a numeric value. Therefore, it's impossible to use unformatted result of max() as a selection value for a date field. So, the correct formula should be: