It looks like you need the firstsortedvalue() function. To get the latest value by User use -TestDate as the sort weight:
SUM(aggr(firstsortedvalue(TestValue,-1*TestDate), UserId, TestName, Year))/Count(DISTINCT UserId)
You are right, it will work way better in the script. You can use firstsortedvalue in the script too, I would create a separate ke yof unique Users/testname / year (if not already) and join onto it your calculation grouped by user with the first sorted value for each.
Autonumber(UserId&TestName&Year) as KEY,
firstsortedvalue(TestValue,-1*TestDate) as latestvalue
group by UserId, TestName, Year;
Let me know if this helps,