I believe the reference line feature is just for a straight line that can be hard-coded or calculated to represent a single value. You may need to use a Combo chart with another calculated expression measure to perform as the reference line. Take a look at this thread where the request is similar - where they are using the calculated measure to plot a trend line:
Let me know if this puts you in the right direction.
When applicable please mark the appropriate replies as CORRECT. This will help community members and Qlik Employees know which discussions have already been addressed and have a possible known solution. Please mark threads as HELPFUL if the provided solution is helpful to the problem, but does not necessarily solve the indicated problem. You can mark multiple threads as HELPFUL if you feel additional info is useful to others.