If you mean with RMSE the root mean square error, there’s no such ready-for-use function in Qlik Sense. Qlik Sense has got many statistical functions (check the help) that should enable you in calculating the RMSE. Check Wikipedia here and here for more background of the RMSE.
I'm aware I didn't really answer your question but I hope this will help a bit.
Something along these lines:
=sqrt(Sum(sqr(val1 - val2)) / Count(val1))
Where val1, val2 are the y and y' values. This assumes that val1 and val2 are properly associated with a 1:1 relationship. Substitute the correct field names for your document.