Re: Api to retrieve data(All rows) based on field/Dimension given
That "IteratePages" call will actually fetch you all the data, but it will do so by fetching a sequence of consecutive pages, each 100 entries long. The Pager.Next method ensures that each page is the next in line. (I see the documentation of that method should be more specific though...).
I assume that what you are really after is a full list of NxCell entries. To get there you could do something like this:
var allCells = allData.SelectMany(pages => pages.First().Matrix).Select(row => row.First());
By the way, I should probably have named that "allData" variable "allPages" instead. That would be more more correct description.
Another approach would be to check the cardinality of the field and then get one single big page that contains it all, but remember that the engine has a limit on the size of pages you can fetch. A page is not allowed to exceed 10 000 cells. So for big fields you will have to do some form of iterations.