In trying to optimize our reload tasks, I was wondering what some of the recommended AWS Instance Types are for a dedicated reload node. Currently we're operating on r6a.4xlarge, a memory optimized instance, with 10 as max concurrent reloads. CPU typically peaks at 96% with 12 max concurrent reloads, after I down scaled to 10 it was roughly 93%, but the sample size of a week post change is still small. We used to get some failing tasks without error log on 12, so I'm trying to look for ways to speed up the queue and reduce the errors.
In my research I see the recommended max concurrent reloads is actually NumberOfCores - 2, which for a r6a.4xlarge with 8 cores would reduce the max concurrent reloads to 6.
Is it worth looking into a Compute optimized instance? What are the recommendations for a dedicated reload node? And could I potentially gain speed and reduce errors with reducing the max concurrent reloads even further? I have to be mindful of time in queue, which I've found hard to quantify in the Intermediate Steps in the Reloads Monitor. Are there better ways to measure time spent in queue for tasks?
