The working set configurations are only for the server and don't touch the publisher directly. If server and publisher run on the same machine the will compete to the system-resources but on different machines they are independent of eachother.
In your case the RAM for the publisher may not be sufficient for this task especially if this app is compressed the real RAM size could be significantly higher. A look on the windows taskmanager if this task is running should show this quite clearly. If this is the case you need to increase the RAM capacity and/or optimizing the application.