That's probably a little overkill, but it will give you room to expand. Your exact memory requirements depend on how many concurrent users you are expecting. For 4GB of data, if you have 5 concurrent users, you'll need about 16GB RAM. If you have 10 concurrent users, you'll need about 20GB RAM. On top of that you'll need to add Windows and application overhead, which can also be a couple GB. So your current configuration of 32GB is only slightly more than you will probably need.
Hope this helps.
On a separate note, is it a bad idea to run server and publisher on the same hardware? I don't see significant downsides.
Depends on your reload schedule and user access patterns. If significant reload and user activity is at the same time. I prefer to split Server and Publisher between two boxes. Publisher is batch and Server is interactive, quite two different resource profiles.
If your reloads are all occuring off hours, no problem.