Recipients can be associated with as many groups as desired when importing them. You have 5 Group fields and each can have several groups listed within them (separated by commas). Therefore, one recipient can potentially belong to several dozen groups!
Use the appropriate group as the recipient in the Report Task and add/remove recipients to that Group as required.
An example would be:
You have recipients Tom, Dick & Harry
You have Report Tasks T1 & T2
You want T1 to only go to Tom & Harry
You want T2 to only go to Dick & Harry
Assign Tom & Harry to group G1 - add G1 as the only recipient in T1
Assign Dick & Harry to group G2 - add G2 as the only recipient in T2
You need to plan this out for yourself according to your requirements.