You could use the option Rajesh mentioned, but then your other charts will respond to nothing anymore and become static charts.
Since the purpose of Qlik Sense is to create interactive and responsive dashboards, a better option is to just exclude the filters used in the drill-down.
Lets say you have a drill-down Productgroup > Product. You must add "Productgroup=, Product=," to the set analysis in your measure expression to achieve this functionality. Add this to every chart which you don't want to react to the active filters.
There is one workaround.
You can create 2 replicas of fields that you want to create a drill down of.
For Ex. You can create 2 sets dimension such as Category & SubCategory and Category1 & SubCategory1.
Then use category1 & SubCategory1 for Drill Down Dimension and nullify those 2 dimension from all other measures when you want there selection unaffected.