You can use the EXCLUDE keyword ( https://help.qlik.com/en-US/sense/June2017/Subsystems/Hub/Content/Scripting/ScriptRegularStatements/Search.htm )
This will help you exclude any fields that you think are unnecessary.
Typically if fields are 'string' intensive, smart search takes lot of time to parse.
Numeric fields shouldn't take much time to parse
although they are just used for pre opening sheets and objects you could try one the "Cache warmer/initializer" applications (and maybe modify one of them to search once so that smart search will be initialized):
http://branch.qlik.com/#!/project/56728f52d1e497241ae69811 - Directlink to github if branch is still down: GitHub - JoeBickley/CacheInitializer: A tool for Qlik Sense to cache warm applications to RAM)