I have a requirement to create a filter based on any combination of data from 39 fields. I realise that this is a lot of combinations, and will try and get this down to, say, up to 8 in each.
I managed to get this working in a POC using just 4 fields. Here's the code:
When I try and increase this to pick 4 combinations from the 39 dims, I have tens of thousands of field combinations, and it takes hours to load.
Can anyone suggest a different way to do this? I was thinking perhaps having a table with the field names which the users can select from, and to try and use these selected fields as a type of dynamic filter which the user could then select the values from. I just can't quite figure out how to do it though.
I think it will be in the end too many distinct and large string-values to work with them in a sensible way within an UI - max. are 2 billions distinct field-values and if you comes near them with large strings ... it won't be very responsive even by a larger cluster environment. Further I'm not sure if the entire requirement is really suitable for a BI tool. The hinted machine-learning project sounds more sensible and if anyway larger loop-approaches are needed to create all possible value-combinations another tool like Python may at all the better solution.
Before that you may consider to implement only parts of the requirement, for example by applying the above mentioned crosstable transform of the 39 fields into rows. In a next steps the rows might be categorized and/or prioritized - means adding extra fields for it - and maybe not only ones else twice/thrice. Based on this you may add further calculations, for example applying accumulations with interrecord-functions like peek() and previous() which may in a following step be clustered with class() and/or flagged in any way. Not each combination will be considered but through the categorizing and prioritizing the most relevant ones should be detectable.
At all I could imagine that such approach may fulfill the real aim better as an AI solution because the results will be further comprehensible while an AI would be a black-box.
