Don't consider the number of records, the issue to look at is the number of unique records and the time taken to process the data. I guess your Group By clause is slow as QlikView is having to create the unique set of values in memory to process the group before summing the data.
First I would move the two IF statements into your first QVD file LANCAMENTO_CONTABIL.qvd
if(CTAPAIANT = IsNull(CTAPAIANT), CONTA_CONTABIL, CTAPAIANT) as CLAPAI5
if(CLAPAIANT = IsNull(CLAPAIANT), CLASSIFICACAO, CLAPAIANT) as CLAPAI6
Then add a compound key CLAPAI3 & CLAPAI5 & CLAPAI6 & CLAPAI4 & CLAPAI1 & CLAPAI2
This means that the generation of the unique values only happens once when the QVD is created, and the grouping will not have any IF clauses.