topic Re: Bulk loading into Cassandra in Talend Studio

Bulk loading into Cassandra

Anonymous — Sat, 16 Nov 2024 10:22:08 GMT

Hello,
I need to move millions of rows from a MSSQL server to Cassandra and i'm using the tCassandraOutputBulkExec. It works fine for thousands of records, but as soon as the number of rows reaches the hundreds of thousands, the job will start to slow down due to no garbage collection.
Is there a way to periodically generate the sstable and garbage collect the uneeded rows? Should I use tCassandraOutput?

Re: Bulk loading into Cassandra

Anonymous — Mon, 26 Sep 2016 07:58:42 GMT

Hi,
Is there any error message printed on console when your job start to slow down? Could you please show us your tCassandraOutputBulkExec component setting screenshot?
Best regards
Sabrina

Re: Bulk loading into Cassandra

Anonymous — Mon, 26 Sep 2016 15:01:38 GMT

Hey! To put it simply, the garbage collector runs out of memory. My guess is the component tries to create one big massive sstable while holding all the data in memory. It must be said, that the table that I'am importing has around 38 columns. I've included screenshots of a successful run, the error and the component settings.
When using a week's worth of data, the job completes successfuly, but it fails when loading more than one month.

Re: Bulk loading into Cassandra

Anonymous — Tue, 27 Sep 2016 05:45:54 GMT

In place of doing Million rows in 1 go, can you try to break them in Chunks and then load? You can create a subjob for processing chunks and give an explicit GC clean command using tJava.
Thanks,
Sankalp
+919811103231

Re: Bulk loading into Cassandra

Anonymous — Thu, 29 Sep 2016 15:03:36 GMT

Hey, I'll probably go that route.
Thanks for the awnsers.