SQLite disk I/O error

[SD3156]

Hello Guys,

I have developed a comparison tool using Talend, which will compare 2 huge SQLite Database (80GB each) and store the compared result in another SQLite Database.

Problem:

1) The tool is unable to read Input SQLite file , if size is beyond 2GB as it shows an error "java.sql.SQLException: database disk image is malformed", but if the file size less then that it works, have checked the Integrity (OK) and it also opens with SQLite tool. So for now I moved to CSV file as Input.

2) But after comparing when I wanted to store the result in SQLite Database it shows "java.sql.SQLException: disk I/O error" after reaching to a file size close to 2GB. I have checked the server size its 300GB free and 16GB RAM available.

Any suggestion what can be done?

Screen Shot 1: database disk image is malformed

Screen Shot 2: disk I/O error

Regards,

SD3156

[gjeremy1617088143]

HI,@Sounav Das​ :

Although Sqlite documentation states that maximum size of database file is ~140 terabytes but your OS imposes its own restrictions on maximum file size for any type of file.

For e.g. if you are using FAT32 disk on Windows, maximum file size for sqLite db file was 2GB. (According to Microsoft site, limit on FAT 32 system is 4GB but sqlite db size was restricted to 2GB). While on Linux ,it's able to reach 3 GB.

I can suggest you to fetch the the read and the write in file < 2GB.

Send me love and kudos

[SD3156]

Hi,

Thank you for the answer.

I am using NTFS format for my disk and as already mentioned that the input file will be 80GB so your suggestion of using <2GB will not help.

Regards,

Sounav