topic Re: SQLite disk I/O error in Talend Studio

SQLite disk I/O error

SD3156 — Fri, 15 Nov 2024 23:42:30 GMT

Hello Guys,

I have developed a comparison tool using Talend, which will compare 2 huge SQLite Database (80GB each) and store the compared result in another SQLite Database.

Problem:

1) The tool is unable to read Input SQLite file , if size is beyond 2GB as it shows an error "java.sql.SQLException: database disk image is malformed", but if the file size less then that it works, have checked the Integrity (OK) and it also opens with SQLite tool. So for now I moved to CSV file as Input.

2) But after comparing when I wanted to store the result in SQLite Database it shows "java.sql.SQLException: disk I/O error" after reaching to a file size close to 2GB. I have checked the server size its 300GB free and 16GB RAM available.

Any suggestion what can be done?

Screen Shot 1: database disk image is malformed

Screen Shot 2: disk I/O error

Regards,

SD3156

Re: SQLite disk I/O error

gjeremy1617088143 — Thu, 23 Sep 2021 09:33:16 GMT

HI,@Sounav Das :

Although Sqlite documentation states that maximum size of database file is ~140 terabytes but your OS imposes its own restrictions on maximum file size for any type of file.

For e.g. if you are using FAT32 disk on Windows, maximum file size for sqLite db file was 2GB. (According to Microsoft site, limit on FAT 32 system is 4GB but sqlite db size was restricted to 2GB). While on Linux ,it's able to reach 3 GB.

I can suggest you to fetch the the read and the write in file < 2GB.

Send me love and kudos

Re: SQLite disk I/O error

SD3156 — Thu, 23 Sep 2021 12:03:04 GMT

Hi,

Thank you for the answer.

I am using NTFS format for my disk and as already mentioned that the input file will be 80GB so your suggestion of using <2GB will not help.

Regards,

Sounav