SQLite browse of large tables

[Marc Vanzegbroeck]

Hello,

I'm using SQlite, and for browsing the data I make a query or the database that is located on the server, and store the fields I want browse, in a temporary dbf-file on the a local path.
Untill now I had no problems, but I created a testtable of 120000 records and the proces to read the data from the server into a temporary file takes a long time.
Is there a faster way to do it? I was thinking first read only 100 records, show them, and then reading the rest into the background and add them to the temporary database. Does anyone created something like that already?

Thanks,
Marc

[James Bott]

Marc,

First I am not sure why you would need 120000 records in a temp file? Perhaps you should re-think the design.

If you are committing the writes to the temp file these commits could be eliminated. If you are use a SQLite file as the temp file then the commits take a really long time. Even if you are using a DBF removing the commits will speed it up.

Since the temp file is only going to be used by the person creating it, you don't need any commits. Well, maybe one at the very end.

Regards,
James

[Francisco Horta]

Marc,

Try using create temporary table in SQL, and charge records from an insert is faster,, i use temp table and work fine in mysql
regards
paco

[Marc Vanzegbroeck]

James, Francisco,

I only create a temporary DBF-file because I'm using the library http://www4.zzz.com.tw/phpbb2/viewforum.php?f=1, and this one doesn't support browsing with xbrowse.
So I first copy the data from the SQL-database to a temp-DBF file (without commits), and browse it like a normaly DBF-file.
Normaly it's no problem, but whith very large files , the copy can take a while.
I solved the problem by limiting the amount of records to browse. The client can now select this number.

Regards,
Marc