Speedtest CLIPPER vs. xHarbour - COMMIT

[Otto]

Speedtest CLIPPER vs. xHarbour - COMMIT

Is CLIPPER still faster in database management?

I am testing the COMMIT statement.
With Clipper the operation takes 9 sec. with xHarbour 77 sec.

Best regards
Otto

This is my code:
CLIPPER

Code: Select all  Expand view

#include "FiveWin.ch"

function main()
local I:=0
msginfo("Start " + str( seconds() ))

use clientes new

for I := 1 to 1000
append blank
clientes->nombre := str(recno())
commit

next

msginfo("End " + str( seconds()) )

return nil

 

xHarbour

Code: Select all  Expand view

#include "FiveWin.ch"
#include "xbrowse.ch"

REQUEST DBFCDX


function main()
local I:=0


msginfo("Start " + str( seconds() ))
  rddsetdefault( "DBFCDX" )
use clientes new

for I := 1 to 1000
append blank
clientes->nombre := str(recno())
commit

next

msginfo("End " + str( seconds()) )
   

return nil

 

[ukoenig]

Hello Otto,

I commit for each record ?

I do this only once at the end of the loop, never for each record.
It forces to write to disk.

Regards
Uwe

[Otto]

Hello Uwe,

>It forces to write to disk

Yes, therefore I commit. In some circumstances I must be sure that the data is on the disk.

The loop is only to show that there is a speed difference and I don’t understand why.

I think if the developer of the RDD adress this issue soon xHarbour – I didn’t test with harbour –
is also in this case better than Clipper.
Best regards,
Otto

[xProgrammer]

Hi Otto

Interesting

Have you tried without the commit to prove its the commit and not the str() function or the append blank or the replace that is responsible for the rather large time difference?

One also wonders if maybe the xHarbour commit is doing more than the Clipper commit?

Have you tried Harbour for a comparison?

For a strict comparison the rddsetdefault() should come out of the timed loop.

xProgrammer

[Enrico Maria Giordano]

I am testing the COMMIT statement.
With Clipper the operation takes 9 sec. with xHarbour 77 sec.

I get 33 with Clipper 5.3b and 46 with latest xHarbour. If you really get 9 then your commit seems to be an empty statement.

EMG

[Otto]

Hello Enrico, hello Doug,

thank you for your interest and help.

Doug, I will change my test program. But I faced the behavior in my real program.
In 286 + 386 times I used a lot of commits because many times in those times the hardware made problems and you never were sure if your data was stored or not. As - thanks to Antonio - also very old code still is running unchanged with harbour there are parts of code I didn’t touch for decades.
Now I noticed that some parts of my program are slower as they used to be with Clipper and I found out that COMMIT is the reason.

Enrico your test does not show much difference. Did you do your tests under VISTA. I will try what results I get using XP or W2000.
I use 5.2 and COMIX as a add on.

I also have a speed problem(reading) with xharbour if a database on a XP file server is opened shared by more than one user.
There is also a post from Marco I think with the same issue. But as XP is not much in use as file server on my clients side I am not effected and forgot about that till I read Marcos post.
Also I changed my programming style. I open the dbf-files as short as possible.

I will report about my tests on the XP and W2000 PC. Enrico would you be so kind to email me your Clipper and xharbour test.exe (datron_____@____aon.at).

Thanks in advance
Otto

[driessen]

Otto,

I remember from my Clipper-days, that commit was slowing down my program, especially on systems with Norton AntiVirus. So now, I only do a commit if I quit my software, never during the use of my application.

I also changed the errsysw.prg to be sure a commit is executed, even if an error occurs.

I use comit this way already for 10 years without any problem.

[Otto]

Hello Michael,

>So now, I only do a commit if I quit my software, never during the use of my application.
In some cases this could be dangerous.
In dbase III times I added the whole day data without commiting. In the evening I had a breakdown of the system and all the data was lost. Since that I always use lots of commits.

I think also clipper only writes to the disk if there is a certain volume of data reached.
Also in a multiuser environment you have to commit that the other users see the changes.

Best regards,
Otto

[Otto]

Hello Enrico,

I tested on the XP system with the same result as on the VISTA PC.
Astonishing that you don’t hear the harddisk when clipper is commiting but you hear every commit if you use the xHarbour test.exe.

Maybe COMMIT from my Clipper installation is not working.

Best regards,
Otto

[Enrico Maria Giordano]

Enrico your test does not show much difference. Did you do your tests under VISTA.

No, Windows XP Pro SP3.

I use 5.2 and COMIX as a add on.

It seems that COMMIT of Clipper 5.2e doesn't really write on disk. I just made the test using that version and it was almost instantaneous.

Enrico would you be so kind to email me your Clipper and xharbour test.exe

It seems that it is no more needed, isn't it?

EMG

[xProgrammer]

Hi Otto, Enrico

Its looks as though my suspicion was correct - that what Clipper is doing in terms of a COMMIT is less than what xHarbour is doing - at least in some environments.

There can be many layers of cache, in both software and hardware, between a programmatic instruction to write to a file and a physical write. This will vary based on OS, hardware drivers, controller, hard disk etc. It looks as though, in Otto's environment at least, xHarbour's COMMIT is forcing a physical disk write and Clipper isn't.

Forcing a disk write across a network connection (is that the case Otto?) might be slow if the COMMIT waits for the physical write to have occurred. I wonder does it?

If the performance hit is too great I guess you have to either forgo the COMMIT or change the database technology you are using. If you don't COMMIT you should only lose recently "written" data. When you lost a whole day's worth I suspect that there must have been some other problem.

Regards
Doug

[Rick Lipkin]

Otto

If I recall in the xHarbour NG ( years ago ) there was a discussion about this... Clipper really did not do a disk write in a Windows environment and the xHarbour developers took some grief about incorporating a TRUE disk write for dbCommit()

I had a situation like yours where I was running thru a loop of 1000 records or more and doing a dbCommit() after each record lock or append .. when I upgraded to a newer version of xHarbour .. I noticed that process was taking a 'brutally' long time to run thru the same loop .. same code.

I went back and reviewed that code and instead of the dbCommit() after each update .. I used a goto recno() which some people say flush the buffer ( to disk ) since you move the record pointer .. and then do the dbCommit() after the loop is done ..

Hope that helps
Rick Lipkin

[Otto]

Hello Rick,

>I noticed that process was taking a 'brutally' long
The same situation was here and I found out that it was commit what consumed time – what is logical.
I thought that Clippers commit does as well a “real” commit and therefore I couldn’t explain the time differences.

Thank you again
Otto

[toninhofwi]

This is the answer to Harbour Dev List from Przemyslaw Czerpak:

dbCommit() make two things:
1. write application memory buffers to file.
2. send to OS request to flush disk buffers releated to open file.

The 1-st action is executed also by any other rdd operation which
may cause record reloading or may need to synchronize modifications,
f.e. unlock. You can simulate it in many different ways, f.e. by skip(0).

The 2-nd action is unique to dbcommit and it's not necessary for any
synchronizations in simultaneous/concurrent/network access. The whole
job here is sending information to Operating System which is automatically
redirected if necessary by OS to File Server that we ask to flush OS or
FS disk I/O write buffers physically to disk. OS / FS can ignore our request,
can execute it immediately or can only mark that it should be done
in some nearest future. It's in practice out of programmer control and
does not have to be.
In [x]Harbour you can disable this part of dbCommit() by:
set( _SET_HARDCOMMIT, .F. )
In COMIX by:
cmxSys( 1002, .f. )
As I said it does not cause any interactions to concurrent access.
Try to add set( _SET_HARDCOMMIT, .F. ) to above code and check the results.
Then write a message to your OS / network client / file server authors
and ask why COMMIT request executed from DOS application makes sth different
then executed from real Windows application. Of course if you need such
information. Probably DOS emulation layer buffers few commit requests in
some short time period, f.e. 1 sec. and then send them as one. But I only
guess. Anyhow it's not Harbour problem. Harbour only sends commit request
for open file handle to the OS and this is single function call inside
each user dbCommit() if _SET_HARDCOMMIT is not disabled. All time overhead
if any is out of Harbour code.

[xProgrammer]

Thanks for the info. Now we all know why and whatwe can do about it.

Regards

Doug