DBF . Commit

[nageswaragunupudi]

I think it's important to distinguish between the record buffer and the hard drive cache.

Yes.

But the documentation says it "hardwrites", same way like "commit".
In any case, (x)Harbour can flush its buffers to the OS buffers and but I doubt if (x)Harbour has an control over when the OS physically writes to the disk. But we can normally trust the OS on this issue even in case of unexpected power failures and shut-downs.

But for the purpose of visibility to all users, flushing harbour dbf buffers to OS is enough.

[Enrico Maria Giordano]

Mr. Enrico

We all accept that you are an authority on Harbour and more on xHarbour.

But the documentation I reproduced here is xHarbour documentation as it is. (This documentation was released by xhb.com)



Do you mean this documentation is wrong?

Try the following example with or without COMMIT and with or without SET HARDCOMMIT OFF (it is a pure console example):

Code: Select all  Expand view  RUN

REQUEST DBFCDX


FUNCTION MAIN()

    LOCAL cDbf := "MYTEST.DBF"

    RDDSETDEFAULT( "DBFCDX" )

    SET EXCLUSIVE OFF

//    SET HARDCOMMIT OFF

    DBCREATE( cDbf, { { "TEST", "C", 30, 0 } } )

    ? FILEDATE( cDbf ), FILETIME( cDbf )

    INKEY( 2 )

    USE ( cDbf )

    APPEND BLANK

    REPLACE FIELD -> test WITH "TEST"

//    COMMIT

    ? FILEDATE( cDbf ), FILETIME( cDbf )

    INKEY( 0 )

    RETURN NIL

[nageswaragunupudi]

Try the following example with or without COMMIT and with or without SET HARDCOMMIT OFF (it is a pure console example):

YES

This is what I was trying to submit to our learned colleagues.

COMMIT or NO COMMIT
SET HARDCOMMIT ON OR OFF

Simply changing the data and if shared executing UNLOCK is all that is enough to ensure visibility of data as well as indexes to all other users on the network immediately


So, please keep aside the documentation and STOP using COMMIT.

Please Note:
I am saying this with confidence after testing this some years and back and again today in network.
I will post a sample to test this on your networks.

[nageswaragunupudi]

I did similar tests long time back on configurations with Network Server and several clients,
Today I do not have access to such networks. I tested on my home network with two PCs connected over WIFI to my router.

I have used our familiar dbf "customer.dbf".
I request other users having access to small of large networks to do this test and post the results for the benefit of all other users.

First make sure the Server (or one of the PCs) has customer.dbf.
Because we want to test the flushing of index butters also, let us create and use customer.cdx.

First run this program to make sure we are using a proper cdx file.

prepare.prg

Code: Select all  Expand view  RUN

#include "fivewin.ch"

REQUEST DBFCDX

function Main()

   local cPath := ""  // network path

   ? "Re-creating index"

   FERASE( cPath + "CUSTOMER.CDX" )
   USE ( cPath + "CUSTOMER" ) EXCLUSIVE VIA "DBFCDX"
   FW_CdxCreate()
   CLOSE DATA

   ? "Finish"

return nil

Please assign your network path of the folder containing the customer.dbf to the "local cPath :="
Then build and run the program.

Now I am giving (1) writer.prg and (2) listner.prg

writer.prg:

Code: Select all  Expand view  RUN

#include "fivewin.ch"
#include "dbinfo.ch"
#include "set.ch"

REQUEST DBFCDX

function Main()

   field CITY
   local oDlg, oFont, oGet
   local cPath    := ""  // set your network path
   local cSave

   USE ( cPath + "CUSTOMER" ) NEW SHARED VIA "DBFCDX"
   GOTO 100
   cSave    := CITY

   //
   DEFINE FONT oFont NAME "TAHOMA" SIZE 0,-16
   DEFINE DIALOG oDlg SIZE 400,180 PIXEL TRUEPIXEL FONT oFont  ;
      TITLE "CUSTOMER(100)"
   RELEASE FONT oFont

   @  40, 20 SAY "CITY : " GET oGet VAR FIELD->CITY SIZE 300,30 PIXEL OF oDlg ;
      WHEN ( DBRLOCK() ) VALID ( oDlg:Update(), .t. ) UPDATE

   @  80, 20 SAY "OrdKeyNo : " + cValToChar( OrdKeyNo( "CITY" ) ) ;
      SIZE 200,30 PIXEL OF oDlg UPDATE

   @ 140, 20 BUTTON "Unlock" SIZE 140,30 PIXEL OF oDlg ;
      ACTION ( DBRUNLOCK(), oDlg:Update(), oGet:SetFocus() )

   @ 140,280 BUTTON "Close" SIZE 100,30 PIXEL OF oDlg ;
      ACTION ( DBRUNLOCK(), oDlg:End() )

   ACTIVATE DIALOG oDlg CENTERED ;
      VALID (DBRLOCK(), CITY := cSave, DBRUNLOCK(), .t. )

return nil

Again assign to "local cPath" your network path the folder containing customer.dbf
Please note that we are using DBRUNLOCK() only to save the changes and not using COMMIT anywhere

listner.prg

Code: Select all  Expand view  RUN

#include "fivewin.ch"

REQUEST DBFCDX

function Main()

   field CITY
   local oDlg, oFont, oSay, oTimer
   local cPath    := ""  // set your network path
   local cSave

   USE ( cPath + "CUSTOMER" ) NEW SHARED VIA "DBFCDX"
   GOTO 100
   cSave    := CITY

   //
   DEFINE FONT oFont NAME "TAHOMA" SIZE 0,-16
   DEFINE DIALOG oDlg SIZE 300,180 PIXEL TRUEPIXEL FONT oFont  ;
      TITLE "CUSTOMER(100)"
   RELEASE FONT oFont

   @  40, 20 SAY "City : " + CITY + CRLF + ;
            "OrdKeyNo : " + cValToChar( OrdKeyNo( "CITY" ) ) + ;
            + CRLF + "Time : " + TIME() ;
           SIZE 260,80 PIXEL OF oDlg UPDATE

   @ 140, 20 BUTTON "Close" SIZE 80,30 PIXEL OF oDlg ;
            ACTION oDlg:End()

   oDlg:bInit := <||
      DEFINE TIMER oTimer OF oDlg INTERVAL 10 ;
      ACTION ( DBSKIP( 0 ), If( CITY == cSave,, ( cSave := CITY, oDlg:Update() ) ) )
      ACTIVATE TIMER oTimer
      return nil
      >

   ACTIVATE DIALOG oDlg CENTERED ;
      VALID ( oTimer:End(), .t. )

return nil

This program displays the values and keeps checking for changes in the data with a timer. The moment, changes are detected, the values are updated and displays with time.

Please also note that the OrdKeyNo() also is displayed. Changes in this value indicate that all index buffers are also flushed and visible to the other users on the network.

1) Build and run the "writer.prg" on one of the client PC or Server.
2) Build and run the "listner.prg" on all other clients in the network.

The moment you change the value and "press Unlock button", you will see the changes on all other clients on the network.

This is enough proof that COMMIT is not necessary.

Next:
Add this line

Code: Select all  Expand view  RUN

SET HARDCOMMIT OFF

to the Writer.prg and rebuild and run.
You will notice the behavior to be the same.

So,
COMMIT of NO COMMIT
SET HARDCOMMIT ON or OFF
Modify and UNLOCK is all that is enough.

[Enrico Maria Giordano]

This is enough proof that COMMIT is not necessary.

Probably COMMIT is not necessary for the visibility of data to other users across the network but it is necessary to write the physical data on the hard disk. In this, the documentation is wrong, as my example demonstrates.

[nageswaragunupudi]

but it is necessary to write the physical data on the hard disk

Kindly reconsider.

Unlike RDMSs, in the case of DBF, each client PC has its own buffer. In fact, each alias in the same application and each application on the same PC.

Unless the contents of the buffer are written to the disk buffers of the OS, other client PC buffers can not be refreshed. So we are clear that simple Unlock is flushing data and index buffers to the disk buffers of the Windows OS. Ultimately any RDD finally writes the data to the disk using fwrite (or similar function).

Harbour can write only to the file buffer of Windows OS, but can not force the OS to physically write to the disk. All that is handled by the Windows OS and we can trust the Windows OS to flush its own buffers to the physical disk safely even in case of unexpected shut downs except in extremely rare situations.

So, I believe that COMMIT does not do anything extra.

[Enrico Maria Giordano]

but it is necessary to write the physical data on the hard disk

Kindly reconsider.

There is nothing to reconsider. Did you try my example? It demonstrates that the DBF is not written without COMMIT. It is not necessary for the visibility, we agree on this.

[Otto]

Enrico, but if you try like Mr. Rao said your sample is working for me.


APPEND BLANK
dbRLock()
REPLACE FIELD ->test WITH "TEST"
DbUnlock()

Best regards,
Otto

Code: Select all  Expand view  RUN

#include "fivewin.ch"
REQUEST DBFCDX
REQUEST DBFFPT



FUNCTION MAIN()

    LOCAL cDbf := "MYTEST.DBF"

    RDDSETDEFAULT( "DBFCDX" )

   SET EXCLUSIVE OFF

//    SET HARDCOMMIT OFF

    DBCREATE( cDbf, { { "TEST", "C", 30, 0 } } )

    ? FILEDATE( cDbf ), FILETIME( cDbf )

    INKEY( 2 )

    USE ( cDbf ) new

       
    APPEND BLANK
     dbRLock()
    REPLACE FIELD ->test WITH "TEST"
    DbUnlock()
   
//    COMMIT

    ? FILEDATE( cDbf ), FILETIME( cDbf )

    inkey(0)

    RETURN
 

[Enrico Maria Giordano]

Enrico, but if you try like Mr. Rao said your sample is working for me.


APPEND BLANK
dbRLock()
REPLACE FIELD ->test WITH "TEST"
DbUnlock()

Best regards,
Otto

Code: Select all  Expand view  RUN

#include "fivewin.ch"
REQUEST DBFCDX
REQUEST DBFFPT



FUNCTION MAIN()

    LOCAL cDbf := "MYTEST.DBF"

    RDDSETDEFAULT( "DBFCDX" )

   SET EXCLUSIVE OFF

//    SET HARDCOMMIT OFF

    DBCREATE( cDbf, { { "TEST", "C", 30, 0 } } )

    ? FILEDATE( cDbf ), FILETIME( cDbf )

    INKEY( 2 )

    USE ( cDbf ) new

       
    APPEND BLANK
     dbRLock()
    REPLACE FIELD ->test WITH "TEST"
    DbUnlock()
   
//    COMMIT

    ? FILEDATE( cDbf ), FILETIME( cDbf )

    inkey(0)

    RETURN
 

Does not work for me. Can I see your output? Anyway, you mixed the R lock flavor and the normal lock flavor. With DBRLOCK() you must use DBRUNLOCK(). And furthermore, APPEND BLANK operation locks the newly appended record so there is no need of another lock.

[Otto]

Enrico,
I am glad that I brought up this topic.
At first, I thought it is a bit weird for an experienced programmer like me to talk about it.
But I see there are many opinions here.

I can only confirm what Mr. Rao says. If you insert UNLOCK into Enrico's code, then the replace is carried out.

I conduct my test as follows.
I open the database with EmagDbu238 while the MsgInfo is displayed. Then I see that "test" is in the database, although the timestamp of the file has not changed.
Without the use of UNLOCK, it does not work. Exactly as Mr. Rao writes.

Constantly committing stresses the hard drive, especially when using an HDD.
This topic became relevant when it was claimed in the mod harbour group that committing was necessary.

However, committing simply sends an additional request to Windows to write the data immediately.
Normally, Windows writes data as needed by the disk operating system.
It also slows down the system.

This is what I have found out.
Best regards,
Otto

[Enrico Maria Giordano]

Otto,

Please reread my posts. You missed the point.

[Otto]

Enrico,
what's not working is:
? FILEDATE( cDbf ), FILETIME( cDbf )
I made a PS script which I call from the test program and get the right time and values.

FOR I := 1 to 5000
APPEND BLANK
REPLACE FIELD ->test WITH "TEST"+ str(i) + time()
UNLOCK
if I = 5000
? FILEDATE( cDbf ), FILETIME( cDbf )
waitrun( 'powershell_ise.exe -file ".\ps_datumzeit.ps1"' )
endif


NEXT

Code: Select all  Expand view  RUN

# Definieren Sie den Pfad zur Datei
$filePath = "c:\fwh2023\samples\MYTEST19.DBF"

# Überprüfen Sie, ob die Datei existiert
if (Test-Path $filePath) {
    # Holen Sie sich die Dateiinformationen
    $fileInfo = Get-Item $filePath
   
    # Drucken Sie die Dateigröße in Bytes
    Write-Host "Dateigröße: $($fileInfo.Length) Bytes"
   
    # Drucken Sie das letzte Änderungsdatum der Datei
    Write-Host "Letztes Änderungsdatum: $($fileInfo.LastWriteTime)"
} else {
    Write-Host "Die Datei '$filePath' existiert nicht."
}

 

I think we need an extended function for these:
To retrieve file size, date, and time information in C++ similarly to how Windows Explorer does, you can use the Windows API. The GetFileAttributesEx function can be used to obtain extended file attributes, including the size, and the FileTimeToLocalFileTime.

Best regards,
Otto

[nageswaragunupudi]

There is nothing to reconsider. Did you try my example? It demonstrates that the DBF is not written without COMMIT. It is not necessary for the visibility, we agree on this.

Thanks a lot Mr. Enrico.
Sorry, I did not know about this before.
Yes, I see the difference in the timestamps of the dbf file (also indexes and memo files, if exist)

Just for information to all:
COMMIT executes the WinAPI function FlushFileBuffers( handle ) thereby ensuring that all the data in the buffers are physically written to the disk. This is executed for dbf, fpt/dbt and all cdx/ntx/idx file handles.
This API function also updates the files' timestamps.
(note: for other operating systems (x)Harbour executes other similar functions)

Please note that there is a similar function HB_FCOMMIT( <filehandle> ) function available to us when we are using raw file functions like fCreate(), fWrite() etc. We can use hb_fcommit(handle) to ensure flushing all buffers to disk.

Mr. Enrico was clear. He said "It is not necessary for the visibility, we agree on this."

So, let is be clear on this:
1. COMMIT is safe and ensures all data is physically written the disk. But not necessary to make the changes visible to other users.
2. UNLOCK is all that is enough to make the changes visible to users across the network.

Hope Mr. Enrico agrees on this.

But commit is costly in terms of performance. In my view, it is always not necessary use COMMIT with every change of data.

Please also see:
https://learn.microsoft.com/en-us/windo ... ilebuffers

[nageswaragunupudi]

Next:
So, what does this HARDCOMMIT setting do?
After going through some low level functions in (x)Harbour rdd like:

Code: Select all  Expand view  RUN

static HB_ERRCODE hb_dbfFlush( DBFAREAP pArea )
{
   // ... some code ...

   if( hb_setGetHardCommit() && errCode == HB_SUCCESS )
   {
      if( pArea->fDataFlush )
      {
         hb_fileCommit( pArea->pDataFile );
         pArea->fDataFlush = FALSE;
      }

Whenever any data is changed pArea->fDataFlush is set to TRUE.
But this function is executed when

Code: Select all  Expand view  RUN

SELF_FLUSH( pArea )

is called and again this is called only CLOSE and COMMIT functions.

What does all this mean?

If SET HARDCOMMIT is ON, does every change when made is also "committed" ? We know that is not happening because we do not notice change in file timestamps.

Even when COMMIT calls this function, this function in turn is only executing hb_fileCommit() only when hb_SetGetHardCommit() is true. Does it mean that when HARDCOMMIT is OFF, does not COMMIT hb_FileCommit() ?

So, for COMMIT to function as expected, set HARDCOMMIT needs to be set ON.
We can confirm this by executing Mr. Enrico's sample by setting this to OFF

[Otto]

Dear Mr. Rao,
thank you.

Can you show me how to add GetFileAttributesEx from WINDOWS API.
Thank you in advance and best regards
Otto

Code: Select all  Expand view  RUN

To retrieve file size, date, and time information in C++ similarly to how Windows Explorer does, you can use the Windows API. The GetFileAttributesEx function can be used to obtain extended file attributes, including the size, and the FileTimeToLocalFileTime along with FileTimeToSystemTime functions can be used to convert the file time to a more human-readable format. Here's an example of how you might do this:

c++
Copy code
#include <windows.h>
#include <iostream>
#include <iomanip>

void DisplayFileInfo(const std::wstring& filePath) {
    WIN32_FILE_ATTRIBUTE_DATA fileAttributeData;

    if (GetFileAttributesEx(filePath.c_str(), GetFileExInfoStandard, &fileAttributeData)) {
        LARGE_INTEGER fileSize;
        fileSize.LowPart = fileAttributeData.nFileSizeLow;
        fileSize.HighPart = fileAttributeData.nFileSizeHigh;

        FILETIME localFileTime;
        SYSTEMTIME systemTime;

        // Convert the last write time to local time
        if (FileTimeToLocalFileTime(&fileAttributeData.ftLastWriteTime, &localFileTime) &&
            FileTimeToSystemTime(&localFileTime, &systemTime)) {
            std::wcout << L"File size: " << fileSize.QuadPart << L" bytes\n"
                       << L"Last modified: "
                       << systemTime.wYear << L'-'
                       << std::setw(2) << std::setfill(L'0') << systemTime.wMonth << L'-'
                       << std::setw(2) << std::setfill(L'0') << systemTime.wDay << L' '
                       << std::setw(2) << std::setfill(L'0') << systemTime.wHour << L':'
                       << std::setw(2) << std::setfill(L'0') << systemTime.wMinute << L':'
                       << std::setw(2) << std::setfill(L'0') << systemTime.wSecond
                       << std::endl;
        } else {
            std::wcerr << L"Failed to convert file time\n";
        }
    } else {
        std::wcerr << L"Failed to get file attributes\n";
    }
}

int main() {
    DisplayFileInfo(L"C:\\path\\to\\your\\file.txt");  // Replace with your file path
    return 0;
}
In this code:

The GetFileAttributesEx function is called to retrieve the file attributes of the specified file.
The file size is extracted from the nFileSizeLow and nFileSizeHigh members of the WIN32_FILE_ATTRIBUTE_DATA structure.
The FileTimeToLocalFileTime function is called to convert the file time from UTC to local time.
The FileTimeToSystemTime function is called to convert the file time to a SYSTEMTIME structure, which is then formatted and displayed.
Make sure to replace "C:\\path\\to\\your\\file.txt" with the actual path to the file you're interested in. This code should display the file size and last modified time in a format similar to how Windows Explorer does.