SubNtx

[Antonio Linares]

Is there any SubNtx and Clipper old user here ?

We are curious to do some speed tests, thanks

[Antonio Linares]

Please review this topic for the speed tests that we are doing:

http://forums.fivetechsoft.com/viewtopic.php?t=12620

[triumvirato]

Antonio,

Very high speed...

[Antonio Linares]

We have adapted Cesar A. Gil's source code to use it and test it from Harbour and xHarbour:

Here it is the first version. Please notice that we supply it a codeblock so we can do what we want with each NTX key. Next step is to create a new NTX file with the keys that match a certain condition:

test.prg

Code: Select all  Expand view

#include "FiveWin.ch"

FIELD First, Last

function Main()

   USE Customer VIA "DBFNTX"

   INDEX ON First+Last TO Customer
   SET INDEX TO
   
   SubNtx( "customer.ntx", { | nRecNo, cKey | Test( nRecNo, cKey ) } )

return nil

function Test( nRecNo, cKey ) // In this test we show the first 10 keys only

   static nTimes := 1
   
   if nTimes < 11
      MsgInfo( cKey, "Key: " + AllTrim( Str( nTimes++ ) ) + ;
               " --> RecNo: " + AllTrim( Str( nRecNo ) ) )
   endif
   
return nil     

#pragma BEGINDUMP

#include <hbapi.h>
#include <hbapifs.h>
#include <hbvm.h>

#define MAX_KEY  256
#define BUF_SIZE 1024

typedef struct
{
   unsigned short int     type;
   unsigned short int     version;
   long     root;
   long     next_page;
   unsigned short int     item_size;
   unsigned short int     key_size;
   unsigned short int     key_dec;
   unsigned short int     max_item;
   unsigned short int     half_page;
   char     key_expr[ MAX_KEY ];
   char     unique;
} NTX_HEADER;

typedef struct
{
   long page;
   long rec_no;
   char key[ 1 ];
} NTX_ITEM;

typedef struct
{
   unsigned short item_count;
   unsigned short item_offset[ 1 ];
} NTX_BUFFER;

static void ReadPage( int hFileIn, long page_offset, PHB_ITEM pCodeBlock, unsigned short int iKeySize )
{
   char ntxPage[ BUF_SIZE ];
    NTX_ITEM * pNtxItem;
    NTX_BUFFER * pNtxBuffer;
   int i;
   
   hb_fsSeek( hFileIn, page_offset, FS_SET );
   
    if( hb_fsRead( hFileIn, ntxPage, BUF_SIZE ) != BUF_SIZE )
       return;
      
    pNtxBuffer = ( NTX_BUFFER * ) ntxPage;   
      
    for( i = 0; i < pNtxBuffer->item_count; i ++ )
    {
       pNtxItem = ( NTX_ITEM * ) ( ntxPage + pNtxBuffer->item_offset[ i ] );

       if( pNtxItem->page )
           ReadPage( hFileIn, pNtxItem->page, pCodeBlock, iKeySize );
           
        hb_vmPushSymbol( &hb_symEval );
        hb_vmPush( pCodeBlock );
        hb_vmPushLong( pNtxItem->rec_no );
        hb_vmPushString( ( char * ) &pNtxItem->key, iKeySize );
        hb_vmFunction( 2 );
   }   

    pNtxItem = ( NTX_ITEM * ) ( ntxPage + pNtxBuffer->item_offset[ pNtxBuffer->item_count ] );
   
    if( pNtxItem->page )
       ReadPage( hFileIn, pNtxItem->page, pCodeBlock, iKeySize );
}

HB_FUNC( SUBNTX )
{
   NTX_HEADER ntx_header;   
   int hFileIn = hb_fsOpen( hb_parc( 1 ), FO_READ );
   PHB_ITEM pCodeBlock = hb_param( 2, HB_IT_BLOCK );

   if( hFileIn != -1 )
   {
      hb_fsRead( hFileIn, ( char * ) &ntx_header, sizeof( NTX_HEADER ) );
     
      ReadPage( hFileIn, ntx_header.root, pCodeBlock, ntx_header.key_size );

      hb_fsClose( hFileIn );   
   } 
}

#pragma ENDDUMP


[Maurizio]

Hello Antonio

a stupid question .
This works with DBFCDX to ?

MAurizio

[Antonio Linares]

Maurizio,

No. Actually it is only for NTX.

But we are curious to check if it could be implemented for CDX too

[StefanHaupt]

Antonio,

just to understand what you are doing.... (so maybe another stupid question )

What is SubNtx and what are the advantages using this function ?

[Antonio Linares]

Stefan,

SubNtx was a very popular Clipper tool that allows to perform incredibly fast queries in DBFs, providing the results in few seconds.

We have done some speed tests doing queries on large DBFs and SubNtx takes 3,5 seconds where Harbour/xHarbour takes almost a minute.

SubNtx basically opens the index itself and makes the search inside the index. It does not uses the RDD internal functions, neither the Harbour/xHarbour virtual machine. Just pure C at the lowest level. The results are really impressive

For example: Imagine that you have a DBF and you want to make a query for different values in different fields. Here is where SubNtx is clearly the fastest option.

[StefanHaupt]

Antonio,

that sounds very promising, thanks for the info