Fulltextsearch (to Marco Boschi)

[MarcoBoschi]

Otto,
I load an entire dbf into an Array using this function
...
LOCAL aClifor := {}
...
aClifor := load_dbf( "clifor" )


....
....
FUNCTION LOAD_DBF( cFile )
LOCAL aRecord := {}
LOCAL aFile := {}
LOCAL nFld
LOCAL nIni

SET DELETED OFF
SELECT 0
USE &cFile
GO TOP

nIni := SECONDS()

DO WHILE !EOF()
aRecord := {}
AADD( aRecord , IIF( DELETED() , "D", " " ) )
FOR nFld := 1 TO FCount()
// here if I remove condition I think I can include numeric
IF fieldtype( nFld ) = "C" .OR. fieldtype( nFld ) = "M"also
AADD( aRecord , UPPER(fieldget( nFld )) )
ENDIF
NEXT i

AADD( aFile , aRecord )
SKIP

ENDDO

SET DELETED ON

RETURN aFile



If I have to search in array call this function

sea_array( ALLTRIM("clifor"), aClifor, UPPER( field->words ) , field->progr )

at this point of the program I use cdx files created by function sea_array
// INDEX ON recno() TAG RICERCA CUSTOM TO &cCdxFile



SET INDEX TO &cIndSearch
SET ORDER TO TAG RICERCA



FUNCTION SEA_ARRAY( cFile , aFile, cCerca, cProgr )

LOCAL nPosLast
LOCAL nPos
LOCAL nFound
LOCAL iCerca
LOCAL nPos2
LOCAL nRecord
LOCAL n1 := SECONDS()
LOCAL aCerca := HB_ATokens( alltrim( cCerca ) , " " )
LOCAL aFound := {}
LOCAL nHandle

LOCAL cCdxFile := "search\" + cProgr

nPosLast := 0

DO WHILE .T.

nPos := ASCAN( aFile , { | aItem | ASCAN( aItem, { | cItem | aCerca[1] $ cItem } ) > 0 } , nPosLast + 1 )

IF nPos > 0
nFound := 1
FOR iCerca := 2 TO LEN( aCerca )
nPos2 := ASCAN( aFile[ nPos ] , { | cItem | aCerca[ iCerca ] $ cItem } )
IF nPos2 > 0
nFound ++
ENDIF
NEXT iCerca
IF nFound = LEN( aCerca )
IF aFile[ nPos , U_DELETE ] <> "D"
AADD( aFound , nPos )
ENDIF
ENDIF
nPosLast := nPos
ELSE
EXIT
ENDIF

ENDDO

ferase( cCdxFile + ".cdx" )

IF LEN( aFound ) > 0

SELECT &cFile

INDEX ON recno() TAG RICERCA CUSTOM TO &cCdxFile
FOR EACH nRecord In aFound
GOTO nRecord
OrdKeyAdd()
NEXT
SET INDEX TO
ELSE
nhandle := fcreate( cCdxFile + ".cdx" )
fclose( nHandle )
ENDIF


RETURN NIL

This is a client/server application
Server side I load into an array the content of dbf table at startup
At the end of this initial operation this app in polling mode wait for requests from clients.
A client makes a request to server that:
- perform search
- create index
the client when index is created use this index.


When a clients modifies a record of table (loaded into array) I update array in this way **

agg_rec( "clifor", aClifor, nRecno )


FUNCTION AGG_REC( cFile, aFile, nRecord )
LOCAL cDelete
LOCAL aRec := {}
LOCAL nFld, nNew
LOCAL nNuovi
LOCAL aNew := {}
LOCAL nStart := LEN( aFile )
LOCAL i
// se ci sono nuovi record prima crea i nuovi elementi dell'array vuoti
IF nRecord > LEN( aFile )

nNuovi := nRecord - LEN( aFile )

FOR nNew := 1 TO nNuovi
aRec := {}
FOR nFld := 1 TO FCount()
IF fieldtype( nFld ) = "C" .OR. fieldtype( nFld ) = "M"
AADD( aRec , "" )
ENDIF
NEXT i

AADD( aFile , aRec )
AADD( aNew , nStart + nNew )

NEXT nNew
ELSE
AADD( aNew , nRecord )
ENDIF

SELECT &cFile

FOR nNew := 1 TO LEN( aNew )

i := aNew[ nNew ]
GOTO aNew[ nNew ]

aFile[ i , U_DELETE ] := cDelete

aFile[ i , U_DELETE ] := " "
aRec := {}

AADD( aRec , IIF( DELETED() , "D", " " ) )
FOR nFld := 1 TO FCount()

IF fieldtype( nFld ) = "C" .OR. fieldtype( nFld ) = "M"
AADD( aRec , UPPER( fieldget( nFld ) ) )
ENDIF

NEXT i

FOR nFld := 1 TO LEN( aFile[ i ] )
IF aFile[ i , nFld ] <> aRec[ nFld ]
aFile[ i , nFld ] := aRec[ nFld ]
ENDIF
NEXT nFld

NEXT nNew

RETURN NIL

** I use another historical program that in real time update from main database (set of dbf tables). After each replace I write into a log file dbftable and record
Another server app read this registrations and replies anyu change in the secondary database

(You do not know how many times they saved my ass)

[StefanHaupt]

James,

Stefan,
Does this include memo fields?

I didn´t test it with memo fields so far, but I think, it should work. HSX uses its own index file, it has nothing to do with the other index files nor with any database relation.

[StefanHaupt]

James,

Stefan,

In your example what is the var HS, an object, an order number?

Can these indexes be part of a single index file and be kept up automatically just like any other CDX index.

Regards,
James

hs is the handle of the index file. It is a single index file, but it is not automatically updated. I´m creating a new index ervery time my search function is called. Nevertheless it´s very fast.

I use hsx this way

Code: Select all  Expand view

SELECT Kunden

hHSX := HS_Index ("Kunden","Vorname+Name+Strasse+Ort",2,0,,.t.,3)

IF MsgGet ("Erweiterte Kundensuche","Suchtext eingeben: (Vorname,Name,Strasse oder Ort)",@cSearch)
  nRec := Kunden->(RecNo())
  Kunden->(OrdSetFocus ("KundenName"))
  Kunden->(dbGoTop())
  cSearch := Alltrim (Lower(cSearch))
 
  cSearch := AnsiToOem (cSearch)

  HS_Set (hHSX, cSearch)
  WHILE (n := HS_Next (hHSX)) > 0
    //? n
    Kunden->(dbGoTo (n))
    IF HS_Verify (hHSX) > 0

       AADD (aFound, {OemToAnsi(Kunden->Name),OemToAnsi(Kunden->Vorname),;
                      OemToAnsi(Kunden->Strasse),OemToAnsi (Kunden->Ort),;
                      Kunden->(RecNo()) } )
    ENDIF
  ENDDO
  HS_Close (hHSX)
ENDIF
 

[StefanHaupt]

James,

I do wonder about what is required to switch from CDX indexes to HiperSix indexes. Is this going to require a lot of code changes due to differences in syntax

there is already a userrdd called hsxrdd included in xharbour. But it seems you can´t use the normal index functions. Here is a sample from the docs

Code: Select all  Expand view

#include "dbinfo.ch"

REQUEST HSXRDD

PROC MAIN()
   FIELD FIRST, LAST, STREET, CITY
   LOCAL n, hs

   rddSetDefault("HSXRDD")
   dbCreate("_tst", {{"FIRST",       "C", 20,  0},;
                     {"LAST",        "C", 20,  0},;
                     {"STREET",      "C", 30,  0},;
                     {"CITY",        "C", 30,  0},;
                     {"STATE",       "C",  2,  0},;
                     {"ZIP",         "C", 10,  0},;
                     {"HIREDATE",    "D",  8,  0},;
                     {"MARRIED",     "L",  1,  0},;
                     {"AGE",         "N",  2,  0},;
                     {"SALARY",      "N",  6,  0},;
                     {"NOTES",       "C", 70,  0}})
   USE _tst
   HSX_CREATE( "_tst", "FIRST+LAST+STREET+CITY", 2, 0, .T., 3 )
   APPEND FROM test

   /* Look for all records which have 'SHERMAN' string inside */
   hs := HSX_HANDLE( "_tst" )
   HS_SET( hs, "SHERMAN" )
   WHILE ( n := HS_NEXT( hs ) ) > 0
      DBGOTO( n )
      IF HS_VERIFY( hs ) > 0
         ? RTRIM( FIRST+LAST+STREET+CITY )
      ENDIF
   ENDDO
   WAIT

   /* Does RDD support Record Map Filters? */
   IF DBINFO( DBI_RM_SUPPORTED )
      /* if yest then let set filter for all records with 'SHERMAN'
         word and look at them in browser */
      HS_FILTER( hs, "SHERMAN" )
      DBGOTOP()
      BROWSE()
   ENDIF
RETURN
 

[James Bott]

Otto,

Have you been able to get a full-text search system going yet?

Regards,
James

[Otto]

Hello James,

here is a screenshot from the application I would like to program.

Best regards,
Otto

[James Bott]

Otto,

You just want to search those tiny memo fields? How many records are there (roughly)?

I would think just doing a DO WHILE ! EOF() would be sufficient as long as there are not too many records.

Regards,
James