A simple LLM using Harbour's hashes

[Antonio Linares]

An initial LLM ("large language model") using Harbour hashes:

llm.prg

Code: Select all  Expand view

static hWords := {=>}

function Main()

   AddSentence( hWords, hb_ATokens( "Hello my friend, Hello how are you ? Hello" ) )

   ? hWords[ "Hello" ][ 2 ][ "how" ]

return nil

function AddSentence( hWords, aWords )

   local cWord
   
   for each cWord in aWords
      if ! hb_HHasKey( hWords, cWord )
         hWords[ cWord ] = { 1, {=>} }
      else
         hWords[ cWord ][ 1 ]++
      endif
      AddSentence( hWords[ cWord ][ 2 ], SubArray( aWords, HB_EnumIndex() + 1 ) )
   next
   
return nil  

function SubArray( aArray, nStart )

   local aSub := {}
   
   for n = nStart to Len( aArray )
      AAdd( aSub, aArray[ n ] )
   next
   
return aSub  

[Otto]

Dear Antonio,
large language model

Can this model also be used with the copy & paste method to submit larger amounts of text to the ChatGPT prompt?

Is there any way to bypass the token limit of approximately 4000 per request here?

I haven't figured out yet if, when I'm within a chat and I post multiple "knowledge bases" there, all of them are considered, and I can input more text than 4000 tokens.

Also, I'm uncertain if, when I later open a chat, all the information will still be there?

Best regards,
Otto

[Antonio Linares]

Enhanced example:

Code: Select all  Expand view

#xcommand TEXT <into:TO,INTO> <v> => #pragma __cstream|<v>:=%s

static hWords := {=>}

function Main()

   local cText := GetText()
   local aSentences := hb_ATokens( cText, "." )
   local cSentence, cKey

   for each cSentence in aSentences
      AddSentence( hWords, hb_ATokens( cSentence ) )
   next      

   for each cKey in hb_HKeys( hWords )
      ? cKey
   next  

return nil

function AddSentence( hWords, aWords )

   local cWord
   
   for each cWord in aWords
      if ! hb_HHasKey( hWords, cWord )
         hWords[ cWord ] = { 1, {=>} }
      else
         hWords[ cWord ][ 1 ]++
      endif
      AddSentence( hWords[ cWord ][ 2 ], SubArray( aWords, HB_EnumIndex() + 1 ) )
   next
   
return nil  

function SubArray( aArray, nStart )

   local aSub := {}
   
   for n = nStart to Len( aArray )
      AAdd( aSub, aArray[ n ] )
   next
   
return aSub

function GetText()

   local cText
   
   TEXT INTO cText
Hello my friend, how are you ? Nice to meet you again. I hope that you are fine.
   ENDTEXT

return cText
 

[Antonio Linares]

Dear Otto,

This is just a very simple example to understand how to build a very simple LLM

[Otto]

Dear Antonio,

Here, with a passing knowledge base, you see, I ask the same question twice.
By sending the knowledge base, I also receive a perfect response.

What I would like is to send the entire FIVEWIN documentation, but it is much larger than the 4000 tokens.

Best regards,
Otto

[Otto]

Dear Antonio,
Somewhere I read that it is possible to link a web page as knowledge base.
Maybe someone knows how to do it.
Then we could have all the documentation linked in a single page.

Best regards,
Otto

[Antonio Linares]

Enhanced version thanks to Mr. Rao

Code: Select all  Expand view

#xcommand TEXT <into:TO,INTO> <v> => #pragma __cstream|<v>:=%s

static hWords := {=>}

function Main()

   local cText := GetText()
   local aSentences := hb_ATokens( cText, "." )
   local cSentence, cKey

   for each cSentence in aSentences
      AddSentence( hWords, hb_ATokens( cSentence ) )
   next      

   ? hWords

return nil

function AddSentence( hWords, aWords, nStart )

   local cWord, n

   if nStart == nil
      nStart = 1
   endif  

   for n := nStart to Len( aWords ) //each cWord in aWords
      cWord    := aWords[ n ]
      if ! hb_HHasKey( hWords, cWord )
         hWords[ cWord ] = { 1, {=>} }
      else
         hWords[ cWord ][ 1 ]++
      endif
      if n < Len( aWords )
         AddSentence( hWords[ cWord ][ 2 ], aWords, n + 1 )
      endif
   next

return nil

function GetText()

   local cText
   
   TEXT INTO cText
Hello my friend ? How are you ? Nice to meet you again.
   ENDTEXT

return cText