A simple LLM using Harbour's hashes

A simple LLM using Harbour's hashes

Postby Antonio Linares » Tue May 16, 2023 6:03 am

An initial LLM ("large language model") using Harbour hashes:

llm.prg
Code: Select all  Expand view
static hWords := {=>}

function Main()

   AddSentence( hWords, hb_ATokens( "Hello my friend, Hello how are you ? Hello" ) )

   ? hWords[ "Hello" ][ 2 ][ "how" ]

return nil

function AddSentence( hWords, aWords )

   local cWord
   
   for each cWord in aWords
      if ! hb_HHasKey( hWords, cWord )
         hWords[ cWord ] = { 1, {=>} }
      else
         hWords[ cWord ][ 1 ]++
      endif
      AddSentence( hWords[ cWord ][ 2 ], SubArray( aWords, HB_EnumIndex() + 1 ) )
   next
   
return nil  

function SubArray( aArray, nStart )

   local aSub := {}
   
   for n = nStart to Len( aArray )
      AAdd( aSub, aArray[ n ] )
   next
   
return aSub  
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
 
Posts: 42065
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain

Re: A simple LLM using Harbour's hashes

Postby Otto » Tue May 16, 2023 6:46 am

Dear Antonio,
large language model

Can this model also be used with the copy & paste method to submit larger amounts of text to the ChatGPT prompt?

Is there any way to bypass the token limit of approximately 4000 per request here?

I haven't figured out yet if, when I'm within a chat and I post multiple "knowledge bases" there, all of them are considered, and I can input more text than 4000 tokens.

Also, I'm uncertain if, when I later open a chat, all the information will still be there?

Best regards,
Otto

Image
********************************************************************
mod harbour - Vamos a la conquista de la Web
modharbour.org
https://www.facebook.com/groups/modharbour.club
********************************************************************
User avatar
Otto
 
Posts: 6312
Joined: Fri Oct 07, 2005 7:07 pm

Re: A simple LLM using Harbour's hashes

Postby Antonio Linares » Tue May 16, 2023 6:52 am

Enhanced example:

Code: Select all  Expand view
#xcommand TEXT <into:TO,INTO> <v> => #pragma __cstream|<v>:=%s

static hWords := {=>}

function Main()

   local cText := GetText()
   local aSentences := hb_ATokens( cText, "." )
   local cSentence, cKey

   for each cSentence in aSentences
      AddSentence( hWords, hb_ATokens( cSentence ) )
   next      

   for each cKey in hb_HKeys( hWords )
      ? cKey
   next  

return nil

function AddSentence( hWords, aWords )

   local cWord
   
   for each cWord in aWords
      if ! hb_HHasKey( hWords, cWord )
         hWords[ cWord ] = { 1, {=>} }
      else
         hWords[ cWord ][ 1 ]++
      endif
      AddSentence( hWords[ cWord ][ 2 ], SubArray( aWords, HB_EnumIndex() + 1 ) )
   next
   
return nil  

function SubArray( aArray, nStart )

   local aSub := {}
   
   for n = nStart to Len( aArray )
      AAdd( aSub, aArray[ n ] )
   next
   
return aSub

function GetText()

   local cText
   
   TEXT INTO cText
Hello my friend, how are you ? Nice to meet you again. I hope that you are fine.
   ENDTEXT

return cText
 
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
 
Posts: 42065
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain

Re: A simple LLM using Harbour's hashes

Postby Antonio Linares » Tue May 16, 2023 6:55 am

Dear Otto,

This is just a very simple example to understand how to build a very simple LLM
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
 
Posts: 42065
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain

Re: A simple LLM using Harbour's hashes

Postby Otto » Tue May 16, 2023 7:38 am

Dear Antonio,

Here, with a passing knowledge base, you see, I ask the same question twice.
By sending the knowledge base, I also receive a perfect response.

What I would like is to send the entire FIVEWIN documentation, but it is much larger than the 4000 tokens.

Best regards,
Otto

Image

Image
********************************************************************
mod harbour - Vamos a la conquista de la Web
modharbour.org
https://www.facebook.com/groups/modharbour.club
********************************************************************
User avatar
Otto
 
Posts: 6312
Joined: Fri Oct 07, 2005 7:07 pm

Re: A simple LLM using Harbour's hashes

Postby Otto » Tue May 16, 2023 7:42 am

Dear Antonio,
Somewhere I read that it is possible to link a web page as knowledge base.
Maybe someone knows how to do it.
Then we could have all the documentation linked in a single page.

Best regards,
Otto
********************************************************************
mod harbour - Vamos a la conquista de la Web
modharbour.org
https://www.facebook.com/groups/modharbour.club
********************************************************************
User avatar
Otto
 
Posts: 6312
Joined: Fri Oct 07, 2005 7:07 pm

Re: A simple LLM using Harbour's hashes

Postby Antonio Linares » Wed May 17, 2023 3:30 pm

Enhanced version thanks to Mr. Rao

Code: Select all  Expand view
#xcommand TEXT <into:TO,INTO> <v> => #pragma __cstream|<v>:=%s

static hWords := {=>}

function Main()

   local cText := GetText()
   local aSentences := hb_ATokens( cText, "." )
   local cSentence, cKey

   for each cSentence in aSentences
      AddSentence( hWords, hb_ATokens( cSentence ) )
   next      

   ? hWords

return nil

function AddSentence( hWords, aWords, nStart )

   local cWord, n

   if nStart == nil
      nStart = 1
   endif  

   for n := nStart to Len( aWords ) //each cWord in aWords
      cWord    := aWords[ n ]
      if ! hb_HHasKey( hWords, cWord )
         hWords[ cWord ] = { 1, {=>} }
      else
         hWords[ cWord ][ 1 ]++
      endif
      if n < Len( aWords )
         AddSentence( hWords[ cWord ][ 2 ], aWords, n + 1 )
      endif
   next

return nil

function GetText()

   local cText
   
   TEXT INTO cText
Hello my friend ? How are you ? Nice to meet you again.
   ENDTEXT

return cText

 
regards, saludos

Antonio Linares
www.fivetechsoft.com
User avatar
Antonio Linares
Site Admin
 
Posts: 42065
Joined: Thu Oct 06, 2005 5:47 pm
Location: Spain


Return to FiveWin for Harbour/xHarbour

Who is online

Users browsing this forum: No registered users and 63 guests