Page 1 of 1
A simple LLM using Harbour's hashes
Posted: Tue May 16, 2023 6:03 am
by Antonio Linares
An initial LLM ("large language model") using Harbour hashes:
llm.prg
Code: Select all | Expand
static hWords := {=>}
function Main()
AddSentence( hWords, hb_ATokens( "Hello my friend, Hello how are you ? Hello" ) )
? hWords[ "Hello" ][ 2 ][ "how" ]
return nil
function AddSentence( hWords, aWords )
local cWord
for each cWord in aWords
if ! hb_HHasKey( hWords, cWord )
hWords[ cWord ] = { 1, {=>} }
else
hWords[ cWord ][ 1 ]++
endif
AddSentence( hWords[ cWord ][ 2 ], SubArray( aWords, HB_EnumIndex() + 1 ) )
next
return nil
function SubArray( aArray, nStart )
local aSub := {}
for n = nStart to Len( aArray )
AAdd( aSub, aArray[ n ] )
next
return aSub
Re: A simple LLM using Harbour's hashes
Posted: Tue May 16, 2023 6:46 am
by Otto
Dear Antonio,
large language model
Can this model also be used with the copy & paste method to submit larger amounts of text to the ChatGPT prompt?
Is there any way to bypass the token limit of approximately 4000 per request here?
I haven't figured out yet if, when I'm within a chat and I post multiple "knowledge bases" there, all of them are considered, and I can input more text than 4000 tokens.
Also, I'm uncertain if, when I later open a chat, all the information will still be there?
Best regards,
Otto
Re: A simple LLM using Harbour's hashes
Posted: Tue May 16, 2023 6:52 am
by Antonio Linares
Enhanced example:
Code: Select all | Expand
#xcommand TEXT <into:TO,INTO> <v> => #pragma __cstream|<v>:=%s
static hWords := {=>}
function Main()
local cText := GetText()
local aSentences := hb_ATokens( cText, "." )
local cSentence, cKey
for each cSentence in aSentences
AddSentence( hWords, hb_ATokens( cSentence ) )
next
for each cKey in hb_HKeys( hWords )
? cKey
next
return nil
function AddSentence( hWords, aWords )
local cWord
for each cWord in aWords
if ! hb_HHasKey( hWords, cWord )
hWords[ cWord ] = { 1, {=>} }
else
hWords[ cWord ][ 1 ]++
endif
AddSentence( hWords[ cWord ][ 2 ], SubArray( aWords, HB_EnumIndex() + 1 ) )
next
return nil
function SubArray( aArray, nStart )
local aSub := {}
for n = nStart to Len( aArray )
AAdd( aSub, aArray[ n ] )
next
return aSub
function GetText()
local cText
TEXT INTO cText
Hello my friend, how are you ? Nice to meet you again. I hope that you are fine.
ENDTEXT
return cText
Re: A simple LLM using Harbour's hashes
Posted: Tue May 16, 2023 6:55 am
by Antonio Linares
Dear Otto,
This is just a very simple example to understand how to build a very simple LLM
Re: A simple LLM using Harbour's hashes
Posted: Tue May 16, 2023 7:38 am
by Otto
Dear Antonio,
Here, with a passing knowledge base, you see, I ask the same question twice.
By sending the knowledge base, I also receive a perfect response.
What I would like is to send the entire FIVEWIN documentation, but it is much larger than the 4000 tokens.
Best regards,
Otto
Re: A simple LLM using Harbour's hashes
Posted: Tue May 16, 2023 7:42 am
by Otto
Dear Antonio,
Somewhere I read that it is possible to link a web page as knowledge base.
Maybe someone knows how to do it.
Then we could have all the documentation linked in a single page.
Best regards,
Otto
Re: A simple LLM using Harbour's hashes
Posted: Wed May 17, 2023 3:30 pm
by Antonio Linares
Enhanced version thanks to Mr. Rao
Code: Select all | Expand
#xcommand TEXT <into:TO,INTO> <v> => #pragma __cstream|<v>:=%s
static hWords := {=>}
function Main()
local cText := GetText()
local aSentences := hb_ATokens( cText, "." )
local cSentence, cKey
for each cSentence in aSentences
AddSentence( hWords, hb_ATokens( cSentence ) )
next
? hWords
return nil
function AddSentence( hWords, aWords, nStart )
local cWord, n
if nStart == nil
nStart = 1
endif
for n := nStart to Len( aWords ) //each cWord in aWords
cWord := aWords[ n ]
if ! hb_HHasKey( hWords, cWord )
hWords[ cWord ] = { 1, {=>} }
else
hWords[ cWord ][ 1 ]++
endif
if n < Len( aWords )
AddSentence( hWords[ cWord ][ 2 ], aWords, n + 1 )
endif
next
return nil
function GetText()
local cText
TEXT INTO cText
Hello my friend ? How are you ? Nice to meet you again.
ENDTEXT
return cText