Using Microsoft AI Phi-2 from FWH

[Antonio Linares]

See the speed, really fine!

This will be included in next FWH build. If you need it before, please send me an email

You don't have to pay OpenAI to have AI on your FWH apps

[Antonio Linares]

Really easy to be used from our FWH apps:

Code: Select all  Expand view  RUN

#include "Fivewin.ch"

function Main()

   local oDlg, cPrompt := PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
   local cAnswer := "", oAnswer, oBtn

   DEFINE DIALOG oDlg SIZE 700, 500 TITLE "FWH AI"

   @ 1, 1 GET cPrompt SIZE 300, 15

   @ 3, 1 GET oAnswer VAR cAnswer MULTILINE SIZE 300, 200

   @ 0.7, 52.5 BUTTON oBtn PROMPT "start" ;
      ACTION ( oBtn:Disable(), Llama( "phi-2_Q4_K_M.gguf", RTrim( cPrompt ),;
               CallBack( { | cStr | oAnswer:SetFocus(), oAnswer:Append( cStr ) } ) ),;
               oBtn:Enable(), oBtn:SetFocus() )

   @ 2.2, 52.5 BUTTON "Clear" ACTION oAnswer:SetText( "" )

   ACTIVATE DIALOG oDlg CENTERED

return nil

DLL FUNCTION Llama( cModel AS LPSTR, cPrompt AS LPSTR, pFunc AS PTR ) AS VOID PASCAL LIB "llama64.dll"

[Jimmy]

hi Antomio,

where is ""llama64.dll" come from

[Antonio Linares]

Dear Jimmy,

For most of GGUF files:
https://github.com/ggerganov/llama.cpp

[Antonio Linares]

Dear Jimmy and friends,

Here you have llama64.dll:
https://github.com/FiveTechSoft/FWH_tools/blob/master/llama64.dll

It has been tested using these models. We recommend Phi-2 as it is the smallest and better. Every few days are appearing new and enhanced models

Microsoft Phi-2
https://huggingface.co/kroonen/phi-2-GGUF/resolve/main/phi-2_Q4_K_M.gguf?download=true

Microsoft Orca2
https://huggingface.co/TheBloke/Orca-2-7B-GGUF/resolve/main/orca-2-7b.Q2_K.gguf?download=true

Here you have a working example:

Code: Select all  Expand view  RUN

#include "Fivewin.ch"

function Main()

   local oDlg, cPrompt := PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
   local cAnswer := "", oAnswer, oBtn

   DEFINE DIALOG oDlg SIZE 700, 500 TITLE "FWH AI"

   @ 1, 1 GET cPrompt MULTILINE SIZE 300, 15

   @ 3, 1 GET oAnswer VAR cAnswer MULTILINE SIZE 300, 200

   @ 0.7, 52.5 BUTTON oBtn PROMPT "start" ;
      ACTION ( oBtn:Disable(), Llama( "phi-2_Q4_K_M.gguf", RTrim( cPrompt ),;
               CallBack( { | cStr | oAnswer:SetFocus(), oAnswer:Append( cStr ) } ) ),;
               oBtn:Enable(), oBtn:SetFocus() )

   @ 2.2, 52.5 BUTTON "Clear" ACTION oAnswer:SetText( "" )

   ACTIVATE DIALOG oDlg CENTERED

return nil

DLL FUNCTION Llama( cModel AS LPSTR, cPrompt AS LPSTR, pFunc AS PTR ) AS VOID PASCAL LIB "llama64.dll"

#pragma BEGINDUMP

#include <Windows.h>
#include <hbapi.h>
#include <hbapiitm.h>

static PHB_ITEM pBlock;

static void callback( char * szMsg )
{
   PHB_ITEM pStr = hb_itemPutC( NULL, szMsg );
   hb_evalBlock1( pBlock, pStr );
   hb_itemRelease( pStr );
}

HB_FUNC( CALLBACK )
{
   pBlock = hb_gcGripGet( hb_param( 1, HB_IT_BLOCK ) );
   hb_retnll( ( HB_LONGLONG ) callback );
}

#pragma ENDDUMP

Unfortunately the llama32.dll can't handle these AI models cause memory size, so you must build it as a Harbour and FWH 64 bits app and use llama64.dll

[Enrico Maria Giordano]

It does not work here. I click on Start and nothing happened. What am I missing? I compiled it with Harbour and FWH64.

[Antonio Linares]

Dear Enrico,

Have you downloaded "phi-2_Q4_K_M.gguf" and have you properly provided its path in the PRG code ?

Do you get any message or warning ?

How many ram do you have in your pc ?

[Enrico Maria Giordano]

Dear Enrico,

Have you downloaded "phi-2_Q4_K_M.gguf" and have you properly provided its path in the PRG code ?

No. Now it works. This is allows it to be compiled with xHarbour too:

Code: Select all  Expand view  RUN

hb_retnll( ( LONGLONG ) callback );

A Stop button would be useful. After a certain amount of line of text it seems to freeze. Maybe a scrolling is needed?

[Antonio Linares]

Please download llama64.dll again:

https://github.com/FiveTechSoft/FWH_tools/blob/master/llama64.dll

[Antonio Linares]

Very interesting:

Fine-tuning Microsoft's Phi-2 on your own data:
https://github.com/brevdev/notebooks/blob/main/phi2-finetune-own-data.ipynb

[Antonio Linares]

This version also "speaks" so there is no need to read

Code: Select all  Expand view  RUN

#include "Fivewin.ch"

function Main()

   local oDlg, cPrompt := PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
   local cAnswer := "", oAnswer, oBtn
   local oVoice := TOleAuto():New( "Sapi.SPVoice" )

   DEFINE DIALOG oDlg SIZE 700, 500 TITLE "FWH AI"

   @ 1, 1 GET cPrompt MULTILINE SIZE 300, 15

   @ 3, 1 GET oAnswer VAR cAnswer MULTILINE SIZE 300, 200

   @ 0.7, 52.5 BUTTON oBtn PROMPT "start" ;
      ACTION ( oBtn:Disable(), Llama( "phi-2_Q4_K_M.gguf", RTrim( cPrompt ),;
               CallBack( { | cStr | oAnswer:SetFocus(), oAnswer:Append( cStr ) } ) ),;
               oBtn:Enable(), oBtn:SetFocus() )

   @ 2.2, 52.5 BUTTON "Clear" ACTION oAnswer:SetText( "" )

   @ 3.2, 52.5 BUTTON "Speak" ACTION oVoice:Speak( oAnswer:GetText() )

   ACTIVATE DIALOG oDlg CENTERED

return nil

DLL FUNCTION Llama( cModel AS LPSTR, cPrompt AS LPSTR, pFunc AS PTR ) AS VOID PASCAL LIB "llama64.dll"

#pragma BEGINDUMP

#include <Windows.h>
#include <hbapi.h>
#include <hbapiitm.h>

static PHB_ITEM pBlock;

static void callback( char * szMsg )
{
   PHB_ITEM pStr = hb_itemPutC( NULL, szMsg );
   hb_evalBlock1( pBlock, pStr );
   hb_itemRelease( pStr );
}

HB_FUNC( CALLBACK )
{
   pBlock = hb_gcGripGet( hb_param( 1, HB_IT_BLOCK ) );
   hb_retnll( ( HB_LONGLONG ) callback );
}

#pragma ENDDUMP

[anserkk]

Will this work in FWH 32bit with Harbour and Borland BCC 7 ?

[Antonio Linares]

Dear Anser,

Unfortunately this only works on 64 bits due to memory requirements as 32 bits is not able to manage memory as efficiently as 64 bits does.

The entire AI model has to be loaded in memory and 32 bits is not able to do it.

[Jimmy]

hi,

have buildh64 Sample with my *.gguf Path but got Error Message loading llama64.dll
llama64.dll is in same Folder like EXE

what i´m doing wrong

[Antonio Linares]

Dear Jimmy,

Please install this:
https://aka.ms/vs/17/release/vc_redist.x64.exe