not ISOEM(), ISANSI() or IsUTF8()

[Jimmy]

hi,

i do try get String from DBF FIELD Type "C"

Code: Select all  Expand view  RUN

     xValue := FieldGet(ii)
      cType := aStruct [ ii ] [ DBS_TYPE ]

than i "test" for ISOEM(), ISANSI() or IsUTF8()

Code: Select all  Expand view  RUN

     DO CASE
        CASE cType = "C" .or. cType = "M"
          IF ISOEM(xValue)
              // OEM
          ELSEIF ISANSI(xValue)
              // ANSI
          ELSEIF IsUTF8(xValue)
              // UTF8
          ELSE
              // what Type is "this" ?
          ENDIF

it "seems" that i got :

ISOEM() when have "äöü ÄÖÜ"
ISANSI() when have "ß"

i never get IsUTF8
but get a lot "ELSE" (no "Umlaute")

so what Type is it when got "ELSE"

---

DBF was open with CODEPAGE, hb_cdpSelect() "DE850" / "DEWIN" and FW_SetUnicode() .T./.F.

[nageswaragunupudi]

We need to keep this in mind.

While all English characters and all characters of other West European languages other than accented characters are represented by a single byte in both ANSI and UTF8 notiation, the accented characters (which include German Umlauts) are represented by a single byte in ANSI and two bytes in UTF8.

Let us consider the example of "ü".
Is it ANSI or UTF8?
We can not say just by looking at it on the screen or in print.
Depends on how this string is encoded. We can find out its encoding by testing with IsUtf8() or still better, with STRTOHEX( c )

The same character is encoded as hex "FC" in ANSI and encoded as hex "C3BC" in UTF8.
When FWH displays, both look exacty alike and in fact both represent the same character.

But both are different. When compared, both are not equal and also we can not concatenate both the strings.

Nothing to do with FW_SetUnicode() and nothing to do with any Codepage

To see the point clearly, let is try this sample.

Code: Select all  Expand view  RUN

#include "fivewin.ch"

REQUEST DBFCDX

#xtranslate enc( <c> ) => If( IsUtf8( <c> ), "UTF8", If( IsAnsi( <c> ), "ANSI", "???" ) )

function Main()

   local cAnsi, cUtf8, cOem

   FW_SetUnicode( .F. )  // does not matter

   cAnsi    := Chr( 0xFC )
   cUtf8    := HEXTOSTR( "C3BC" )

   ? cAnsi, enc( cAnsi ) // ü, ANSI
   ? cUtf8, enc( cUtf8 ) // ü, UTF8
   ? cAnsi == cUtf8  // --> .F.
   ? cAnsi + cUtf8, enc( cAnsi + cUtf8 )   // --> üü, ANSI

   // TEST WITH DBF
   // FW_SetUnicode( .f. ), NO CODEPAGE

   DBCREATE( "UM.DBF", { { "FANSI", "C", 10, 0 }, ;
                         { "FUTF8", "C", 20, 0 } }, ;
      "DBFCDX", .T., "UM" )
   FW_ArrayToDBF( { { Replicate( cAnsi, 5 ), ;
                      Replicate( cUtf8, 5 ) } } )
   GO TOP

   ? enc( FieldGet( 1 ) ) // --> ANSI
   ? enc( FieldGet( 2 ) ) // --> UTF8
   // But when we see, both look alike

   XBROWSER ALIAS() COLUMNS "FANSI", "FUTF8", ;
         "STRTOHEX(TRIM(FANSI)) AS HEXANSI", ;
         "STRTOHEX(TRIM(FUTF8)) AS HEXUTF8"

return nil

In the xbrowse we see both the strings as same and when we print also both the strings are same (because FWH handles both ANSI and UTF8, transparently) but with Excel it is different.





We are proposing to initiate a full length discussion on this subject separately very soon.

[Jimmy]

hi,

thx for Andwer.

it was not the Problem when "hardcode" using UTF8-Editor and not when create a "new" DBF

it was from some (40 Year old) DBF, start with 7-Bit C/PM under Dbase ]} over. Dbase III+. Cli*pper, Xbase++, HMG and now Fivewin
these DBF (still) have OEM "Umlaute" not ANSI "Umlaute"

but there is no OEMtoUTF8() Function in FiveWin, or

i have try these OEM

Code: Select all  Expand view  RUN

  // OEM
   cText := STRTRAN( cText, CHR( 132 ), "ä" )
   cText := STRTRAN( cText, CHR( 148 ), "ö" )
   cText := STRTRAN( cText, CHR( 129 ), "ü" )

these are value from "old" DOS Style but seems not to have right Effect

using Windows Notepad i have create in ANSI

Code: Select all  Expand view  RUN

#ifdef Use_ANSI
   cText := STRTRAN( cIn, "ä", "ae" )
   cText := STRTRAN( cText, "ö", "oe" )
   cText := STRTRAN( cText, "ü", "ue" )
   cText := STRTRAN( cText, "ß", "ss" )

   cText := STRTRAN( cText, "Ä", "Ae" )
   cText := STRTRAN( cText, "Ö", "Oe" )
   cText := STRTRAN( cText, "Ü", "Ue" )
#endif
 

but it also "fail" when "open" *.XLSX ... all Type "C" are missing

i end up in UTF8 using Notepad++ as UTF8 Editor

Code: Select all  Expand view  RUN

#ifdef Use_UTF8_Sign
   cText := STRTRAN( cText, CHR( 228 ), "ä" )
   cText := STRTRAN( cText, CHR( 246 ), "ö" )
   cText := STRTRAN( cText, CHR( 252 ), "ü" )

This CODE made under UTF8 and work now as i need for those "old" Style OEM DBF with XLSwriter

here my full German OEM (DE850) "translate" for UTF8 Editor

cSign = "ä" ASC( cSign ) = 228
cSign = "ö" ASC( cSign ) = 246
cSign = "ü" ASC( cSign ) = 252
cSign = "Ä" ASC( cSign ) = 196
cSign = "Ö" ASC( cSign ) = 214
cSign = "Ü" ASC( cSign ) = 220
cSign = "ß" ASC( cSign ) = 223

---

i have add some "counter" in DO CASE

nOEM = 107712 nANSI = 17642 nELSE = 5821450

for nOEM or nANSI i need to "translate" Type "C" but not for nELSE = 5821450 whichh are 99,98 %

[nageswaragunupudi]

Mr. Jimmy
Sorry.
Now I understand.
By the way you remind me of programming with CP/M (8-bit) those days.

but there is no OEMtoUTF8() Function in FiveWin, or

But we have OEMtoANSI() function. Is it useful for conerting German Umlauts to ANSI?
Once we convert to ANSI, we can easily convert to UTF8 and we discussed this in the other thread.

This CODE made under UTF8 and work now as i need for those "old" Style OEM DBF with XLSwriter

That is nice. So are you able to finally solve the conversion issue?

If you don't mind, I like to ask you some help purely for my knowledge and educational purpose.

Can you please post some OEM strings here in Hex?

Code: Select all  Expand view  RUN

cTextOEM := FieldGet( n )
cTextHex := STRTOHEX( cTextOEM )
 

and please post the hex string. A few strings like that.
I am not sure if I can do, but would like to try to convert to Ansi.
if that is ok with you.

[Jimmy]

hi,

forgot to say that Xbase++ still use default OEM Setting

Xbase++ use OEM <-> ANSI "translation" internal "on-fly"
to use Xbase++ with ANSI you need to compile with -a

[Jimmy]

hi,

That is nice. So are you able to finally solve the conversion issue?

Yes, after STRTRAN() "Umlaute" and use this CODE to "check" Sign

Code: Select all  Expand view  RUN

FUNCTION CheckValue( xValue, lNoCheck )
LOCAL cRet     := ""
LOCAL aSign    := "abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-+/.,:;!$%&/()=?#"
LOCAL cSign, ii, iMax, cLine := TRIM( xValue )

DEFAULT lNoCheck := .F.

#ifdef Use_UTF8_Sign   // NEED UTF8 Edtitor
   aSign += "äöüÄÖÜß"
#endif
   aSign += CHR( 34 )

   IF lNoCheck = .T.
      cRet := cLine
   ELSE
      iMax := LEN( cLine )
      FOR ii := 1 TO iMax
         cSign := SUBSTR( cLine, ii, 1 )
         // IF AT(cSign,cLine) > 0
         IF cSign $ aSign
            cRet += cSign
         ELSE
fwlog cSign, ASC( cSign ), cRet, RECNO()
            cRet := ""
            EXIT
         ENDIF
      NEXT
   ENDIF

RETURN cRet

If you don't mind, I like to ask you some help purely for my knowledge and educational purpose.

Can you please post some OEM strings here in Hex?

Code: Select all  Expand view  RUN

cTextOEM := FieldGet( n )
cTextHex := STRTOHEX( cTextOEM )
 

and please post the hex string. A few strings like that.
I am not sure if I can do, but would like to try to convert to Ansi.
if that is ok with you.

ok i will use STRTOHEX instead of ASC()

[Jimmy]

hi,

this i got when have use DBU and fill it with German Umlaute

xValue = "ä" STRTOHEX(xValue) = "84" ASC(xValue) = 132
xValue = "ö" STRTOHEX(xValue) = "94" ASC(xValue) = 148
xValue = "ü" STRTOHEX(xValue) = "81" ASC(xValue) = 129
xValue = "Ä" STRTOHEX(xValue) = "8E" ASC(xValue) = 142
xValue = "Ö" STRTOHEX(xValue) = "99" ASC(xValue) = 153
xValue = "Ü" STRTOHEX(xValue) = "9A" ASC(xValue) = 154
xValue = "ß" STRTOHEX(xValue) = "E1" ASC(xValue) = 225

i need UTF8 Value which i get this Way

Code: Select all  Expand view  RUN

#define Use_OEM
#ifdef Use_OEM
   // OEM
   cText := STRTRAN( cText, CHR( 132 ), "ä" )
   cText := STRTRAN( cText, CHR( 148 ), "ö" )
   cText := STRTRAN( cText, CHR( 129 ), "ü" )

   cText := STRTRAN( cText, CHR( 142 ), "Ä" )
   cText := STRTRAN( cText, CHR( 153 ), "Ö" )
   cText := STRTRAN( cText, CHR( 154 ), "Ü" )

   cText := STRTRAN( cText, CHR( 225 ), "ß" )
#endif

[nageswaragunupudi]

Code: Select all  Expand view  RUN

OemToAnsi( cOem ) --> cAnsi

seems to function perfectly and much simpler to use.

We can also convert from Ansi to Utf8 as explained earlier

[nageswaragunupudi]

I made a simple test by first creating a small dbf with 2 fields and 10 records all containing random OEM strings of Umlauts.

This is the code:

Code: Select all  Expand view  RUN

   field FLD1, FLD2
   local aoem  := { "84", "94", "81", "8E", "99", "9A", "E1" }
   local n

   // STEP-1: Create DBF with OEM Umlaut chars
   DBCREATE( "\fwh\samples\UMLOEM.DBF", { { "FLD1", "C", 7, 0 }, ;
            { "FLD2", "C", 7, 0 } }, "DBFNTX", .T., "UML" )

   for n := 1 to 10
      DBAPPEND()

      aoem  := AShuffle( aoem )
      FLD1  := HEXTOSTR( FW_ArrayAsList( aoem, "" ) )

      aoem  := AShuffle( aoem )
      FLD2  := HEXTOSTR( FW_ArrayAsList( aoem, "" ) )

   next
   CLOSE DATA
 

We know it is easy to view OEM characters from DOS prompt.
I used a DBEDIT uitily I have with me to view this OEM data in Command Window.
This is the code;

Code: Select all  Expand view  RUN

   WaitRun( "\fwh\samples\dbedit32c \fwh\samples\umloem.dbf" )
 

This is the result and we are sure that the data is OEM and we can view them from a harbour program run in command window.



Now XBrowse also can display OEM data by converting to ANSI, if we set oBrw:lOemAnsi := .t.. If we want this conversion only for some columns but not all, we can set oCol:lOemAnsi := .t. for the required columns instead of setting for the entire browse. Not only display but we can also edit.
When displaying the data XBrowse converts the data from OEM to ANSI and displays. If the user modifies the data though inline edit and save the changes, then XBrowse will convert ANSI to OEM and saves the data. So, the DBF continues to be OEM.

oBrw:Report() and oBrw:ToExcell() also work perfectly using ANSI converted data.

Now, we will try to view the OEM dbf with XBrowse

Code: Select all  Expand view  RUN

   XBROWSER "\FWH\SAMPLES\UMLOEM.DBF" SETUP ( ;
      oBrw:lOemAnsi := .t., oBrw:nWidths := 80 )
 

Finally we will convert to OEM DBF to ANSI DBF and we can then keep aside the OEM dbf and start using the ANSI DBF

Code: Select all  Expand view  RUN

   USE \FWH\SAMPLES\UMLOEM VIA "DBFNTX"
   DBCREATE( "\FWH\SAMPLES\UMLANSI.DBF", DBSTRUCT(), "DBFNTX", .T., "UMLA" )
   SELECT UMLOEM
   DBEVAL( <||
         UMLA->( DBAPPEND() )
         UMLA->FLD1  := OEMTOANSI( UMLOEM->FLD1 )
         UMLA->FLD2  := OEMTOANSI( UMLOEM->FLD2 )
         return nil
         > )
   CLOSE DATA

   XBROWSER "\FWH\SAMPLES\UMLANSI.DBF" SETUP ( oBrw:nWidths := 80 )
 

This is the full program

Code: Select all  Expand view  RUN

#include "fivewin.ch"

function Main()

   field FLD1, FLD2
   local aoem  := { "84", "94", "81", "8E", "99", "9A", "E1" }
   local n

   // STEP-1: Create DBF with OEM Umlaut chars
   DBCREATE( "\fwh\samples\UMLOEM.DBF", { { "FLD1", "C", 7, 0 }, ;
            { "FLD2", "C", 7, 0 } }, "DBFNTX", .T., "UML" )

   for n := 1 to 10
      DBAPPEND()

      aoem  := AShuffle( aoem )
      FLD1  := HEXTOSTR( FW_ArrayAsList( aoem, "" ) )

      aoem  := AShuffle( aoem )
      FLD2  := HEXTOSTR( FW_ArrayAsList( aoem, "" ) )

   next
   CLOSE DATA

   // STEP-2
   // View it from DOS-PROMPT with DBEDIT
   // We can browse only OEM data in DOS prompt
   //
//   WaitRun( "\fwh\samples\dbedit32c \fwh\samples\umloem.dbf" )
   // This works only for me. You may please comment it our


   // STEP-3: XBrowse OEM data using oBrw:lOemAnsi
   XBROWSER "\FWH\SAMPLES\UMLOEM.DBF" SETUP ( ;
      oBrw:lOemAnsi := .t., oBrw:nWidths := 80 )

   // STEP-4
   // CONVERT UMLOEM.DBF TO UMLANSI.DBF ONCE AND FOR ALL
   // AND FORGET ABOUT OEM

   USE \FWH\SAMPLES\UMLOEM VIA "DBFNTX"
   DBCREATE( "\FWH\SAMPLES\UMLANSI.DBF", DBSTRUCT(), "DBFNTX", .T., "UMLA" )
   SELECT UMLOEM
   DBEVAL( <||
         UMLA->( DBAPPEND() )
         UMLA->FLD1  := OEMTOANSI( UMLOEM->FLD1 )
         UMLA->FLD2  := OEMTOANSI( UMLOEM->FLD2 )
         return nil
         > )
   CLOSE DATA

   XBROWSER "\FWH\SAMPLES\UMLANSI.DBF" SETUP ( oBrw:nWidths := 80 )

   // Now, we can keep the OEM dbf in a safe place
   // stop using it and start using UMLANSI.DBF in its place
   // from now onwards

return nil

NOTES: NO UNICODE and NO CODEPAGE
Please keep it simple for this Test

===========================

But the problem with the DBF of Mr. Jimmy is not as simple as this.
We again come back to first question he raised.
If we know the text is OEM, we can use OemToAnsi.
But what if ISOEM( cText ) is .F. and also ISANSI( cText ) is .F.?

This involves more complex work.
I think (I may not be right) first let us convert all values where ISOEM( value ) == .t. to ANSI
and later handle values which are not ISANSI(), character by character.
The is problem is interesting.

[nageswaragunupudi]

Another way to convert Umlaut OEM to ANSI and UTF8

Code: Select all  Expand view  RUN

#include "fivewin.ch"

function Main()

   local cOemHex  :=  "8494818E999AE1"
   loCAL cOemText := HEXTOSTR( cOemHex )
   local cAnsiText, cUtf8Text

   cAnsiText   := UML_OEMTOANSI( cOemText )
   ? cAnsiText, Len( cAnsiText )

   // no proble, even if we input ANSI by mistake
   cAnsiText   := UML_OEMTOANSI( cAnsiText )
   ? cAnsiText, Len( cAnsiText )

   cUtf8Text   := UTF16toUTF8( strtoWide( cAnsiText ) )
   ? cUtf8Text, STRTOHEX( cUtf8Text ), Len( cUtf8Text )

return nil

#pragma BEGINDUMP
#include <hbapi.h>

HB_FUNC( UML_OEMTOANSI )
{
unsigned char achar[] = {
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,
54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,
79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,
103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,
122,123,124,125,126,127,128,252,130,131,228,133,134,135,136,137,138,139,140,
141,196,143,144,145,146,147,246,149,150,151,152,214,220,155,156,157,158,159,
160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,
179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,
198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,
217,218,219,220,221,222,223,224,223,226,227,228,229,230,231,232,233,234,235,
236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,
255 };

   char * pText = ( char * ) hb_parc( 1 );
   int iLen = hb_parclen( 1 );
   int i;
   unsigned char c;

   for ( i = 0; i < iLen; i++ )
   {
      c = pText[ i ];
      pText[ i ] = achar[ c ];
   }

   hb_retc( pText );
}
#pragma ENDDUMP
 

[Jimmy]

hi,

thx for your Sample

i have try OemToAnsi() before and now also UML_OemToAnsi() ... but this is not the Problem

i have disable

Code: Select all  Expand view  RUN

*   hb_LangSelect( cLangCode )
*   hb_cdpSelect( cCodepage )
*   FW_SetUnicode( .T. )
USE (cDBF)  // CODEPAGE cCodepage EXCLUSIVE

now please try to "export" to Excel using DrXlsx, NOT XBROWSE

i like to get a working Excel Sheet with German "Umlaute" using DrXlsx

---

when you ask me for STRTOHEX() i got

xValue = "„" STRTOHEX(xValue) = "84" ASC(xValue) = 132
xValue = "”" STRTOHEX(xValue) = "94" ASC(xValue) = 148
xValue = "" STRTOHEX(xValue) = "81" ASC(xValue) = 129
xValue = "Ž" STRTOHEX(xValue) = "8E" ASC(xValue) = 142
xValue = "™" STRTOHEX(xValue) = "99" ASC(xValue) = 153
xValue = "š" STRTOHEX(xValue) = "9A" ASC(xValue) = 154
xValue = "á" STRTOHEX(xValue) = "E1" ASC(xValue) = 225

but for DrXlsx i need UTF8 so these Value

cSign = "ä" STRTOHEX( cSign ) = "E4" ASC( cSign ) = 228
cSign = "ö" STRTOHEX( cSign ) = "F6" ASC( cSign ) = 246
cSign = "ü" STRTOHEX( cSign ) = "FC" ASC( cSign ) = 252
cSign = "Ä" STRTOHEX( cSign ) = "C4" ASC( cSign ) = 196
cSign = "Ö" STRTOHEX( cSign ) = "D6" ASC( cSign ) = 214
cSign = "Ü" STRTOHEX( cSign ) = "DC" ASC( cSign ) = 220
cSign = "ß" STRTOHEX( cSign ) = "DF" ASC( cSign ) = 223

so what i need is OEMtoUTF8()

Code: Select all  Expand view  RUN

132 -> 228
148 -> 246
129 -> 252
142 -> 196
153 -> 214
154 -> 220
225 -> 223

[nageswaragunupudi]

i like to get a working Excel Sheet with German "Umlaute" using DrXlsx

Suggested:
OEM -> ANSI -> UTF8
For ansi to utf8

Code: Select all  Expand view  RUN

UTF16toUTF8( strToWide( cAnsi ) )

So we can use:

Code: Select all  Expand view  RUN

UTF16toUTF8( strToWide( UML_OEMTOANSI( fieldGet( n ) ) ) )

Please use UML_OEMTOANSI() but not OemToAnsi()

[nageswaragunupudi]

We can also make all this in a single function.
For that can you provide me a list of ALL accented characters in German language?
Not only the 7 chars. We need all chars.
For example, there are other chars like: Ûúùõòóô, etc.

[nageswaragunupudi]

Next point, in the next Version, FWH will be providing DBFTOXLSX using DrXlsx. This function will automatically take care of ANSI to UTF8 conversion.

[Jimmy]

hi,

Suggested:
OEM -> ANSI -> UTF8
So we can use:

Code: Select all  Expand view  RUN

UTF16toUTF8( strToWide( UML_OEMTOANSI( fieldGet( n ) ) ) )

Please use UML_OEMTOANSI() but not OemToAnsi()

YES this "seems" to work, no Error in LOG

but when try to open Excel Sheet i got Warning and all Type "C" are empty ...

---

i have compare

Code: Select all  Expand view  RUN

  cText := UTF16toUTF8( strToWide( UML_OEMTOANSI( cText ) ) )

vs.

Code: Select all  Expand view  RUN

#ifdef Use_UTF8_Sign
   // using UTF8-Editor
   cText := STRTRAN( cText, CHR( 228 ), "ä" )
   cText := STRTRAN( cText, CHR( 246 ), "ö" )
   cText := STRTRAN( cText, CHR( 252 ), "ü" )
   cText := STRTRAN( cText, CHR( 196 ), "Ä" )
   cText := STRTRAN( cText, CHR( 214 ), "Ö" )
   cText := STRTRAN( cText, CHR( 220 ), "Ü" )

   cText := STRTRAN( cText, CHR( 223 ), "ß" )
#endif

both "seems" to be same (!) even HEX are same
so i don´t understand why UTF16toUTF8( strToWide( UML_OEMTOANSI())) did not work to create "right" Excel Sheet