FiveTech Software tech support forums

Hello Fivewinners.
Hola Fivewinners

Need to open 160.000+ xml files and extract some info
Necesito abrir mas de 160.000 archivos xml y sacar informacion

I use tXmlDocument class
Uso la clase TxmlDocument

I have the file names in an Array
Tengo los nombres de los archivos en un arreglo

Code: Select all  Expand view

    For X=1 to 160.xxx

        If z=400
           HB_GcAll()
           SysRefresh()
           z:=0
        Endif

        LeeFileXml(hFile[x])
 
    Next
--------------------------------------
--------------------------------------
Function LeeFile Xml(hFile)
Local oXmlDoc := TXmlDocument():New( hFile )
Local oXmlIter := TXmlIterator():New( oXmlDoc:oRoot )
Local Data1:=0
Local Data2:=""

   while .T.
         oTagActual = oXmlIter:Next()
         If oTagActual != nil

            If oTagActual:cName ="END"
              Exit
            Endif

           If oTagActual:cName ="ValorXxx"
              Data1:=Val(oTagActual:cData)
           Endif

           If oTagActual:cName ="ValorYyy"
              Data2:=oTagActual:cData
           Endif

      End

FClose( hFile )
oXmlDoc:=Nil
oXmlIter:=Nil

 oSvr:Insert("detalle",{"NUMERO","DESCRIPCION"},{Data1,Data2})

Return Nil


 

It does what it is supposed to do, But I get this error each time Between 5.000 and 6.000 processed files
Hace lo que se supone debe hacer, pero tengo cada vez tengo este error entre los 5.000 y 6.000 archivos procesados

Application Internal Error - C:\SQL\xNewAqua\xNewII.Exe
Terminated at: 2024-03-20 16:18:46
Error irrecuperable 9009: hb_xrealloc no puede reubicar la memoria......

It fails in no specific file, just after 5000+ file
No falla en ningun archivo especifico, solo despues de los 5000

Any help will be appreciated
Cualquier ayuda se agradece

HARBOUR, FWH, MSVC, WINDOWS 11, 32GB ram, NVME disk

Run out of memory ?

Maybe you can put a part of this code inside the loop en show the result while running. Than you see if there is a memory leak.

Code: Select all  Expand view

cErrorLog += "   Hardware memory: " + ;
                    cValToChar( Int( nExtMem() / ( 1024 * 1024 ) ) + 1 ) + " MB ( available: "+;
                    cValToChar (Int (nAvailMem() / (1024 * 1024 ) ) +1 ) + " MB )" + CRLF + CRLF
 

Hi Marc

To Avoid Memory leaks, within the loop I call an external function each time I read a xml file, close it and turn to nil each class instance.
For every 400 files I do a HB_GcAll() and a SysRefresh(), the biggest xml file is just 29 kb

Im going to put your code inside the loop, and show it every 1.000 files, hope it helps to know

Thanks

nAvailMem() seems not to be standard.

These seems to be ok.

cErrorLog += " Dynamic memory consume:" + CRLF
cErrorLog += " Actual Value: " + Str( MemUsed() ) + " bytes" + CRLF
cErrorLog += " Highest Value: " + Str( MemMax() ) + " bytes" + CRLF

I found the error

Disable Avast Antivirus, and it works with no problem.

Code: Select all  Expand view

FUNCTION TU_Func()

   // inicio
   hb_gcAll( .F. )

   FOR X = 1 TO 160.xxx

      SYSREFRESH()

      IF z = 400

         // HB_GcAll()

         // SysRefresh()

         z := 0

      ENDIF

      LeeFileXml( hFile[ x ] )

   NEXT

   // Salida
   hb_gcAll( .T. )

RETURN NIL

FUNCTION LeeFile Xml( hFile )

   LOCAL oXmlDoc := TXmlDocument():New( hFile )
   LOCAL oXmlIter := TXmlIterator():New( oXmlDoc:oRoot )
   LOCAL Data1 := 0
   LOCAL Data2 := ""

   hb_gcAll( .F. )

   WHILE( .T. )

      SYSREFRESH() // aqui

      oTagActual = oXmlIter:Next()

      IF oTagActual != nil

         IF oTagActual:cName = "END"

            EXIT

         ENDIF

         IF oTagActual:cName = "ValorXxx"

            Data1 := Val( oTagActual:cData )

         ENDIF

         IF oTagActual:cName = "ValorYyy"

            Data2 := oTagActual:cData

         ENDIF

      ENDIF

   ENDDO // ??

   FClose( hFile )

   oXmlDoc  := Nil
   oXmlIter := Nil

   oSvr:Insert( "detalle", { "NUMERO", "DESCRIPCION" }, { Data1, Data2 } )

   hb_gcAll( .T. )

RETURN NIL
 

Regards, saludos.