Strange program behavior...

[HunterEC]

I compiled the following program for xHarbour (buildx.bat). When I runned it with the TEST.CSV file (25MB) it process, display the last 2 MSgInfo() dialogs and ends. When I run it with a file called NP091011.CSV (3.28GB) it returns to the command prompt after 2-3 seconds. When I look at Task Manager it is running in the background and splitting the input file. Any ideas of what's going on ? Thank you.


#define CFILENAME "TEST.CSV"
#define CR CHR(13)
#define LF CHR(10)
#define LINEFEED CHR(13) + CHR(10)
#define NCHARS 1
#define NMAXLINELENGTH (4096 * 3)
#define NSPLIT_LINES 25000


#include "Common.ch"
#include "FiveWin.ch"


FUNCTION Main (cFile)

#define nHandle aVars[1]
#define cInFile aVars[2]
#define cOutFile aVars[3]
#define nPieces aVars[4]
#define nLineCtr aVars[5]
#define nTotLines aVars[6]
#define nI aVars[7]
#define cChars aVars[8]
#define cText aVars[9]

LOCAL aVars[9]

IF cFile == NIL
cFile := ""
ENDIF
IF ! FILE(cFile)
MsgInfo("El Archivo De Entrada: " + cFile + " No Existe")
cFile := CFILENAME
ENDIF

nHandle := {-1, -1}
cInFile := cFile
IF (nHandle[1] := FOPEN(cInFile, 0)) >= 0
nPieces := 0
FileCreate (@aVars)

nLineCtr := nTotLines := 0
cText := ""
cChars := FREADSTR(nHandle[1], NCHARS)

DO WHILE ! EMPTY(cChars) .AND. nHandle[2] >= 0
ReadLine (@aVars)

WriteLine (@aVars)

nLineCtr++
nTotLines++
IF nLineCtr >= NSPLIT_LINES
CloseFile (@aVars, 2)

nLineCtr := 0

FileCreate (@aVars)
ENDIF

cText := ""
cChars := FREADSTR(nHandle[1], NCHARS)
ENDDO
IF ! FCLOSE(nHandle[1])
MsgInfo("NO Se Pudo Cerrar El Archivo")
ENDIF

WriteLine (@aVars)

CloseFile (@aVars, 2)

ELSEIF nHandle[1] < 0
MsgInfo("El Archivo: " + cInFile + " NO Se Pudo Abrir")
ENDIF

IF nHandle[1] >= 0 .AND. nHandle[2] >= 0
MsgInfo("Archivo " + cInFile + " Dividido En " + LTRIM(STR(nPieces,4)) + " Pedazo(s)")
MsgInfo(TRANSFORM(nTotLines, "9,999,999") + " Líneas Procesadas")
ENDIF
RETURN NIL
* EOP: Main


STATIC FUNCTION FileCreate (aVars)
cOutFile := LEFT(cInFile,4) + STRTRAN(STR(++nPieces,4), " ", "0") + ".CSV"
IF ! ((nHandle[2] := FCREATE(cOutFile, 0)) >= 0)
MsgInfo("El Archivo: " + cOutFile + " NO Se Pudo Crear")
ENDIF
RETURN (nHandle[2] >= 0)
* EOF: FileCreate


STATIC FUNCTION ReadLine (aVars)
DO WHILE ! (cChars == LF) .AND. ! (cChars == "")
cText += cChars
cChars := FREADSTR(nHandle[1], NCHARS)
ENDDO
RETURN (.T.)
* EOF: ReadLine


STATIC FUNCTION WriteLine (aVars)
IF FWRITE(nHandle[2], cText + LINEFEED) != (LEN(cText) + 2)
MsgInfo("La Línea NO Se Escribio Completa")
ENDIF
RETURN (.T.)
* EOF: WriteLine


STATIC FUNCTION CloseFile (aVars, nIndex)
IF ! FCLOSE(nHandle[nIndex])
MsgInfo("NO Se Pudo Cerrar El Archivo " + cOutFile)
ENDIF
RETURN (.T.)
* EOF: CloseFile

#undef nHandle
#undef cInFile
#undef cOutFile
#undef nPieces
#undef nLineCtr
#undef nTotLines
#undef nI
#undef cChars
#undef cText

[James Bott]

Hunter,

First, what is the program supposed to do?

When I look at Task Manager it is running in the background and splitting the input file.

When running in the background is it using CPU time?

When you say "splitting the input file" did you mean splitting the output file into multiple segments?

I do see these:

#define NMAXLINELENGTH (4096 * 3)
#define NSPLIT_LINES 25000

Perhaps it is doing what it is told?

James

[HunterEC]

James:

Thank you for your reply. Indeed under processes it consumes 25% of CPU time. This program takes one huge CSV file and split it in 25,000 lines pieces.

[James Bott]

Hunter,

Thank you for your reply. Indeed under processes it consumes 25% of CPU time. This program takes one huge CSV file and split it in 25,000 lines pieces.

So, it is done splitting the file, but still remains in memory, or is it not done and remaining in memory? Maybe it is not done, and still working? Can you define the problem in more detail?

James

[Antonio Linares]

Hunter,

Try to call SysRefresh() from inside the do while loop, so you let Windows process pending messages.