hb_xrealloc can't reallocate memory

[TimStone]

I checked old posts, found two with this error, but never a resolution.

I have a Function creating an XML file for a data submission. It reads through several files, creates data lines, and then writes the results to an XML file.

It starts by opening a file on the drive:
nFileHandle := FCREATE( filename, 0 )

Then data is stored in a variable,
oxData := ""
oxData += newdata

The file tree is:
Customer
Service Item
Service Tickets

After each customer, I issue:
FWrite( nFileHandle, oxData )
sysrefresh( )

This works well for about 9 MB of data. Then, I suddenly get:

Unrecoverable error 9009 hb_xrealloc can't reallocate memory

After the error, I can look at the XML file, and it is written to the same point each time. I checked and there is no error in the database. I figure that the FWRITE( ) function must have a limit in file size. I'm using the latest xHarbour.com for this build.

There is plenty of memory on the computer. In fact, it has happened on two different computers ( maybe more ).

Can anyone provide suggestions for how I might work around this issue ? Your thoughts would be greatly appreciated.

Tim

[TimStone]

Found the cause of the problem ... fixed it ... but still ... any thoughts on this error would be nice ...

In this case, a key field was empty on one record, sending the program into a looping search. I'll build a trap for that !

Tim

[Antonio Linares]

Tim,

If the memory was allocated from an endless loop then you could experience that error

[Carlos Mora]

It's not necesary to be in an endless loop or sth like that to be in trouble with memory reallocation.

Clipper and [x]Harbour do a great job managing memory for us. Those who have written programs in low level languages as C or Pascal know what i mean.
Every not fixed lenght var, like strings or arrays, allocates memory outside the regular stack and data segment, in the memory heap. Harbour provides the management services required for that, so usually there ir no problem, but under certain circunstances that service is unable to handle the requirement.
That condition is usually produced by the progressive incremental allocation of memory, like Tim's code shows, concatenating small strings into a buffer. oxData, as a (I guess) LOCAL var resides in the stack, but just only the 'var' component of it. It's value is stored in the heap, using the Harbour services. The stack portion of oxData stores the handle of that memory chunck.
At startup, all the available heap conforms a big unique block of memory. Usually a several Gb segment free.
Let's take an example to make it more graphic: oxData:= Space(100). That means that there is a piece of 100 bytes stored in the heap. If originaly the original heap had 1000 bytes we got the heap split in two fragments, one of 100 byte and other, free, of 900. That limit of 1000 bytes is not real, it's for the sake of the example, usually it is way bigger than that, in the order of Gb, but an small heap is needed to show how the problem arises.
Now lets do a concatenation over the same var: oxData += 'xyz' . Harbour estimates the resulting string, what will be 103 bytes long, so it issues a hb_xrealloc() to the current pointer stored in oxData. The memory service will look for a 103 byte chunck free in memory, copy the curren 100 bytes content, amd return the new pointer. So far so good. But the problem comes from the old 100 byte chunck. What happen with it? It's not neccesary anymore, so it's marked as available, and now the memory has got fragmented into 3 pieces: 1 of 100 bytes (free), 1 of 103 (oxData), and the remaining 797 bytes as the last free block.
Do it again: oxData += 'xyz'. Now the resulting string will be 106, so, after processing it, the heap memory map becomes: 1 of 100 bytes (free), 1 of 103 (free), 106 used (oxData), and the last 691 free bytes.
One more time: oxData += 'xyz'. Now the heap is: 1 of 100 bytes (free), 1 of 103 (free), 1 of 106 (free), 109 used (oxData), and the last 582 free bytes.
In few iterations we will run out of memory, but not for not having free memory, but because the memory has got so fragmented that the memory service cannot provide a big enough piece of memory as required.
There is a very well known process called 'Garbage collection', wich among other things, recovers all the free pieces into bigger ones, defragmenting the memory.
Usually our processor intensive functions don't allow the Harbour's core to activate the garbage collector, but we are able to manually execute it calling HB_GcAll(). In Clipper that process was triggered by calling Memory(-1). So you can introduce calls to the garbage collector in your code so your memory don't get fragmented.

Arrays also have the same problem (issuing aAdds to create long arrays), but are less frequent and have a much better solution: Preallocation.

There is a way to warn Harbour that we are creating an array but it will be x elements long, check the xHarbour functions ASIZEALLOC() and ALENALLOC(), and the Harbour counterparts.

The real memory management is more complicated than all this, it includes things like virtual memory and much more, but basically the example shows the concept behind the problem.

Sorry for the long speech, but it's a little bit difficult to me to explain this concept without all this. And no graphics.

[James Bott]

Carlos,

Thanks for that detailed explanation.

So, are you saying that you could reduce the fragmentation by pre-allocating a string of spaces, the replacing the spaces sequentially rather than starting with an empty string and adding strings to it? I suppose we could create a class to handle this.

I still remember on my first computer, a Radio Shack TRS-80, when I was processing lots of data (in 8K of RAM), the garbage collection would sometimes take up to an hour. At that time, myself, and nobody I knew, understood what was going on. The computer would just appear to freeze. I accidentally discovered that if I waited long enough it would continue. Of course, waiting an hour was still better then doing the work by hand.

James

[Carlos Mora]

Hi James

So, are you saying that you could reduce the fragmentation by pre-allocating a string of spaces, the replacing the spaces sequentially rather than starting with an empty string and adding strings to it? I suppose we could create a class to handle this.

In my case, i changed the function to not use the buffer, but write directly to the resulting file, but it may not be the case.
I have to check the Harbour's sources. There should be functions to make the preallocation, so we don't need to do nothing special.
In xHarbour
May be adding some well-timed hb_GcAll() calls do the job as well.
Last minute news:

2010-06-02 10:30 UTC+0200 Przemyslaw Czerpak (druzus/at/priv.onet.pl)
* harbour/src/vm/garbage.c
+ added new PRG function HB_GCSETAUTO( [nNewVal] ) - nPrevVal
which allows to set the frequency of automatic GC activation.
nNewVal is number of GC items (in thousands). If total number
of new GC items allocated from last GC pass exceed this value
then GC is automatically activated. Setting 0 disable automatic
GC activation. Automatic GC activation is necessary for programs
which allocates big number of complex items with cyclic references
without entering any idle state which can activate GC. In such
case users have to add to their code call to hb_gcAll() function.
In xHarbour they used sometimes background task for this job
but it's inefficient workaround for the problem due to reduced
performance.
This code is still experimental code enabled by HB_GC_AUTO Harbour
compile time macro.

[TimStone]

Carlos,

Thank you very much for the detailed explanation. That really helps me better understand how memory is managed.

I assumed the function SysRefresh( ) was issued to clean up the memory

Tim

[Enrico Maria Giordano]

James,

I still remember on my first computer, a Radio Shack TRS-80,

EMG

[Enrico Maria Giordano]

Was it this?

https://en.wikipedia.org/wiki/TRS-80

EMG

[TimStone]

My first was a Toshiba T100 running CP/M ( with dBase II even ), pictured here: http://www.oldcomputers.net/toshiba-t100.html Not shown in this listing are the Dual floppy drive box and the monitor.

[Enrico Maria Giordano]

Tim,

great!

EMG

[James Bott]

Carlos,

In my case, i changed the function to not use the buffer, but write directly to the resulting file,

Just a few days ago I was processing a lot of large files with a program I wrote in FWH. It was reading and writing each line from and to disk. I was taking an incredibly long time with the disk thrashing all the time.

I converted it to using a 400 line buffer for the output and the time decreased by at least an order of magnitude.

Also, you may remember just a week or two ago we discovered that the ADORDD was doing a commit after each record during APPENDs. This was taking 11 hours to append 26 thousand records. Appending the same file using DBFs took 1 second.

So, buffering before writing to disk provides a tremendous increase in speed. One would expect this since writing to disk is a physical process which will be much slower than a computation process.

So I am thinking that in order to use a buffer and prevent memory fragmentation I could pre-allocate a blank string of 400 line by 80 characters as my buffer. Then just keep replacing the spaces in the string with text until it is full, then write to disk.

Thoughts?

James

[James Bott]

Enrico,

Yes that is the one. It only had a cassette drive to store data. It took about a half hour just to load the program I wrote (in BASIC). Then another half hour to load the data. Then after it started processing it would hang for 10 minutes to an hour. Whew!

It was still better than a pocket calculator.

James

[Enrico Maria Giordano]

James,

I think that you must have the problematic code in your hands, then try a possible solution.

EMG

[Enrico Maria Giordano]

James,

Enrico,

Yes that is the one. It only had a cassette drive to store data. It took about a half hour just to load the program I wrote (in BASIC). Then another half hour to load the data. Then after it started processing it would hang for 10 minutes to an hour. Whew!

It was still better than a pocket calculator.

James

Other times...

My first computer was a Sinclair ZX81.

EMG