• Please review our updated Terms and Rules here

Memory allocation and usage monitoring in Open Watcom C large model.

Mike Chambers

Veteran Member
Joined
Sep 2, 2006
Messages
2,621
Hi everyone, I haven't been around here much in quite a while! I see both a lot of familiar names still posting, and some new so I'm glad to see the forum is still alive and kicking. :)

I'm porting a piece of network software from *nix to real mode DOS using Open Watcom and the Watt32 lib. It's essentially working, but I'm wondering about memory allocation in the large model and how OW tries to malloc memory. My understanding is that in large, malloc() is equivalent to _fmalloc() and all pointers are far across the whole program. It should be able to use all memory available on the system, with the caveat that no single malloc() can give you more than 64 KB at once.

This software is a server program, and I'm running out of RAM after about 9 or 10 connections. I don't really know of a good way to dynamically monitor total RAM usage in this environment. There's the _memavl() function in Watcom, but it returns an unsigned 16-bit, so it obviously can't report back the entire amount of free memory in large. This number starts at about 29 KB, though as I get more connections, it drops down pretty quickly. The number doesn't change until several connections have already been made though.

I'm just not sure if it's actually using up all available RAM on the system before eating up memory from what is being reported by _memavl(). What 64 KB area is that reporting on? The near heap?

Is there a way to really see what's going on with RAM allocation? It's a fairly big program. The EXE is 400 KB, and I know Watt32 is hungry for TCP buffer memory so maybe I really am out of RAM already. Is it initially using what's available in the far heap, then digs into the near heap when that's exhausted? This may explain why _memavl() doesn't report any difference for the first several client connections.

I've been out of the retroprogramming hobby for a while. I'm feeling a little rusty. If I'm really out of RAM, maybe I can tinker with Watt32 to make it less memory hungry at the expense of performance. I could also have OW optimize for size instead of speed to possibly gain another 10-20 KB...
 
Last edited:
Hi everyone, I haven't been around here much in quite a while! I see both a lot of familiar names still posting, and some new so I'm glad to see the forum is still alive and kicking. :)

I'm porting a piece of network software from *nix to real mode DOS using Open Watcom and the Watt32 lib. It's essentially working, but I'm wondering about memory allocation in the large model and how OW tries to malloc memory. My understanding is that in large, malloc() is equivalent to _fmalloc() and all pointers are far across the whole program. It should be able to use all memory available on the system, with the caveat that no single malloc() can give you more than 64 KB at once.

This software is a server program, and I'm running out of RAM after about 9 or 10 connections. I don't really know of a good way to dynamically monitor total RAM usage in this environment. There's the _memavl() function in Watcom, but it returns an unsigned 16-bit, so it obviously can't report back the entire amount of free memory in large. This number starts at about 29 KB, though as I get more connections, it drops down pretty quickly. The number doesn't change until several connections have already been made though.

I'm just not sure if it's actually using up all available RAM on the system before eating up memory from what is being reported by _memavl(). What 64 KB area is that reporting on? The near heap?

Is there a way to really see what's going on with RAM allocation? It's a fairly big program. The EXE is 400 KB, and I know Watt32 is hungry for TCP buffer memory so maybe I really am out of RAM already. Is it initially using what's available in the far heap, then digs into the near heap when that's exhausted? This may explain why _memavl() doesn't report any difference for the first several client connections.

I've been out of the retroprogramming hobby for a while. I'm feeling a little rusty. If I'm really out of RAM, maybe I can tinker with Watt32 to make it less memory hungry at the expense of performance. I could also have OW optimize for size instead of speed to possibly gain another 10-20 KB...

Mike, welcome back. Any new games on the horizon?
 
Figuring out how much memory is available from the DOS pool is a little complicated. The problem is that memory gets fragmented, so you have to crawl the memory allocation chain to see if there's a free block large enough. For example, try doing "mem /p" or "mem /fu" and look at the result:
Code:
Segment Total Name Type
------- ---------------- ------------ -------------
0000 1,024 (1K) interrupt vector table
0040 768 (1K) BIOS data area
0070 8,576 (8K) IO system data
0288 368 (0K) DOS system data
028a 192 (0K) FILES FILES=40 (3 in this block)
0297 144 (0K) EMS device driver
02a0 176 (0K) free
02ac 656 (1K) HXLDR32 program
02d6 352 (0K) MEM environment
02ed 55,008 (54K) MEM program
105c 588,320 (575K) free
a000 135,168 (132K) reserved
c100 5,024 (5K) DOS system data
c102 784 (1K) CDROM device driver
c134 1,904 (2K) FILES FILES=40 (32 in this block)
c1ac 2,288 (2K) LASTDRV LASTDRIVE=Z
c23b 96 (0K) free
c242 3,008 (3K) COMMAND program
c2ff 131,808 (129K) free
e32e 2,304 (2K) free
e3bf 1,024 (1K) COMMAND environment

When will someone fix the paste into "code" tags so that spaces aren't compressed?
 
If you run a mem command, how is the "Largest executable program size"?

I'm not familiar with OW, but if you are using a large model and not something with an extender, then what OW will manage is a heap in conventional memory. If your program size is 400K already, then that doesn't leave a lot of room - even if you have 600K conventional, that is only 200K available.

Are you compiling with debug information? In Borland, debug info makes the executable larger.
 
That's the largest contiguous block of memory available. However, that doesn't account for fragmentation that may occur with repeated malloc() and free() calls. It's perfectly possible to fragment, say, 400K of memory such that a call for 64K allocation will always fail. The point is that when malloc() is called, it returns a pointer to memory that is unchanged throughout program execution. BASIC on the other hand, dynamically allocates string space and can defragment on the fly by tracking the location of all string variables. C++, like BASIC usually has its own space management and can "garbage collect" on the fly.
 
Mike, welcome back. Any new games on the horizon?

Thank you! I never really wrote any games though. Are you thinking of another Mike? There are a lot of us. :)


Figuring out how much memory is available from the DOS pool is a little complicated. The problem is that memory gets fragmented, so you have to crawl the memory allocation chain to see if there's a free block large enough. For example, try doing "mem /p" or "mem /fu" and look at the result:
Code:
Segment Total Name Type
------- ---------------- ------------ -------------
0000 1,024 (1K) interrupt vector table
0040 768 (1K) BIOS data area
0070 8,576 (8K) IO system data
0288 368 (0K) DOS system data
028a 192 (0K) FILES FILES=40 (3 in this block)
0297 144 (0K) EMS device driver
02a0 176 (0K) free
02ac 656 (1K) HXLDR32 program
02d6 352 (0K) MEM environment
02ed 55,008 (54K) MEM program
105c 588,320 (575K) free
a000 135,168 (132K) reserved
c100 5,024 (5K) DOS system data
c102 784 (1K) CDROM device driver
c134 1,904 (2K) FILES FILES=40 (32 in this block)
c1ac 2,288 (2K) LASTDRV LASTDRIVE=Z
c23b 96 (0K) free
c242 3,008 (3K) COMMAND program
c2ff 131,808 (129K) free
e32e 2,304 (2K) free
e3bf 1,024 (1K) COMMAND environment

When will someone fix the paste into "code" tags so that spaces aren't compressed?

I think this is the secret sauce I was missing. I knew there must be a way to do this by poking at DOS structures, because I've seen memory usage maps before. Looks like memory control blocks are what I need to be reading!​ Thanks, Chuck.



If you run a mem command, how is the "Largest executable program size"?

I'm not familiar with OW, but if you are using a large model and not something with an extender, then what OW will manage is a heap in conventional memory. If your program size is 400K already, then that doesn't leave a lot of room - even if you have 600K conventional, that is only 200K available.

Are you compiling with debug information? In Borland, debug info makes the executable larger.

I'll need to get on my XT tonight and try that. Maybe I just need to trim the fat from this program and get the executable as small as I can. I will also be looking at Watt-32. I believe it increased the default socket buffer size allocation from the 2 KB that WatTCP used. 2 KB buffers would be fine for me.

This is a port of an IRC server, ngIRCd 26.1. I don't need huge buffers for IRC. It runs surprisingly well at 4.77 MHz, btw. If I can get it handling a couple dozen concurrent connections with 640 KB, I think I'll be pretty happy.
 
I managed to shrink the executable by 50 KB, and I've decreased the TCP receive buffer per connection from 5840 bytes to 1460 bytes, equal to the MSS. In theory, I should be able to handle dozens of connections in 640 KB now as long as memory isn't very fragmented.

I'll get the new build onto my XT tonight for testing. I have a few people who like vintage machines hanging out on there already. Maybe I'll advertise the server here once I get the new build up if anybody else wants to drop in for the novelty of being on an 8088-powered IRC server. :p

Only remaining issue is that occasionally it doesn't catch a client the first time it tries to connect, and they need to retry. Not sure if this is a Watt-32/WatTCP quirk, or if I'm doing something wrong. It's probably me.

I would have used Mike Brutman's mTCP stack, but Watt-32 already has all the wrapper functions I needed to port over programs that use BSD-style sockets interfaces.
 
Back
Top