Conference turris::digital_unix

9605.0. "3.2C, 3.2G RSS shrinks after deallocate, but not 4.0...intended?" by PEACHS::DALEY ('Course it don't come with @!# wafers!) Wed Apr 23 1997 17:39

Hello,

This is very weird, and may belong in digital_unix or Fortran.  But it has
to do with VSZ and RSS, so I thought I'd start here.

Below is some code.  Here's the problem:

If I run this on UNIX 3.2G with DFARTL371 installed, the size of the program
in memory shrinks after it deallocates the memory and *does something*.  The
VSZ and RSS shrink back to the size that it was before it allocated the
memory.  Doesn't matter if swap is lazy or immediate mode.

So in other words, it you catch it with ps alx between allocating and 
deallocating memory, then catch it after deallocating but before it exits,
you'll see that RSS shrinks.

The same does NOT happen in UNIX 4.0A, 4.0B.  After allocation, the 
virtual size and RSS stay higher until the process exits.

Question:  Is this intended?  Looks like maybe a performance enhancement to 
me.  Maybe 4.0B was *designed* not to give the memory back, because you might
need it again really soon.

Customer's problem is that his real app eats up tons of memory and it was nice
to get RSS size down every once int a while, which he sees with 3.2(3.2C in
his case).  RSS is all he cares about in this, not VSZ.


Credits and Liner Notes:

UNIX 4.0A and 4.0B have DFARTL374 installed.  Using F90.  Compiling
-non_shared on 4.0B causes run on3.2G to also fail to shrink the memory, but 
the issue could be with 4.0 libc?  

Any insight is appreciated.  -John

Build following thus:  # f90 -free -o memtest memtest.f
-------------------------------------------------------------------------
MODULE prec
  
  IMPLICIT NONE
  
  INTEGER, PARAMETER :: precision = SELECTED_REAL_KIND (P=12)
  
END MODULE prec




PROGRAM test

  USE prec

  IMPLICIT NONE

  INTEGER, PARAMETER :: MEMORY= 30000

  INTEGER, PARAMETER :: DOMAX= 25000

  INTEGER, ALLOCATABLE, DIMENSION (:) :: int

  REAL (KIND= precision), ALLOCATABLE, DIMENSION (:,:) :: rl

  INTEGER :: i, j, k, alloc_error

  REAL (KIND= precision), DIMENSION (0:9) :: a= (/1., 2., 3., 4., 5., 6., 7., 8.
, 9., 0. /)

  REAL (KIND= precision) :: b, c, d, e, f
  

  WRITE (*,*)
  WRITE (*,*) 'Program Executing; Wait a While'
  WRITE (*,*) 

  DO j= 1, DOMAX
     DO i= 1, DOMAX
        
        DO k= 1, 5

           b= a (0) * a (1)
           c= a (2) * a (3)
           d= a (4) * a (5)
           e= a (6) * a (7)
           f= a (8) * a (9)

        END DO
     END DO
  END DO


  WRITE (*,*) 'Allocating memory'
  WRITE (*,*) 
  WRITE (*,*) 'Hit RETURN to Continue'
   READ (*,*) 
  WRITE (*,*) 'Program Continuing; Wait a While'
  WRITE (*,*) 
   
  ALLOCATE (INT (MEMORY), rl (MEMORY,5), STAT= alloc_error)

  int= 2

  rl (:,1)= 1.
  rl (:,2)= 2.
  rl (:,4)= 4.
  rl (:,5)= 5.


  WRITE (*,*) 'Memory allocated'
  WRITE (*,*) 
  WRITE (*,*) 'Hit RETURN to Continue'
   READ (*,*) 
  WRITE (*,*) 'Program Continuing; Wait a While'
  WRITE (*,*)      


  DO j= 1, DOMAX
     DO i= 1, DOMAX
        
        DO k= 1, 5

           b= a (0) * a (1) * int (i) * rl (j,1)
           c= a (2) * a (3) * int (i) * rl (j,2)
           d= a (4) * a (5) * int (i) * rl (j,3)
           e= a (6) * a (7) * int (i) * rl (j,4)
           f= a (8) * a (9) * int (i) * rl (j,5)

        END DO
     END DO
  END DO


  DEALLOCATE (INT, rl, STAT= alloc_error)



  WRITE (*,*) 'Memory dealloctaed'
  WRITE (*,*) 
  WRITE (*,*) 'Hit RETURN to Continue'
   READ (*,*) 
  WRITE (*,*) 'Program Continuing; Wait a While'
  WRITE (*,*) 

  DO j= 1, DOMAX
     DO i= 1, DOMAX
        
        DO k= 1, 25

           b= a (0) * a (1)
           c= a (2) * a (3)
           d= a (4) * a (5)
           e= a (6) * a (7)
           f= a (8) * a (9)

        END DO
     END DO
  END DO


  WRITE (*,*) 'Program Finished'
  WRITE (*,*) 

END PROGRAM test

>but the issue could be with 4.0 libc?

If the allocate/deallocate translates to malloc()/free(), then it is possible
that you are seeing the effect of a libc change.  In V4.0, libc uses the
sbrk() system call, whereas in V3.2 it used a shared library version.  Under
certain conditions free() will do a -ve sbrk to get rid of virtual (and any
resident) memory.  The difference is that the sbrk() system call does not do
-ve sbrk.  The v3.2 library version of sbrk() uses munmap() for -ve sbrk.

There was an attempt to enable -ve sbrk in the system call, but it resulted
in severe performance degradations in standard benchmarks.  I believe this
was due to shells' use of sbrk.

--shashi
[Posted by WWW Notes gateway]

Thanks for the note, Shashi,

Just to be clear,  about malloc and free: there are no guarantees that 
free()ing memory will return it to the operating system.  If you malloc a 
block of memory, it's likely to stay yours, and effect RZZ, VSZ and
possibly swap space for as long as your process lives.

So don't count on free() to change your processes memory allocations.

This is even more the case in v4.0n of UNIX, and is done for performance 
reasons.

Is the above correct?

-JD

>Just to be clear,  about malloc and free: there are no guarantees that 
>free()ing memory will return it to the operating system.

Even under V3.2C, I believe free() returns memory to the OS only under
certain conditions.

>If you malloc a 
>block of memory, it's likely to stay yours, and effect RZZ, VSZ and
>possibly swap space for as long as your process lives.

The RSS may change if the system is low in memory as the pageout daemon
will reclaim most of it.

>So don't count on free() to change your processes memory allocations.

>This is even more the case in v4.0n of UNIX, and is done for performance 
>reasons.

That is debatable.  As I said, in V3.2C free() did return memory to the
system.  The performance problem was with changing sbrk() behaviour,which is
used by the shells.  In V4.0 malloc() also switched to use the sbrk() system
call.  See note 9843 on how you can get old free() behaviour back.  You may
or maynot see a performance problem from the shells.

--shashi

[Posted by WWW Notes gateway]