GNU Info

(python2.1-api.info)Overview

---

Next: Memory Interface Prev: Memory Management Up: Memory Management

---

Enter node , (file) or (file)node

Overview
========

Memory management in Python involves a private heap containing all
Python objects and data structures. The management of this private heap
is ensured internally by the _Python memory manager_.  The Python
memory manager has different components which deal with various dynamic
storage management aspects, like sharing, segmentation, preallocation
or caching.

At the lowest level, a raw memory allocator ensures that there is
enough room in the private heap for storing all Python-related data by
interacting with the memory manager of the operating system. On top of
the raw memory allocator, several object-specific allocators operate on
the same heap and implement distinct memory management policies adapted
to the peculiarities of every object type. For example, integer objects
are managed differently within the heap than strings, tuples or
dictionaries because integers imply different storage requirements and
speed/space tradeoffs. The Python memory manager thus delegates some of
the work to the object-specific allocators, but ensures that the latter
operate within the bounds of the private heap.

It is important to understand that the management of the Python heap is
performed by the interpreter itself and that the user has no control on
it, even if she regularly manipulates object pointers to memory blocks
inside that heap.  The allocation of heap space for Python objects and
other internal buffers is performed on demand by the Python memory
manager through the Python/C API functions listed in this document.

To avoid memory corruption, extension writers should never try to
operate on Python objects with the functions exported by the C library:
`malloc()' , `calloc()' , `realloc()'  and `free()' .  This will result
in mixed calls between the C allocator and the Python memory manager
with fatal consequences, because they implement different algorithms
and operate on different heaps.  However, one may safely allocate and
release memory blocks with the C library allocator for individual
purposes, as shown in the following example:

         PyObject *res;
         char *buf = (char *) malloc(BUFSIZ); /* for I/O */
     
         if (buf == NULL)
             return PyErr_NoMemory();
         ...Do some I/O operation involving buf...
         res = PyString_FromString(buf);
         free(buf); /* malloc'ed */
         return res;

In this example, the memory request for the I/O buffer is handled by
the C library allocator. The Python memory manager is involved only in
the allocation of the string object returned as a result.

In most situations, however, it is recommended to allocate memory from
the Python heap specifically because the latter is under control of the
Python memory manager. For example, this is required when the
interpreter is extended with new object types written in C. Another
reason for using the Python heap is the desire to _inform_ the Python
memory manager about the memory needs of the extension module.  Even
when the requested memory is used exclusively for internal,
highly-specific purposes, delegating all memory requests to the Python
memory manager causes the interpreter to have a more accurate image of
its memory footprint as a whole. Consequently, under certain
circumstances, the Python memory manager may or may not trigger
appropriate actions, like garbage collection, memory compaction or
other preventive procedures. Note that by using the C library allocator
as shown in the previous example, the allocated memory for the I/O
buffer escapes completely the Python memory manager.