memory_profiling

Module: tools.memory_profiling

External profiling code Imported by Jan-Hendrik Metzen

This module exposes 10 functions and 2 classes to obtain lengths and sizes of Python objects (for Python 2.2 or later [1]).

The main changes in this version are new function calcsize(), use gc.get_objects() to get all objects and improvements in this documentation.

Public Functions [2]

Function asizeof calculates the combined (approximate) size in bytes of one or several Python objects.

Function asizesof returns a tuple containing the (approximate) size in bytes for each given Python object separately.

Function asized returns for each object an instance of class Asized containing all the size information of the object and a tuple with the referents.

Functions basicsize and itemsize return the basic respectively item size of the given object.

Function flatsize returns the flat size of a Python object in bytes defined as the basic size plus the item size times the length of the given object.

Function leng returns the length of an object, like standard len but extended for several types, e.g. the leng of a multi- precision int (or long) is the number of digits [3]. The length of most mutable sequence objects includes an estimate of the over-allocation and therefore, the leng value may differ from the standard len result.

Function refs returns (a generator for) the referents of the given object, i.e. the objects referenced by the given object.

Function calcsize is equivalent to standard struct.calcsize but handles format characters ‘z’ for signed C type Py_ssize_t and ‘Z’ for unsigned C type size_t.

Certain classes are known to be sub-classes of or to behave as dict objects. Function adict can be used to install other class objects to be treated like dict.

Public Classes [2]

An instance of class Asized is returned for each object sized with the asized function or method.

Class Asizer can be used to accumulate the results of several asizeof or asizesof calls. After creating an Asizer instance, use methods asizeof and asizesof to size additional objects.

Call methods exclude_refs and/or exclude_types to exclude references to or instances or types of certain objects.

Use one of the print_... methods to report the statistics.

Duplicate Objects

Any duplicate, given objects are sized only once and the size is included in the combined total only once. But functions asizesof and asized do return a size value respectively an Asized instance for each given object, the same for duplicates.

Definitions [4]

The size of an object is defined as the sum of the flat size of the object plus the sizes of any referents. Referents are visited recursively up to a given limit. However, the size of objects referenced multiple times is included only once.

The flat size of an object is defined as the basic size of the object plus the item size times the number of allocated items. The flat size does include the size for the items (references to the referents), but not the referents themselves.

The flat size returned by function flatsize equals the result of the asizeof function with options code=True, ignored=False, limit=0 and option align set to the same value.

The accurate flat size for an object is obtained from function sys.getsizeof() where available. Otherwise, the length and size of sequence objects as dicts, lists, sets, etc. is based on an estimate for the number of allocated items. As a result, the reported length and size may substantially differ from the actual length and size.

The basic and item sizes are obtained from the __basicsize__ respectively __itemsize__ attribute of the (type of the) object. Where necessary (e.g. sequence objects), a zero __itemsize__ is replaced by the size of a corresponding C type.

The basic size (of GC managed objects) objects includes the overhead for Python’s garbage collector (GC) as well as the space needed for refcounts (only in certain Python builds).

Optionally, sizes can be aligned to any power of 2 multiple.

Size of (byte)code

The (byte)code size of objects as classes, functions, methods, modules, etc. can be included by setting option code.

Iterators are handled similar to sequences: iterated object(s) are sized like referents if the recursion limit permits. Also, function gc.get_referents() must return the referent object of iterators.

Generators are sized as (byte)code only, but generated objects are never sized.

Old- and New-style Classes

All old- and new-style class, instance and type objects, are handled uniformly such that (a) instance and class objects can be distinguished and (b) instances of different old-style classes can be dealt with separately.

Class and type objects are represented as <class ....* def> respectively as <type ... def> where an ‘*’ indicates an old- style class and the def suffix marks the definition object. Instances of old-style classes are shown as new-style ones but with an ‘*’ at the end of the name, like <class module.name*>.

Ignored Objects

To avoid excessive sizes, several object types are ignored [4] by default, e.g. built-in functions, built-in types and classes [5], function globals and module referents. However, any instances thereof are sized and module objects will be sized when passed as given objects. Ignored object types are included if option ignored is set accordingly.

In addition, many __...__ attributes of callable objects are ignored, except crucial ones, e.g. class attributes __dict__, __doc__, __name__ and __slots__. For more details, see the type-specific _..._refs() and _len_...() functions below.

Option all can be used to size all Python objects and/or get the referents from gc.get_referents() and override the type- specific __..._refs() functions.

Note

[1] Tested with Python 2.2.3, 2.3.7, 2.4.5, 2.5.1, 2.5.2, 2.6.2,

3.0.1 or 3.1a2 on CentOS 4.6, SuSE 9.3, MacOS X 10.4.11 Tiger (Intel) and 10.3.9 Panther (PPC), Solaris 10 (Opteron) and Windows XP all 32-bit Python and on RHEL 3u7 and Solaris 10 (Opteron) both 64-bit Python.

[2] The functions and classes in this module are not thread-safe.

[3] See Python source file .../Include/longinterp.h for the

C typedef of digit used in multi-precision int (or long) objects. The size of digit in bytes can be obtained in Python from the int (or long) __itemsize__ attribute. Function leng (rather _len_int) below deterimines the number of digits from the int (or long) value.

[4] These definitions and other assumptions are rather arbitrary

and may need corrections or adjustments.

[5] Types and classes are considered built-in if the module of

the type or class is listed in _builtin_modules below.

Original BSD license:

---------------------------------------------------------------------
       Copyright (c) 2002-2009 -- ProphICy Semiconductor, Inc.
                        All rights reserved.

 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions
 are met:

 - Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

 - Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in
   the documentation and/or other materials provided with the
   distribution.

 - Neither the name of ProphICy Semiconductor, Inc. nor the names
   of its contributors may be used to endorse or promote products
   derived from this software without specific prior written
   permission.

 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
 FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE
 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 OF THE POSSIBILITY OF SUCH DAMAGE.
---------------------------------------------------------------------

This is the latest, enhanced version of the asizeof.py recipes at <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/546530> <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/544288>

Inheritance diagram for pySPACE.tools.memory_profiling:

Class Summary

Asized(size, flat[, refs, name])	Store the results of an asized object
Asizer(\*\*opts)	Sizer state and options.

Function Summary

calcsize(fmt)	struct.calcsize() handling ‘z’ for signed Py_ssize_t and ‘Z’ for unsigned size_t.
adict(\*classes)	Install one or more classes to be handled as dict.
asized(\*objs, \*\*opts)	Return a tuple containing an Asized instance for each object passed as positional argment using the following options.
asizeof(\*objs, \*\*opts)	Return the combined size in bytes of all objects passed as positional argments.
asizesof(\*objs, \*\*opts)	Return a tuple containing the size in bytes of all objects passed as positional argments using the following options.
basicsize(obj, \*\*opts)	Return the basic size of an object (in bytes).
flatsize(obj[, align])	Return the flat size of an object (in bytes), optionally aligned to a given power of 2.
itemsize(obj, \*\*opts)	Return the item size of an object (in bytes).
leng(obj, \*\*opts)	Return the length of an object (in items).
refs(obj[, all])	Return (a generator for) specific referents of an object.

Classes

Asized

class pySPACE.tools.memory_profiling.Asized(size, flat, refs=(), name=None)

Bases: object

Store the results of an asized object in these 4 attributes:

Size:total size of the object Flat:flat size of the object Name:name or repr of the object Refs:tuple containing an instance of Asized for each referent

Class Components Summary

__str__()
format(named)	Format name from _NamedRef instance.
strf

strf = ('%s', '[K] %s', '[V] %s')

__init__(size, flat, refs=(), name=None)

__str__()

format(named)

Format name from _NamedRef instance.

__weakref__

list of weak references to the object (if defined)

Asizer

class pySPACE.tools.memory_profiling.Asizer(**opts)

Bases: object

Sizer state and options.

Class Components Summary

_align_
_all_
_clear()	Clear state.
_clip_
_code_
_cutoff
_depth
_derive_
_detail_
_duplicate
_excl_d
_get_duplicate()	Number of duplicate objects.
_get_missed()	Number of objects missed due to errors.
_get_total()	Total size accumulated so far.
_ign_d
_incl
_infer_
_limit_
_mask
_missed
_nameof(obj)	Return the object’s name.
_prepr(obj)	Like prepr().
_prof(key)	Get _Prof object.
_profile
_profs
_repr(obj)	Like repr().
_seen
_sizer(obj, deep, sized)	Size an object, recursively.
_sizes(objs[, sized])	Return the size or an Asized instance for each given object and the total size.
_stats_
_total
asized(\*objs, \*\*opts)	Size each object and return an Asized instance with size information and referents up to the given detail level (and with modified options, see method set).
asizeof(\*objs, \*\*opts)	Return the combined size of the given objects (with modified options, see also method set).
asizesof(\*objs, \*\*opts)	Return the individual sizes of the given objects (with modified options, see also method set).
duplicate	Number of duplicate objects.
exclude_refs(\*objs)	Exclude any references to the specified objects from sizing.
exclude_types(\*objs)	Exclude the specified object instances and types from sizing.
missed	Number of objects missed due to errors.
print_profiles([w, cutoff])	Print the profiles above cutoff percentage.
print_stats([objs, opts, sized, sizes, stats])	Print the statistics.
print_summary([w, objs])	Print the summary statistics.
print_typedefs([w])	Print the types and dict tables.
reset([align, all, clip, code, derive, ...])	Reset options, state, etc.
set([align, code, detail, limit, stats])	Set some options.
total	Total size accumulated so far.

_align_ = 8

_all_ = False

_clip_ = 80

_code_ = False

_derive_ = False

_detail_ = 0

_infer_ = False

_limit_ = 100

_stats_ = 0

_cutoff = 0

_depth = 0

_duplicate = 0

_ign_d = 'ignored'

_incl = ''

_mask = 7

_missed = 0

_profile = False

_profs = None

_seen = None

_total = 0

__init__(**opts)

See method reset for the available options.

_excl_d = None

_clear()

Clear state.

_nameof(obj)

Return the object’s name.

_prepr(obj)

Like prepr().

_prof(key)

Get _Prof object.

_repr(obj)

Like repr().

_sizer(obj, deep, sized)

Size an object, recursively.

_sizes(objs, sized=None)

Return the size or an Asized instance for each given object and the total size. The total includes the size of duplicates only once.

asized(*objs, **opts)

Size each object and return an Asized instance with size information and referents up to the given detail level (and with modified options, see method set).

If only one object is given, the return value is the Asized instance for that object.

asizeof(*objs, **opts)

Return the combined size of the given objects (with modified options, see also method set).

asizesof(*objs, **opts)

Return the individual sizes of the given objects (with modified options, see also method set).

exclude_refs(*objs)

Exclude any references to the specified objects from sizing.

While any references to the given objects are excluded, the objects will be sized if specified as positional arguments in subsequent calls to methods asizeof and asizesof.

exclude_types(*objs)

Exclude the specified object instances and types from sizing.

All instances and types of the given objects are excluded, even objects specified as positional arguments in subsequent calls to methods asizeof and asizesof.

print_profiles(w=0, cutoff=0, **print3opts)

Print the profiles above cutoff percentage.

w=0 – indentation for each line cutoff=0 – minimum percentage printed print3options – print options, as in Python 3.0

print_stats(objs=(), opts={}, sized=(), sizes=(), stats=3.0, **print3opts)

Print the statistics.

w=0 – indentation for each line objs=() – optional, list of objects opts={} – optional, dict of options used sized=() – optional, tuple of Asized instances returned sizes=() – optional, tuple of sizes returned stats=3.0 – print statistics and cutoff percentage print3options – print options, as in Python 3.0

print_summary(w=0, objs=(), **print3opts)

Print the summary statistics.

w=0 – indentation for each line objs=() – optional, list of objects print3options – print options, as in Python 3.0

print_typedefs(w=0, **print3opts)

Print the types and dict tables.

w=0 – indentation for each line print3options – print options, as in Python 3.0

set(align=None, code=None, detail=None, limit=None, stats=None)

Set some options. Any options not set remain the same as the previous setting.

align=8 – size alignment code=False – incl. (byte)code size detail=0 – Asized refs level limit=100 – recursion limit stats=0.0 – print statistics and cutoff percentage

_get_duplicate()

Number of duplicate objects.

duplicate

Number of duplicate objects.

_get_missed()

Number of objects missed due to errors.

missed

Number of objects missed due to errors.

_get_total()

Total size accumulated so far.

total

Total size accumulated so far.

reset(align=8, all=False, clip=80, code=False, derive=False, detail=0, ignored=True, infer=False, limit=100, stats=0)

Reset options, state, etc.

The available options and default values are:

align=8 – size alignment all=False – all current GC objects and referents clip=80 – clip repr() strings code=False – incl. (byte)code size derive=False – derive from super type detail=0 – Asized refs level ignored=True – ignore certain types infer=False – try to infer types limit=100 – recursion limit stats=0.0 – print statistics and cutoff percentage

See function asizeof for a description of the options.

__weakref__

list of weak references to the object (if defined)

Functions

calcsize()

pySPACE.tools.memory_profiling.calcsize(fmt)

struct.calcsize() handling ‘z’ for signed Py_ssize_t and ‘Z’ for unsigned size_t.

adict()

pySPACE.tools.memory_profiling.adict(*classes)

Install one or more classes to be handled as dict.

asized()

pySPACE.tools.memory_profiling.asized(*objs, **opts)

Return a tuple containing an Asized instance for each object passed as positional argment using the following options.

align=8 – size alignment all=False – all current GC objects and referents clip=80 – clip repr() strings code=False – incl. (byte)code size derive=False – derive from super type detail=0 – Asized refs level ignored=True – ignore certain types infer=False – try to infer types limit=100 – recursion limit stats=0.0 – print statistics and cutoff percentage

If only one object is given, the return value is the Asized instance for that object.

Set detail to the desired referents level (recursion depth).

See function asizeof for descriptions of the other options.

The length of the returned tuple matches the number of given objects, if more than one object is given.

asizeof()

pySPACE.tools.memory_profiling.asizeof(*objs, **opts)

Return the combined size in bytes of all objects passed as positional argments.

The available options and defaults are the following.

align=8 – size alignment all=False – all current GC objects and referents clip=80 – clip repr() strings code=False – incl. (byte)code size derive=False – derive from super type ignored=True – ignore certain types infer=False – try to infer types limit=100 – recursion limit stats=0.0 – print statistics and cutoff percentage

Set align to a power of 2 to align sizes. Any value less than 2 avoids size alignment.

All current GC objects are sized if all is True and if no positional arguments are supplied. Also, if all is True the GC referents are used instead of the limited ones.

A positive clip value truncates all repr() strings to at most clip characters.

The (byte)code size of callable objects like functions, methods, classes, etc. is included only if code is True.

If derive is True, new types are handled like an existing (super) type provided there is one and only of those.

By default, certain base types like object are ignored for sizing. Set ignored to False to force all ignored types in the size of objects.

By default certain base types like object, super, etc. are ignored. Set ignored to False to include those.

If infer is True, new types are inferred from attributes (only implemented for dict types on callable attributes as get, has_key, items, keys and values).

Set limit to a positive value to accumulate the sizes of the referents of each object, recursively up to the limit. Using limit zero returns the sum of the flat [1] sizes of the given objects. High limit values may cause runtime errors and miss objects for sizing.

A positive value for stats prints up to 8 statistics, (1) a summary of the number of objects sized and seen, (2) a simple profile of the sized objects by type and (3+) up to 6 tables showing the static, dynamic, derived, ignored, inferred and dict types used, found respectively installed.

The fractional part of the stats value (x 100) is the cutoff percentage for simple profiles. Objects below the cutoff value are not reported.

[1] See the documentation of this module for the definition

of flat size.

asizesof()

pySPACE.tools.memory_profiling.asizesof(*objs, **opts)

Return a tuple containing the size in bytes of all objects passed as positional argments using the following options.

align=8 – size alignment all=False – use GC objects and referents clip=80 – clip repr() strings code=False – incl. (byte)code size derive=False – derive from super type ignored=True – ignore certain types infer=False – try to infer types limit=100 – recursion limit stats=0.0 – print statistics and cutoff percentage

See function asizeof for a description of the options.

The length of the returned tuple equals the number of given objects.

basicsize()

pySPACE.tools.memory_profiling.basicsize(obj, **opts)

Return the basic size of an object (in bytes).

Valid options and defaults are

derive=False – derive type from super type infer=False – try to infer types save=False – save typedef if new

flatsize()

pySPACE.tools.memory_profiling.flatsize(obj, align=0, **opts)

Return the flat size of an object (in bytes), optionally aligned to a given power of 2.

See function basicsize for a description of the other options. See the documentation of this module for the definition of flat size.

itemsize()

pySPACE.tools.memory_profiling.itemsize(obj, **opts)

Return the item size of an object (in bytes).

See function basicsize for a description of the options.

leng()

pySPACE.tools.memory_profiling.leng(obj, **opts)

Return the length of an object (in items).

See function basicsize for a description of the options.

refs()

pySPACE.tools.memory_profiling.refs(obj, all=False, **opts)

Return (a generator for) specific referents of an object.

If all is True return the GC referents.

See function basicsize for a description of the options.