Skip to content

Commit f1cee23

Browse files
committed
added max child usage marker and update readme
1 parent aab15ed commit f1cee23

2 files changed

Lines changed: 54 additions & 8 deletions

File tree

README.rst

Lines changed: 43 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -107,10 +107,10 @@ decorator function. Use as follows::
107107
del b
108108
return a
109109

110-
If a python script with decorator ``@profile`` is called using ``-m
110+
If a python script with decorator ``@profile`` is called using ``-m
111111
memory_profiler`` in the command line, the ``precision`` parameter is ignored.
112112

113-
Time-based memory usage
113+
Time-based memory usage
114114
==========================
115115
Sometimes it is useful to have full memory usage reports as a function of
116116
time (not line-by-line) of external processes (be it Python scripts or not).
@@ -131,14 +131,14 @@ e.g. `mprof run -h`.
131131
In the case of a Python script, using the previous command does not
132132
give you any information on which function is executed at a given
133133
time. Depending on the case, it can be difficult to identify the part
134-
of the code that is causing the highest memory usage.
134+
of the code that is causing the highest memory usage.
135135

136136
Adding the `profile` decorator to a function and running the Python
137-
script with
137+
script with
138138

139139
mprof run <script>
140140

141-
will record timestamps when entering/leaving the profiled function. Runnning
141+
will record timestamps when entering/leaving the profiled function. Running
142142

143143
mprof plot
144144

@@ -152,16 +152,49 @@ A discussion of these capabilities can be found `here <http://fa.bianp.net/blog/
152152

153153
.. warning:: If your Python file imports the memory profiler `from memory_profiler import profile` these timestamps will not be recorded. Comment out the import, leave your functions decorated, and re-run.
154154

155-
The available commands for `mprof` are:
155+
The available commands for `mprof` are:
156156

157-
- ``mprof run``: running an executable, recording memory usage
157+
- ``mprof run``: running an executable, recording memory usage
158158
- ``mprof plot``: plotting one the recorded memory usage (by default,
159159
the last one)
160160
- ``mprof list``: listing all recorded memory usage files in a
161161
user-friendly way.
162162
- ``mprof clean``: removing all recorded memory usage files.
163163
- ``mprof rm``: removing specific recorded memory usage files
164164

165+
Tracking forked child processes
166+
===============================
167+
In a multiprocessing context the main process will spawn child processes whose
168+
system resources are allocated separately from the parent process. This can
169+
lead to an inaccurate report of memory usage since by default only the parent
170+
process is being tracked. The ``mprof`` utility provides two mechanisms to
171+
track the usage of child processes: sum the memory of all children to the
172+
parent's usage and track each child individual.
173+
174+
To create a report that combines memory usage of all the children and the
175+
parent, use the ``include_children`` flag in either the ``profile`` decorator or
176+
ass a command line argument to ``mprof``::
177+
178+
mprof run --include-children <script>
179+
180+
The second method tracks each child independently of the main process,
181+
serializing child rows by index to the output stream. Use the ``multiprocess``
182+
flag and plot as follows::
183+
184+
mprof run --multiprocess <script>
185+
mprof plot
186+
187+
This will create a plot using matplotlib similar to this:
188+
189+
.. image:: https://cloud.githubusercontent.com/assets/745966/24075879/2e85b43a-0bfa-11e7-8dfe-654320dbd2ce.png
190+
: target: https://github.com/fabianp/memory_profiler/pull/134
191+
: height: 350px
192+
193+
You can combine both the ``include_children`` and ``multiprocess`` flags to show
194+
the total memory of the program as well as each child individually.
195+
196+
.. warning:: currently the child tracking only works if a ``stream`` is provided to the ``profile`` (e.g. from the command line or in the decorator).
197+
165198
Setting debugger breakpoints
166199
=============================
167200
It is possible to set breakpoints depending on the amount of memory used.
@@ -260,7 +293,7 @@ LogFile of memory profiler module.
260293
>>> import sys
261294
>>> sys.stdout = LogFile('memory_profile_log')
262295

263-
``Customised reporting:``
296+
``Customized reporting:``
264297

265298
Sending everything to the log file while running the memory_profiler
266299
could be cumbersome and one can choose only entries with increments
@@ -412,6 +445,8 @@ cleanup.
412445

413446
`Dmitriy Novozhilov <https://github.com/demiurg906>`_ and `Sergei Lebedev <https://github.com/superbobry>`_ added support for `tracemalloc <https://docs.python.org/3/library/tracemalloc.html>`_.
414447

448+
`Benjamin Bengfort <https://github.com/bbengfort>`_ added support for tracking the usage of individual child processes and plotting them.
449+
415450
=========
416451
License
417452
=========

mprof

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -400,6 +400,8 @@ def plot_file(filename, index=0, timestamps=True, children=True, options=None):
400400

401401
# plot children, if any
402402
if len(chld) > 0 and children:
403+
cmpoint = (0,0) # maximal child memory
404+
403405
for idx, (proc, data) in enumerate(chld.items()):
404406
# Create the numpy arrays from the series data
405407
cts = np.asarray([item[1] for item in data]) - global_start
@@ -409,6 +411,15 @@ def plot_file(filename, index=0, timestamps=True, children=True, options=None):
409411
pl.plot(cts, cmem, "+-" + mem_line_colors[idx+1 % len(mem_line_colors)],
410412
label="child {}".format(proc))
411413

414+
# Detect the maximal child memory point
415+
cmax_mem = cmem.max()
416+
if cmax_mem > cmpoint[1]:
417+
cmpoint = (cts[cmem.argmax()], cmax_mem)
418+
419+
# Add the marker lines for the maximal child memory usage
420+
pl.vlines(cmpoint[0], pl.ylim()[0]+0.001, pl.ylim()[1] - 0.001, 'r', '--')
421+
pl.hlines(cmpoint[1], pl.xlim()[0]+0.001, pl.xlim()[1] - 0.001, 'r', '--')
422+
412423
# plot timestamps, if any
413424
if len(ts) > 0 and timestamps:
414425
func_num = 0

0 commit comments

Comments
 (0)