blog.dbrgn.ch - pytesthttps://blog.dbrgn.ch/2016-02-18T00:00:00+01:00Overriding Default Arguments in Python2016-02-18T00:00:00+01:002016-02-18T00:00:00+01:00Danilo Bargentag:blog.dbrgn.ch,2016-02-18:/2016/2/18/overriding_default_arguments_in_pytest/<p class="first last">How to override the default arguments of a function.</p>
<p>Sometimes you want to change the behavior of a function call in a Python test.
Let's assume you have the following code:</p>
<div class="highlight"><pre><span></span><span class="c1"># a.py</span>
<span class="kn">from</span> <span class="nn">b</span> <span class="kn">import</span> <span class="n">subfunc</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="c1"># do something</span>
<span class="n">subfunc</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="c1"># do something else</span>
<span class="c1"># b.py</span>
<span class="k">def</span> <span class="nf">subfunc</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="c1"># step1</span>
<span class="c1"># step2</span>
<span class="c1"># step3</span>
</pre></div>
<p>You are testing the <tt class="docutils literal">func</tt> function and would to change the behavior of step2
in <tt class="docutils literal">subfunc</tt> without affecting step1 or step3.</p>
<div class="section" id="mocking-replacement-function">
<h2>Mocking: Replacement Function</h2>
<p>One way to solve this would be to mock the entire <tt class="docutils literal">subfunc</tt>:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_func</span><span class="p">(</span><span class="n">monkeypatch</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">_mocked</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="c1"># step1</span>
<span class="c1"># step3</span>
<span class="n">monkeypatch</span><span class="o">.</span><span class="n">setattr</span><span class="p">(</span><span class="s1">'b.subfunc'</span><span class="p">,</span> <span class="n">_mocked</span><span class="p">)</span>
<span class="c1"># do testing of func()</span>
</pre></div>
<p>(Note, all example code assumes that you're using <a class="reference external" href="http://pytest.org/">pytest</a> with the
<a class="reference external" href="https://pytest.org/latest/monkeypatch.html">monkeypatch</a> fixture. But you can also use other testing frameworks and the
<a class="reference external" href="https://docs.python.org/3/library/unittest.mock.html">mock</a> library instead.)</p>
<p>But that would require you to copy the body of the function and adjust it as
desired. This violates the DRY principle and could be a source of bugs (e.g. if
step1 and step3 change later on).</p>
</div>
<div class="section" id="dependency-injection">
<h2>Dependency Injection</h2>
<p>A cleaner way to make the <tt class="docutils literal">subfunc</tt> more dynamic is dependency injection. We
simply add a new argument with a default value, and act depending on that value:</p>
<div class="highlight"><pre><span></span><span class="c1"># b.py</span>
<span class="k">def</span> <span class="nf">subfunc</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">do_step2</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="c1"># step1</span>
<span class="k">if</span> <span class="n">do_step2</span> <span class="ow">is</span> <span class="kc">True</span><span class="p">:</span>
<span class="c1"># step2</span>
<span class="c1"># step3</span>
</pre></div>
<p>Now we can simply manipulate the value of the <tt class="docutils literal">do_step</tt> parameter to change
the behavior. But how do we actually do that? Two methods come to mind:</p>
<div class="section" id="monkey-patching-defaults">
<h3>Monkey patching __defaults__</h3>
<p>Every Python function with default arguments has a <tt class="docutils literal">__defaults__</tt> attribute.
It contains a tuple with all default argument values:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">def</span> <span class="nf">subfunc</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">do_step2</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="o">...</span> <span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">do_step2</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">subfunc</span><span class="o">.</span><span class="vm">__defaults__</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span>
</pre></div>
<p>We can manipulate that attribute to change the defaults:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_func</span><span class="p">(</span><span class="n">monkeypatch</span><span class="p">):</span>
<span class="n">monkeypatch</span><span class="o">.</span><span class="n">setattr</span><span class="p">(</span><span class="s1">'b.subfunc.__defaults__'</span><span class="p">,</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="kc">False</span><span class="p">))</span>
<span class="c1"># do testing of func()</span>
</pre></div>
<p>This works nicely, there are two downsides though. First of all, it's hacky due
to the use of double underscore methods. But even worse, we have to specify the
default argument for the first kwarg too! That violates the DRY principle and
could be a source of bugs. Sounds familiar, right?</p>
<p>Of course, we could try to retrieve the initial defaults, manipulate them and
then monkey patch the <tt class="docutils literal">__defaults__</tt> attribute again. But that's even more
hacky...</p>
</div>
<div class="section" id="using-a-partial-function">
<h3>Using a partial function</h3>
<p>A much nicer way is to use partial function application. It's a method mainly
coming from functional programming. You can use it to override the value of some
arguments and/or keyword arguments, yielding a higher order function.</p>
<p>As a short example, let's create a function that adds 2 to an input value:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">partial</span>
<span class="c1"># Regular add function</span>
<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="k">assert</span> <span class="n">add</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">42</span>
<span class="c1"># Partial function that overrides b</span>
<span class="n">add_two</span> <span class="o">=</span> <span class="n">partial</span><span class="p">(</span><span class="n">add</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">add_two</span><span class="p">(</span><span class="mi">40</span><span class="p">)</span> <span class="o">==</span> <span class="mi">42</span>
</pre></div>
<p>Now that you know how partial functions work, let's use them to override the
default argument of our <tt class="docutils literal">subfunc</tt>:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">partial</span>
<span class="k">def</span> <span class="nf">test_func</span><span class="p">(</span><span class="n">monkeypatch</span><span class="p">):</span>
<span class="kn">from</span> <span class="nn">b</span> <span class="kn">import</span> <span class="n">subfunc</span>
<span class="n">monkeypatch</span><span class="o">.</span><span class="n">setattr</span><span class="p">(</span><span class="s1">'b.subfunc'</span><span class="p">,</span> <span class="n">partial</span><span class="p">(</span><span class="n">subfunc</span><span class="p">,</span> <span class="n">do_step2</span><span class="o">=</span><span class="kc">False</span><span class="p">))</span>
<span class="c1"># do testing of func()</span>
</pre></div>
<p>Now we have a clean way to modify function behavior from tests through
dependency injection, without having to resort to Python internals hackery :)</p>
</div>
</div>
Profiling Pytest Tests2014-09-08T00:00:00+02:002014-09-08T00:00:00+02:00Danilo Bargentag:blog.dbrgn.ch,2014-09-08:/2014/9/8/profiling_pytest_tests/<p class="first last">How to run pytest with cProfile.</p>
<p>If you want to <a class="reference external" href="http://stackoverflow.com/q/18830232/284318">run a profiler</a>
over your <a class="reference external" href="http://pytest.org/">pytest</a> cases (including underlying code), the easiest way is to run
pytest directly using cPython:</p>
<pre class="literal-block">
python -m cProfile -o profile $(which py.test)
</pre>
<p>You can even pass in optional arguments:</p>
<pre class="literal-block">
python -m cProfile -o profile $(which py.test) \
tests/worker/test_tasks.py -s campaigns
</pre>
<p>This will create a binary file called <tt class="docutils literal">profile</tt> in your current directory.
The file can then be analyzed with <a class="reference external" href="https://docs.python.org/2/library/profile.html#module-pstats">pstats</a>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pstats</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">pstats</span><span class="o">.</span><span class="n">Stats</span><span class="p">(</span><span class="s1">'profile'</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">strip_dirs</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">sort_stats</span><span class="p">(</span><span class="s1">'tottime'</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">print_stats</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span>
</pre></div>
<p>This will print the 50 lines with the longest total duration. You can also sort
by other columns, e.g. <tt class="docutils literal">'cumtime'</tt>.</p>