django on roschegel

Simplified Django Tests With Pytest and Pytest FactoryBoy

Sun, 17 May 2020 00:00:00 +0000

I’m always trying to find ways to make tests easier to read and extend. I hate working through a really hard feature and having to spend a big amount of time writing tests.

Lately, I’ve been bothered by the amount of boilerplate on my test code. So, I decided to do some research and look for alternatives.

When testing Django applications, I use a combination of Pytest fixtures and FactoryBoy to write tests that need database records.

Pytest Fixtures allow you to abstract the initialization of your test cases into functions (fixtures) and help you re-use code. Pytest uses dependency injection to detect when a fixture is needed on a test by matching the name provided on the parameter with the name of the fixture. Learn more about Pytest Fixtures on the Pytest documentation.
FactoryBoy allows you to create factories for your models with pre-defined values, which makes it easy to create model objects on the go. It handles relationships by allowing you to define subfactories and integrates with the faker library to provide the ability to use randomized values. It’s a good alternative to Django fixtures.

The Problem

Let’s look through an example to understand what we are trying to fix.

We will write a test case for the following function which returns a list of Post’s titles:

def get_posts_titles(include_unpublished=False):
    queryset = Post.objects.all()
    if not include_unpublished:
        queryset = queryset.filter(status="published")
    return  list(queryset.values_list("title", flat=True))

First, in factories.py we define the factory that will later be used to create the model objects on the test. In this case, it’s just a simple database with a Post model. So we just need one factory:

import factory

class PostFactory(factory.django.DjangoModelFactory):
    title = "Combining Pytest and FactoryBoy for simplified tests in Django"
    status = "draft"

    class Meta:
        model = "blog.Post"

Then in the test_posts.py file, we import the factory and create 2 fixtures that use it, post_draft and post_published:

import pytest

from factories import PostFactory


@pytest.fixture
def post_draft():
    return PostFactory()


@pytest.fixture
def post_published():
    return PostFactory(status="published")


@pytest.mark.django_db
def test_get_posts_titles(post_published, post_draft):
    assert get_posts_titles() == [post_published.title]
    assert get_posts_titles(include_unpublished=True) == [post_published.title, post_draft.title]

Notice the amount of boilerplate on the code, we had to:

create a PostFactory on the factories.py file;
create a fixture for a draft post;
create a fixture for a published post.

Avoiding boilerplate code with pytest-factoryboy

Let’s see how we could re-write the code from the previous section using pytest-factoryboy.

We’ll still have the factories defined in the factories.py file:

import factory


class PostFactory(factory.django.DjangoModelFactory):
    title = "Combining Pytest and FactoryBoy for simplified tests in Django"
    status = "draft"

    class Meta:
        model = "blog.Post"

But now, we register that factory in conftests.py so that it is available as a fixture for all tests:

from pytest_factoryboy import register

from factories import PostFactory


register(PostFactory)
register(PostFactory, "draft_post", title="Draft post", status="draft")

The fixture will take the name of the Factory (post by default) but you can optionally provide it a name. Two fixtures will be created: post and draft_post.

Then, in the tests, we can start using the registered fixtures:

import pytest

from blog.utils import get_posts_titles


@pytest.mark.django_db
def test_get_posts_titles(post, draft_post):
    assert get_posts_titles() == [post.title]
    assert get_posts_titles(include_unpublished=True) == [post.title, draft_post.title]

Since pytest-factoryboy took care of registering the factories as fixtures for us, we won’t need to:

manually create the fixtures;
manually import the PostFactory on the test file.

More about pytest-factoryboy

Being able to register a factory as a fixture is just one of pytest-factoryboy’s features. Let’s see some more.

Set custom attribute values for you model fixtures

There will be some test cases in which you need to customize an attribute of your model fixture. You can easily do so using pytest.mark.parametrize:

import pytest

from blog.utils import get_posts_titles


@pytest.mark.django_db
@pytest.mark.parametrize("post__title", ["Custom title"])
@pytest.mark.parametrize("post_draft__title", ["Custom title - Draft"])
def test_get_posts_titles(post, draft_post):
    assert get_posts_titles() == ["Custom title"]
    assert get_posts_titles(include_unpublished=True) == ["Custom title", "Custom title - Draft"]

Override a fixture’s value

If you are modifying the same attribute on multiple test functions within a module, you can opt to override the attribute on the fixture for the entire module. The following code will set post.status to "draft" for the entire module:

import pytest

from blog.utils import get_posts_titles


@pytest.fixture
def post__status():
    """Override blog's status to draft."""
    return "draft"


@pytest.mark.django_db
def test_get_posts_titles(post):
    assert get_posts_titles() == []
    assert get_posts_titles(include_unpublished=True) == [post.title]

Factories are automatically registered as fixtures

Sometimes you will just want to interact with the factory directly, without using the model fixture (e.g. post). pytest-factoryboy automatically registers the factory as a fixture too. Notice that the fixture name is the snake_case version of the factory name:

import pytest

from blog.utils import get_posts_titles


@pytest.mark.django_db
def test_get_posts_titles(post_factory):
    post = post_factory(status="draft")
    assert get_posts_titles() == []
    assert get_posts_titles(include_unpublished=True) == [post.title]

Assign other fixtures as values with parametrize

When your models have ForeignKey relationships you might want to assign another fixture as the value. You can do so using parametrize and LazyFixture.

Given a Post with a blog attribute as a ForeignKey, we could have the following factories:

import factory


class BlogFactory(factory.django.DjangoModelFactory):
    name = "roschegel"

    class Meta:
        model = "blog.Blog"


class PostFactory(factory.django.DjangoModelFactory):
    title = "Combining Pytest and FactoryBoy for simplified tests in Django"
    status = "draft"
    blog = factory.SubFactory(BlogFactory)

    class Meta:
        model = "blog.Post"

In our test case we could then override the blog attribute using LazyFixture:

import pytest

from pytest_factoryboy import LazyFixture


@pytest.fixture
def blog_2(blog_factory):
    return blog_factory(name="some_other_blog")


@pytest.mark.django_db
@pytest.mark.parametrize("post__blog", [LazyFixture("blog_2")])
def test_lazy_fixture(post, blog_2):
    assert post.blog.id == blog_2.id

Wrapping up

Next time you find yourself writing factories and fixtures that just return instances of your models, consider using pytest-factoryboy. It will save you a considerable amount of boilerplate code. And that means more time to work on meaningfull stuff, which is always welcome.

Making your Django tests faster

Sat, 09 May 2020 00:00:00 +0000

Tests need to be fast. If tests are slow, our development process is affected and we end up spending a considerable amount of time waiting for the results.

I will go through some of the techniques that I’ve applied to speed up tests on Django applications.

Tips for speeding up test execution

1. Run your tests in parallel

If you are running your tests on multi-core hardware, running your tests in parallel is probably the best optimization you can make if you aren’t doing it yet.

If you are using pytest, install the pytest-xdist plugin:

$ pip install pytest-xdist

And then run your tests with the -n parameter:

$ pytest -n auto

The -n parameter accepts either a number, indicating the number of cores in which you would like to run the tests, or the value auto in which case the plugin will automatically detect the number of cores that are available on your machine and use them all.

If you are using the Django test runner, simply run the tests using the --parallel parameter:

$ ./manage.py test --parallel

You can also indicate the specific number of cores in which you would like your tests to run by providing a number:

$ ./manage.py test --parallel 4

2. Use fixtures for modularized setup (Pytest only)

In the typical xUnit based testing frameworks (such as unittest) you create a setup method within your test class in which you set up everything you need for your tests.

Pytest fixtures are a replacement for the conventional setup method. To declare a fixture, you create a function and decorate it with the @pytest.fixture decorator.

To use a fixture on a test case you can:

Make it run automatically for each test case with the autouse parameter: @pytest.fixture(autouse=True)
Add it to your test case parameter list. This will trigger the fixture to be executed.
Add the usefixtures mark decorator to your test class: @pytest.mark.usefixtures("your_fixture")

One of the greatest advantages of using fixtures is that you can modularize your setup into smaller and less generic functions.

By splitting your setup method into multiple methods, you will be able to select only the ones you need for each test case. This will ensure that you execute exactly what is needed for each test case and nothing else.

3. Use the database only when it is needed

It is common to see lots of database records being created on the setup method of a test class. Usually, not all the records are used by all the test cases, but they still get created each time.

Make sure you only create the records that will be used by all the tests on your setup method. If there is a database record that is not being used by all the test cases, either:

Move it to a fixture (if you are using pytest)
Move it to the specific test where you need it

4. Search for the slowest tests and look for possible optimizations (Pytest only)

Pytest provides a way to find which are the slowest tests by executing them using the --durations parameter.

The following example command will execute all the tests and print a summary at the end with a list of the 5 tests that took the longest to run:

$ pytest --durations=5

5. Squash your migrations

Consider squashing your migrations if you have too many. Squashing refers to reducing the number of migrations you have by merging them into one.

Migrations are run each time you execute your test suite. Squashing migrations would not only speed up your tests but also your development setup.

Tips for speeding up your tests execution locally

When running tests locally while developing a new feature, fixing a bug or refactoring, there are few more things you can do to speed up your tests:

1. Re-use the database among test runs

Every time you run your Django tests, a new test database is created, this takes a considerable amount of time. When you are not making changes to your models or migration files, you can skip this step by telling the runner that you want to preserve the database among test executions.

If you are using Pytest, you can use pytest-django’s --reuse-db parameter:

$ pytest --reuse-db

If you are using the Django test runner you can accomplish the same with the --keep-db parameter:

$ ./manage.py test --keep-db

2. Disable Migrations with —no-migrations

When the test runner starts, the database is created and migrations run one by one to reach the current state of your models. If you choose to run your tests without executing the migrations, Django will create the database using the current state of your models.

Skipping migrations should be safe unless you are making changes to your migrations, in which case you would also want to ensure they are working as expected.

Depending on the number of migrations you have, they could take a lot of time. But most of the time, while testing locally, we don’t need them to run.

To disable migrations, include the --no-migrations parameter:

$ pytest --no-migrations

Or with the Django test runner:

$ ./manage.py test --no-migrations

3. Run only the tests that failed on the last run (Pytest only)

If you just need to know if the tests that were previously failing are still failing, use pytest’s --lf parameter:

$ pytest --lf

Wrapping up

Ensuring tests run fast is key to having a healthy project. Applying these techniques should help you take your first steps into faster and more efficient tests with Django.

If your application is big enough you might want to consider applying the Repository Pattern and making your tests completely independent of the database.

Optimizing Django ORM Queries

Sun, 03 May 2020 00:00:00 +0000

The Django ORM (Object Relational Mapping) is one of the most powerful features of Django. It enables us to interact with the database using Python code instead of SQL.

It has multiple advantages:

The database engine is abstracted from us, so it is possible to switch to another database system with ease.
It supports migrations: we can easily change our tables by updating our models and Django will automatically generate the migration scripts needed to update the database tables.
It supports transactions: you can make multiple updates to the database within a transaction and, if something fails, roll it back to the way it was when you started.

But it also comes with some disadvantages:

Since it is an abstraction on top of SQL, it is obscure, and we don’t know exactly which SQL queries will be generated from our Python code.
Django has no way to guess when we will need to use a related table, so it won’t do JOINs for us when we need them.
The ORM gives us the wrong sensation that what we are doing is not expensive. We have no easy way to know that accessing an attribute in an object might trigger a query to the database that could have been prevented with a JOIN.

To overcome the disadvantages we need to become more acquainted with it and understand what is happening under the hood.

Find out what’s happening under the hood

First, we need to understand what is happening in our system, which SQL queries are being run, and what’s costing us the most.

Here are some different mechanisms to inspect SQL queries as they are executed:

1. connection.queries

When debug=True, it is possible to access the queries that have been executed by printing connection.queries:

>>> from django.db import connection
>>> Post.objects.all()
>>> connection.queries
[
   {
      'sql': 'SELECT "blogposts_post"."id", "blogposts_post"."title", '
             '"blogposts_post"."content", "blogposts_post"."blog_id", '
             '"blogposts_post"."published" FROM "blogposts_post" LIMIT 21',
      'time': '0.000'
   }
]

connection.queries holds a list of SQL queries in the form of dictionaries containing the SQL code and the time it took to run.

The queries list could get convoluted very easily. To fix that, Django provides a way to clean them up:

>>> from django.db import reset_queries
>>> reset_queries()

2. shell_plus –print-sql

The django-extensions project is great and comes with a handful of useful features.

shell_plus is one of them. It’s a Django shell with extra additions. If you call it with the --print-sql parameter, it will print the SQL queries as they are executed when you run your code.

I will use shell_plus throughout this post so that you can see the SQL queries that are being executed as the code is ran.

Here is a quick example of how the output would look like:

$ ./manage.py shell_plus --print-sql
>>> post = Post.objects.get(id=1)
SELECT "blogposts_post"."id",
     	"blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
ORDER BY "blogposts_post"."id" ASC
LIMIT 1

3. django-silk

django-silk is a profiling tool. It intercepts requests, records the SQL queries that were performed, and provides a way to visualize them.

You will be able to browse through the requests, see a list of SQL queries that were performed, and look at the details about a specific query including which line of code caused a certain query to run.

django-debug-toolbar adds a toolbar on your browser that will show you lots of debugging information while you browse your Django project. Using it, you can see the number of SQL queries that were performed on a request. It is also possible to inspect these queries further, check the SQL code and see in which order they were performed and how much time each one took.

Optimize your queries

Introducing an Example Database Model

We will use the following database models as an example for the upcoming sections:

class Blog(models.Model):
   name = models.CharField(max_length=250)
   url = models.URLField()

   def __str__(self):
       return self.name


class Author(models.Model):
   name = models.CharField(max_length=250)
   email = models.EmailField()

   def __str__(self):
       return self.name


class Post(models.Model):
   title = models.CharField(max_length=250)
   content = models.TextField()
   published = models.BooleanField(default=False)

   blog = models.ForeignKey(Blog, on_delete=models.CASCADE)
   authors = models.ManyToManyField(Author, related_name="posts")

   def __str__(self):
       return self.title

Use cached Foreign Key ids

If we just need to access the id of a ForeignKey field, we can use the cached id that Django already has cached for us via <field_name>_id.

Let see it through an example:

>>> Post.objects.first().blog.id
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
ORDER BY "blogposts_post"."id" ASC
LIMIT 1

Execution time: 0.001668s [Database: default]
SELECT "blogposts_blog"."id",
      "blogposts_blog"."name",
      "blogposts_blog"."url"
 FROM "blogposts_blog"
WHERE "blogposts_blog"."id" = 1
LIMIT 21

Execution time: 0.000197s [Database: default]

Accessing the blog’s id through the nested object blog generated a new SQL query to obtain the entire blog object. But since we won’t need to access any other attribute from the blog object, we could completely avoid the above query from being executed by doing:

>>> Post.objects.first().blog_id
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
ORDER BY "blogposts_post"."id" ASC
LIMIT 1

Execution time: 0.000165s [Database: default]

Let Django know what you will need in advance

Use select_related for Foreign Keys

Django has no way of anticipating when we will need to access a ForeignKey relationship from within the model we are querying. The select_related utility allows us to tell Django exactly which related models we want, so that it can perform JOINs.

In our example, we have a Post model. A Post belongs to a specific Blog. This relationship is represented on the database through a ForeignKey from the Post to the Blog.

To access a specific Post object we could do:

>>> post = Post.objects.get(id=1)
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
ORDER BY "blogposts_post"."id" ASC
LIMIT 1

If we wanted to access the Blog object from within the Post, we could do:

>>> post.blog
SELECT "blogposts_blog"."id",
      "blogposts_blog"."name",
      "blogposts_blog"."url"
 FROM "blogposts_blog"
WHERE "blogposts_blog"."id" = 1
LIMIT 21

Execution time: 0.000602s [Database: default]
<Blog: Rocio's Blog>

However, this statement generated a new query to grab the information from the blog. We want to avoid that. This is when select_related comes to our rescue. To use it we can update our original query to be:

>>> post = Post.objects.select_related("blog").get(id=1)
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published",
      "blogposts_blog"."id",
      "blogposts_blog"."name",
      "blogposts_blog"."url"
 FROM "blogposts_post"
INNER JOIN "blogposts_blog"
   ON ("blogposts_post"."blog_id" = "blogposts_blog"."id")
WHERE "blogposts_post"."id" = 1
LIMIT 21

Execution time: 0.000150s [Database: default]

Notice now how Django used a JOIN on the SQL query above to also grab the attributes from the blog table for us. Now, when accessing the Blog object from within the Post, it will not require an extra query since it will already be cached:

>>> post.blog
<Blog: Rocio's Blog>

select_related also works for querysets. We could pre-select the blog object for an entire queryset. If there were 50 Posts and we didn’t use select_related to pre-select the blog object, it would take Django 50 queries to run the following code. With select_related it just takes one:

>>> posts = Post.objects.select_related("blog").all()
>>> for post in posts:
       post.blog

SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published",
      "blogposts_blog"."id",
      "blogposts_blog"."name",
      "blogposts_blog"."url"
 FROM "blogposts_post"
INNER JOIN "blogposts_blog"
   ON ("blogposts_post"."blog_id" = "blogposts_blog"."id")

Execution time: 0.000224s [Database: default]

Use prefetch_related for ManyToMany fields

prefetch_related is similar to select_related, but it is used for pre-selecting ManyToMany fields. prefetch_related works differently, let’s see it through an example.

Let’s say we wanted to grab all the Posts and then print the Authors for each of the Posts. We could do the following:

>>> for post in Post.objects.all():
       post.authors.all()

SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"

Execution time: 0.000158s [Database: default]
<QuerySet []>
SELECT "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_author"."id" = "blogposts_post_authors"."author_id")
WHERE "blogposts_post_authors"."post_id" = 1
LIMIT 21

Execution time: 0.000101s [Database: default]
<QuerySet []>
SELECT "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_author"."id" = "blogposts_post_authors"."author_id")
WHERE "blogposts_post_authors"."post_id" = 2
LIMIT 21
Execution time: 0.001043s [Database: default]

Execution time: 0.000101s [Database: default]
<QuerySet []>
SELECT "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_author"."id" = "blogposts_post_authors"."author_id")
WHERE "blogposts_post_authors"."post_id" = 3
LIMIT 21
Execution time: 0.001043s [Database: default]

Notice that the code above generated 4 queries, one to grab the posts, and then one query for each of the posts to grab the authors (there were 3 posts in total).

This is the famous N + 1 problem. Given N posts, N + 1 queries will be performed. In this scenario we have 3 posts, which translate to 4 queries. It isn’t that much, but this could very easily escalate as we create new Posts. With 50 Posts, this code would generate 51 queries.

To avoid that, we could pre-select the Authors by using prefetch_related:

>>> for post in Post.objects.prefetch_related("authors").all():
       post.authors.all()

SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"

Execution time: 0.000158s [Database: default]
SELECT ("blogposts_post_authors"."post_id") AS "_prefetch_related_val_post_id",
      "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_author"."id" = "blogposts_post_authors"."author_id")
WHERE "blogposts_post_authors"."post_id" IN (1, 2, 3)

Execution time: 0.001043s [Database: default]

With our updated code only 2 queries were performed. When prefetch_related is used, Django first grabs all the posts and then runs another SQL query that retrieves all the authors for all the posts.

Customizing Prefetch

In some scenarios prefetch_related basic syntax is not enough to prevent Django from doing extra queries. To further control the prefetch you can use the Prefetch object.

In our example database, there is a Post model and an Author model. The Post model is related to the Author model through a ManyToMany field. Let’s say we wanted to go author by author and grab all the posts that were published by that author:

>>> authors = Author.objects.all()
>>> for author in authors:
       print(author.posts.filter(published=True))

SELECT "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"

Execution time: 0.000251s [Database: default]
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_post"."id" = "blogposts_post_authors"."post_id")
WHERE ("blogposts_post_authors"."author_id" = 1 AND "blogposts_post"."published" = 1)
LIMIT 21

Execution time: 0.000178s [Database: default]
<QuerySet [<Post: Optimizing Django ORM Queries>, <Post: Placeholder Post>, <Post: Placeholder Post 2>, <Post: Placeholder Post 3>, <Post: Placeholder Post 4>, <Post: Placeholder Post 6>, <Post: Placeholder Post 7>, <Post: Placeholder Post 8>, <Post: Placeholder Post 9>]>
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_post"."id" = "blogposts_post_authors"."post_id")
WHERE ("blogposts_post_authors"."author_id" = 2 AND "blogposts_post"."published" = 1)
LIMIT 21

Execution time: 0.000081s [Database: default]
<QuerySet [<Post: Optimizing Django ORM Queries>]>

As you can see, the above code generated 3 queries, 1 to grab the author, and then 2 queries to grab the posts for each of the authors.

What if we used prefetch_related? It seems to be the right thing to do:

>>> authors = Author.objects.prefetch_related("posts").all()
>>> for author in authors:
       print(author.posts.filter(published=True))

SELECT "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"

Execution time: 0.000097s [Database: default]
SELECT ("blogposts_post_authors"."author_id") AS "_prefetch_related_val_author_id",
      "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_post"."id" = "blogposts_post_authors"."post_id")
WHERE "blogposts_post_authors"."author_id" IN (1, 2)

Execution time: 0.000190s [Database: default]
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_post"."id" = "blogposts_post_authors"."post_id")
WHERE ("blogposts_post_authors"."author_id" = 1 AND "blogposts_post"."published" = 1)
LIMIT 21

Execution time: 0.000074s [Database: default]
<QuerySet [<Post: Optimizing Django ORM Queries>, <Post: Placeholder Post>,
<Post: Placeholder Post 2>, <Post: Placeholder Post 3>,
<Post: Placeholder Post 4>, <Post: Placeholder Post 6>,
<Post: Placeholder Post 7>, <Post: Placeholder Post 8>,
<Post: Placeholder Post 9>]>
SELECT "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_post"."id" = "blogposts_post_authors"."post_id")
WHERE ("blogposts_post_authors"."author_id" = 2 AND "blogposts_post"."published" = 1)
LIMIT 21

Execution time: 0.000070s [Database: default]
<QuerySet [<Post: Optimizing Django ORM Queries>]>

What did just happen? We used prefetch_related to reduce the number of queries, and we actually increased it by 1.

This is happening because we are filtering the posts with published=True. Django can’t use our cached posts since they were not filtered when they were queried. To avoid this from happening, we can customize the queryset using the Prefetch object:

>>> authors = Author.objects.prefetch_related(
       Prefetch(
          "posts",
          queryset=Post.objects.filter(published=True),
          to_attr="published_posts",
       )
    )
>>> for author in authors:
      print(author.published_posts)

SELECT "blogposts_author"."id",
      "blogposts_author"."name",
      "blogposts_author"."email"
 FROM "blogposts_author"

Execution time: 0.000129s [Database: default]
SELECT ("blogposts_post_authors"."author_id") AS "_prefetch_related_val_author_id",
      "blogposts_post"."id",
      "blogposts_post"."title",
      "blogposts_post"."content",
      "blogposts_post"."blog_id",
      "blogposts_post"."published"
 FROM "blogposts_post"
INNER JOIN "blogposts_post_authors"
   ON ("blogposts_post"."id" = "blogposts_post_authors"."post_id")
WHERE ("blogposts_post"."published" = 1 AND "blogposts_post_authors"."author_id" IN (1, 2))

Execution time: 0.000089s [Database: default]
[<Post: Optimizing Django ORM Queries>, <Post: Placeholder Post>,
<Post: Placeholder Post 2>, <Post: Placeholder Post 3>,
<Post: Placeholder Post 4>, <Post: Placeholder Post 6>,
<Post: Placeholder Post 7>, <Post: Placeholder Post 8>,
<Post: Placeholder Post 9>]
[<Post: Optimizing Django ORM Queries>]

We used the Prefetch object to tell Django to:

Use a specific queryset to retrieve the posts - through the queryset parameter.
Store the filtered posts in a new attribute (published_posts) - through the to_attr parameter.

When author.published_posts is executed, no queries will be run since everything will already be cached. No matter the number of authors on our system, the operation will always take 2 SQL queries.

Wrapping up

While working with the Django ORM, it is extremely important that we think about what’s happening under the hood.

The concepts that you learned on this blog post will help you write more optimized queries and be in the lookout for possible optimizations while reviewing code. Beware though, that you should always measure the time a query is taking before and after the optimization to ensure that the optimization worked. Sometimes less queries doesn’t necessairly mean less time, JOINs could be expensive too.

django on roschegel

Simplified Django Tests With Pytest and Pytest FactoryBoy

The Problem

Avoiding boilerplate code with pytest-factoryboy

More about pytest-factoryboy

Set custom attribute values for you model fixtures

Override a fixture’s value

Factories are automatically registered as fixtures

Assign other fixtures as values with parametrize

Wrapping up

Making your Django tests faster

Tips for speeding up test execution

1. Run your tests in parallel

2. Use fixtures for modularized setup (Pytest only)

3. Use the database only when it is needed

4. Search for the slowest tests and look for possible optimizations (Pytest only)

5. Squash your migrations

Tips for speeding up your tests execution locally

1. Re-use the database among test runs

2. Disable Migrations with —no-migrations

3. Run only the tests that failed on the last run (Pytest only)

Wrapping up

Optimizing Django ORM Queries

Find out what’s happening under the hood

1. connection.queries

2. shell_plus –print-sql

3. django-silk

4. django-debug-toolbar

Optimize your queries

Introducing an Example Database Model

Use cached Foreign Key ids

Let Django know what you will need in advance

Use select_related for Foreign Keys

Use prefetch_related for ManyToMany fields

Customizing Prefetch

Wrapping up