edx_django_utils.db package#

Subpackages#

Submodules#

edx_django_utils.db.queryset_utils module#

Utils related to QuerySets.

edx_django_utils.db.queryset_utils.chunked_queryset(queryset, chunk_size=2000)#

Slice a queryset into chunks.

The function slices a queryset into smaller QuerySets containing chunk_size objects and then yields them. It is used to avoid memory error when processing huge querysets, and also to avoid database errors due to the database pulling the whole table at once. Additionally, without using a chunked queryset, concurrent database modification while processing a large table might repeat or skip some entries.

Warning: It throws away your sorting and sort queryset based on pk. Only recommended for large QuerySets where order does not matter. (e.g: Can be used in management commands to back-fill data based on Queryset having millions of objects.)

Source: https://www.djangosnippets.org/snippets/10599/

Example Usage:

queryset = User.objects.all() for chunked_queryset in chunked_queryset(queryset):

print(chunked_queryset.count())

Argument:

chunk_size (int): Size of desired batch.

Returns

Iterator with sliced Queryset.

Return type

QuerySet

edx_django_utils.db.read_replica module#

Tools for making queries read from the read-replica database, rather than from the writer database.

The read-replica can be used for

  1. long-running queries that aren’t time sensitive

  2. reads of rows that are frequently written, but where reads can be eventually consistent

Settings:
EDX_READ_REPLICA_DB_NAME: The name of the read-replica in the DATABASES django setting.

Defaults to “read_replica”.

EDX_WRITER_DB_NAME: The name of the writer in the DATABASES django setting.

Defaults to “default”.

class edx_django_utils.db.read_replica.ReadReplicaRouter#

Bases: object

A database router that by default, reads from the writer database, but can be overridden with a context manager to route all reads to the read-replica.

See https://docs.djangoproject.com/en/2.2/topics/db/multi-db/#automatic-database-routing

allow_migrate(db, app_label, model_name=None, **hints)#

All non-auth models end up in this pool.

allow_relation(obj1, obj2, **hints)#

Relations between objects are allowed if both objects are in either the read-replica or the writer.

db_for_read(model, **hints)#

Reads go the active reader name

db_for_write(model, **hints)#

Writes always go to the writer.

edx_django_utils.db.read_replica.read_queries_only()#

A context manager that sets all reads inside it to be from the read-replica.

It is an error to call this from inside a write_queries context.

The ReadReplicaRouter must be used for this decorator to affect queries.

edx_django_utils.db.read_replica.read_replica_or_default()#

If there is a database called READ_REPLICA_DB, return READ_REPLICA_DB, otherwise return WRITER_NAME.

This function is similiar to use_read_replica_if_available, but is be more syntactically convenient for method call chaining. Also, it always falls back to WRITER_NAME, no matter what the queryset was using before.

Example usage:

queryset = SomeModel.objects.filter(…).using(read_replica_or_default())

Returns: str

edx_django_utils.db.read_replica.use_read_replica_if_available(queryset)#

If there is a database called ‘read_replica’, use that database for the queryset / manager.

Example usage:

queryset = use_read_replica_if_available(SomeModel.objects.filter(…))

Parameters

queryset (QuerySet) –

Returns: QuerySet

edx_django_utils.db.read_replica.write_queries()#

A context manager that sets all reads inside it to be from the writer. Use this to annotate code that has both reads and writes, where the writes depend on the values read. This will allow all of that code to exist within a transaction.

Using this contextmanager will prevent any contained call from using read_queries_only in order to read from the read-replica.

It is an error to call this from inside a read_queries_only context.

The ReadReplicaRouter must be used for this decorator to affect queries.

Module contents#

Utilities for working effectively with databases in django.

read_replica: Tools for making queries from the read-replica. queryset_utils: Utils to use with Django QuerySets.