Mitigating Django migration conflicts

Django migration conflicts are common when developing a Django application. In this blog, we will leverage AWS Lambda, Bitbucket webhooks, and CircleCI workflows to prevent the introduction of Django migration conflict to the dev/main branch.

Infrastructure requirements

Before you read any further, be aware that this blog post assumes that you have these resources available at your disposal:

• Webhook service
• VCS
• CI/CD

For this blog, we will use the following infrastructure:

• AWS Lambda as webhook service
• Bitbucket for VCS
• CircleCI as CI/CD.

You can use other solutions for webhook service, VCS, and CI/CD. The idea is the same, but you will have to figure out how to make this work on your infrastructure. If you use the same infrastructure, be sure that you have sufficient permissions for API Gateway and Lambda AWS services. You will also need administrator rights to your project's Bitbucket repository.

Motivation

Since you are reading this article, you may as well already know what Django migration conflict is. Still, we need to set up a common ground before we move on. Assuming that you work in a company that develops Django apps, you can easily get Django migration conflict. All it requires is for you and your colleague to edit the same Django model or create a migration file with the same order number at the same time. You both work on some features and each of you creates a separate branch for your implementation. Your colleague managed to get his branch merged into the main branch sooner than you and had his migrations applied. In such a case, you have to pull the newest changes and resolve conflicts, if there are any. However, it is not that hard for humans to forget, right? If you forget to pull your colleague's changes before you merge your branch into the main branch, you will introduce Django migration conflict into the main branch. Someone will have to pull the main branch, fix it and push the changes, or undo your merge to the main branch. Especially in agile and incremental development, where there are several pull requests merged on daily basis, it can become very tedious to fix forgotten Django migration conflicts in the main branch, especially if they can be prevented.

Prevention

As the name of this blog post suggests, we can use CI/CD. Let us go back to the scenario with you and your colleague. For this prevention to work, you need to have set up CI/CD workflows for your branches. Let us assume that you have different workflows set up for different types of branches. It is logical to have a different workflow for the feature/bugfix branch and dev/main branch. With CI/CD in place, we can add Django migration conflict check to feature/bugfix workflow. This way, you and your colleagues can be aware of possible conflicts before propagating them further and simply fix them on your branch. Since these conflict checks will be done on feature/bugfix branches, your main/dev branches will not require these checks.

Prevention flow

![Flow](https://github.com/remastr/django-migration-conflict-check/raw/main/docs/img/flow.png "Flow diagram" 100x20)

At the first stage, a developer does an action (commit, push, merge, ...) in VCS. This VCS action triggers an event, which is sent to a Webhook service. Webhook service provides you with the means to process an event and send a response. In our case, we will respond to CI/CD associated with our VCS and trigger an action (rebuild project, trigger new pipeline, ...).

Setup

The setup procedure involves several steps. Please follow them strictly in this order:

• Create API Keys.
• Set up your <Webhook-Service> + <CI/CD> pair.
• Create Bitbucket webhook.
• CircleCI config - Django migration check.

Create API keys

Create Bitbucket App Password:

Bitbucket App Password is your API key. With this key, you can interact with your Bitbucket. At the time of creation, you have to assign permissions to the key. Permissions cannot be modified afterward. This key does not require 2-step verification, so be careful when sharing this key. Lastly, please note that you can see the content of the key only at the time of creation.

• Sign in to the Bitbucket Cloud.
• Go to your "Personal settings".
• Under the "ACCESS MANAGEMENT" section, go to "App passwords" and click there on the "Create app password" button.
• Select permissions according to your use case and click on "Create".
• Save your App password in a safe place. You will need it later on.

Create CircleCI Personal API Token:

As with the Bitbucket App Password, this is also your API key and you can see the context only when creating the key. However, in this case, you cannot assign any permissions. This key has by default full access rights, so you have to be extremely careful with whom you share this key.

• Sign in to the CircleCI.
• Go to your "User settings".
• Click on "Personal API Tokens" and there click on the "Create New Token" button.
• Name accordingly your token and hit "Add API Token".
• Save your API token in a safe place.

AWS Lambda + CircleCI

We will use AWS Lambda to run our webhook processing script in a container. Running code in a container without any server is called serverless computing. The first sentence from Wiki explains it perfectly - "Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on-demand, taking care of the servers on behalf of their customers." This container will be reachable from the Internet thanks to the HTTP trigger that we are going to set up. HTTP trigger is done via API Gateway. API Gateway is a service used for creating API. It will provide us with a web address. We can configure these web address URL paths and what code to execute on each of them. Since our webhook has only one use case, the root web address will do. Lastly, we need to upload the source code of our webhook processing script into the AWS Lambda function we have created. At our Github repository, we already have a script available that you can use. All that's left is to set up AWS Lambda function environment variables. The script will need your Bitbucket username, API keys you have created, and lastly, a slug of the project on which you want to Django migration check.

Create a function:

• Sign in to the AWS Management console.
• Go to Lambda service.
• Click on the "Create function" button.
• Select the "Author from scratch" option.
• Name your function.
• Select Python3.8 runtime.
• Click on the "Create function" button.

Add HTTP trigger:

• On the function page, click on the "Add trigger" button.
• Select "API Gateway" from the dropdown menu.
• Next, from the second dropdown menu, choose "Create an API".
• Select "HTTP API" API type.
• Select "Open" security from the third dropdown menu.
• Click on the orange "Add" button.

Upload code:

• Download the latest zip package from releases.
• On the function page, under "Code" section, click on "Upload from" button and choose ".zip file" option.
• Upload the "aws_deployment.zip" you have downloaded in the first step.

Set Lambda Environment variables:

• On the function page, under "Configuration" section, click on "Environment variables".
• Click on the "Edit" button and add following variables with your values:

BITBUCKET_API_URL	https://api.bitbucket.org
BITBUCKET_APP_PASSWORD	<Your-Bitbucket-App-Password>
BITBUCKET_USERNAME	<Your-Bitbucket-Username>
CIRCLECI_API_TOKEN	<Your-Personal-CircleCI-API-Token>
CIRCLECI_PROJECT_SLUG   bb/<Your-CircleCI-Organization-Name>/<CircleCI-Project-Name>

• Click on the "Save" button.

Create Bitbucket Webhook

The final step is to configure the Bitbucket webhook. In this step, you will set up Bitbucket to send notification (JSON payload) to your AWS Lambda function API gateway URL. Notifications will be sent only if the trigger specified by you will get triggered. The script that we provided you with will read the JSON payload and trigger new pipelines on project branches that have PR to the branch that just triggered the "PR merged" trigger.

• Go to your Bitbucket repository.
• Click on "Repository settings".
• Click on "Webhooks" in the menu.
• Click on "Add webhook", type in the title and URL you got from the HTTP trigger (API Gateway).
• Under Triggers, select "Pull request merged".
• Click on the "Save" button.

After you are done, it should look similar to this, except for the URL and name you chose for the webhook.

CircleCI config - Django migration check

The last step required is to add a migration check to your CircleCI config.yml. An example of how you can implement this:

version: 2
jobs:
  # Some jobs before

  migration_check:
    docker:
      - image: cimg/python:3.7
      - image: circleci/postgres:11.5-alpine-ram
    steps:
      - checkout
      - run:
          name: Merge dev branch to PR branch
          command: |
            git config --global user.email "dummy@mail.com"
            git config --global user.name "dummy_user"
            git merge --no-commit dev
      - restore_cache:
          keys:
            - dependencies-{{ checksum "your-lock-file" }}
            - dependencies-
      - run:
          name: Check for conflicting migrations
          command: poetry run python src/manage.py migrate

  # Some jobs after

workflows:
  version: 2
  your_project:
    jobs:
      # Some jobs before

      - migration_check:
          requires:
            - build
          filters:
            branches:
              ignore:
                - master
                - staging
                - dev

      # Some jobs after

Explanation

In the example CircleCI config.yml above, we set up a job called migration_check, this job checks not only Django migrations but also git merge conflicts as a bonus. The thing is, if you remember the scenario in the motivation section, your colleague has pushed the changes to the dev branch. Naturally, you have to pull these changes to your branch and resolve the conflicts if any occur. This job will do that for you and in other words, also check if you have not by accident forgot to pull the changes. If you forgot, then this job will fail and such a result is to be expected. Once it has pulled the changes, it will restore your project dependencies from a cache, if you have any (it is recommended by CircleCI to save dependencies from a build job to a cache). After it has the dependencies loaded, it will run the Django command for migration. Django command will fail only in case there are some Django migration conflicts.

Conclusion

There you have it. With this infrastructure, you can find and resolve git merge conflicts and Django migration conflicts and push clean into the dev/main branch. This setup might look like a lot of work, but it is worth the time. Imagine you have 7 PRs to the dev/main branch and your lead developer will start to merge them. The first merge might pass, but it could cause the other 6 PRs to introduce conflicts to your branch. The lead developer will have to roll back the merge, give you notice, and order you to resolve the conflicts, push changes again and then try to merge. This infrastructure saves you time since no rollback will be needed and you instantly know if there are conflicts. Also now you have the template on how to create Bitbucket webhooks and Bitbucket offers a lot more triggers than just "pull request merged". You can tweak this to your needs and enjoy the benefits.