May 4, 2020

Manually Checking/Applying Django Migrations on GKE

Manually Checking/Applying Django Migrations on GKE

Where does this fit in?

Containerisation has been a big IT buzzword for several years now - powered by the Docker platform - and in that time, it has driven new ways of setting up infrastructure, most notably in the cloud. It helps isolate developers from the details of infrastructure setup and the many potential pitfalls of having different environments locally versus the cloud. At DotModus, one tried and tested way we have made use of containerisation is to help power scalable, reliable applications in the cloud. The use case I’ll be touching on in this article is how to manually check on and maintain Django migrations (database schema updates, for those less familiar with the open-source Python web development framework) for a Django app running on Google Kubernetes Engine (GKE).

Why would I need to do things like this manually?

If you’re following development best practices and making life easy for yourself with a proper CI/CD pipeline, then in theory any migrations that need to be made are being applied automatically from your pipeline every time you deploy changes. Hopefully, that means you never need to get your hands dirty with things like manual checks and maintenance - but when theory meets practice, anything and everything will go wrong at some point. Perhaps you’d just like to double check things? Whatever the case may be, the process described here can easily be extended to many other use cases on GKE, and the manage.py commands apply to any Django application.

Enough introduction, here is the step-by-step:

  1. Open Google Cloud Console
  2. Check that you are on the correct GCP project (top left of the page, just to the right of "Google Cloud Platform")
  3. Open the menu on the left (the three horizontal lines to the left of "Google Cloud Platform")
  4. Scroll down to the "Compute" heading, select "Kubernetes Engine", then "Workloads" in the sub-menu (shortcut)
  5. Select the Workload that includes the Django containers. A Workload is a set of pods that maintains a certain amount of pods (possibly auto-scaling to demand, depending on settings)
  6. When you scroll towards the bottom of the page, there should be a section titled "Managed pods". Select any of the pods in the list by clicking on the name of the pod
  7. On the pod details page, to the right of the "Pod details" heading, there is a button marked "Kubectl". This provides shortcuts for running various Kubectl commands on your own Cloud Shell instance. Under that menu, go to "Exec" and click on the container in the pod that is running Django
  8. A connection to your Cloud Shell will be opened at the bottom of the page. A command will be pre-loaded, ready to run, in the format:
gcloud container clusters get-credentials<GKE cluster name> --zone <zone> --project <project name>\ & kubectl logs <pod ID> -c <container name>
Note that <something> terms indicate values that should be auto-filled for you by GCP

10. To manage Django migrations (or do just about anything else that requires direct access to the container), replace ls with bash, and add -it after the exec command, to get a command like:

11.  gcloud container clusters get-credentials

<GKE cluster name> --zone <zone> --project <project name> && kubectl exec -it <pod ID> -c <container name> -- bash

12.  Note: the -it flags are important if you want to use more than one command before disconnecting. The -it flags stand for --interactive + --tty. These flags allow us to use the terminal inside of the container. By not specifying the -ti flags the commands will run but there will be no output and we will not be able to see what the result of the commands were. otherwise you can skip it. If you are working with an unusual container, you may need to replace bash with an appropriate alternative.

13.  When you press Enter on that command, after a short delay as it retrieves credentials and sets up the connection, you should see an interactive shell prompt. At this point, you can run the command(s) you need:

  • Apply any unapplied existing migrations:
    python manage.py migrate
  • Check that migrations were made (and make them if not):
    python manage.py makemigrations
    Note: apply any existing migrations first, to avoid potential issues with migration dependencies later on
  • View migration status:
    python manage.py showmigrations
  • View SQL statements for a migration (should you need to):
    python manage.py sqlmigrate <app name> <migration name>

Notes and troubleshooting:

  • Like most actions on GCP, accessing a container this way requires the correct IAM roles: GKE Developer is the basic role required for access, but GKE Admin and the project-wide Owner or Editor roles also provide the required access. Additionally, per-cluster RBAC (Role-Based Access Control) can provide the required access in more complex permissions setups.
  • While you can access the Kubernetes pod logs by editing the kubectl exec command into a kubectl logs command, it will usually be easier and more useful to find and follow the link called "Container logs" instead of doing Step 6 to access the logs for all pods in the workload, or instead of Step 7 to access the logs for only a specific pod. This will open the logs in GCP’s Stackdriver Logs Viewer, which provides a powerful, queryable view into your logs. For the sake of completeness, though, here is the example command to access the logs via the Cloud Shell:
gcloud container clusters get-credentials<GKE cluster name> --zone <zone> --project <project name>\ & kubectl logs <pod ID> -c <container name>
  • It’s certainly possible to run more or less the exact same process on a local terminal on your computer (probably using a kubectl get pods command to get the pod ID), but the process described above tends to be very convenient if this sort of direct access to the pods is only infrequently required. The auto-filling of relevant fields becomes even more convenient if you work with multiple namespaces, clusters, or GCP projects.

I hope you find this guide useful. If you have any questions, or know an easier way - drop us a message!