/var/

Various programming stuff

Hello! If you are using an ad blocker but find something useful here and want to support me please consider disabling your ad blocker for this site.

Thank you,
Serafeim

Using clojure from Windows

In this small article I’m going to post a guide on how to install and use clojure from Windows using good old’ cmd.exe.

Unfortunately, most guides on the official clojure site have instructions on using Clojure from Windows through Powershell or WSL. For my own reasons I hate both these approaches and only use the cmd.exe to interact with the Windows command line.

There are more or less two approaches to using clojure. Using leiningen or using the clj tools. The clojure official guide seems to be biased towards clj tools. However I think that leiningen may be easier for new users. I’ll cover both approaches here.

Warning Before doing anything else please make sure to install Java. You need a version of java that is at least 1.8. Try running java -version in cmd.exe to make sure you have java and it is the correct version.

Leiningen

To install leiningen you just download the lein.bat file from their page and put it in a folder in your PATH. You’ll then run lein and it will download all dependencies and install itself!

To start a clojure repl to be able to play with clojure you write lein repl. If everything went smooth you should see a prompt and if you write (+ 1 2) you should get 3. To exit press ctrl+d or write exit.

To start a new project you’ll use lein new [template name] [project name]. For example, to create a new app you’ll write: lein new app leinapp. You’ll get a new directory called leinapp. The important stuff in this directory are:

  • project.clj: The basic descriptor of your project; here you can set various attrs of your project and also add dependencies
  • src\leinapp: The source directory of your project. This is where you’ll put your code.
  • test\leinapp: Add tests here

There should be a core.clj file inside your src\leinapp folder. The main function is the entry point of the app. Try running lein run from the project folder and you should get the output of the main function.

Add this to the end of the core.clj to define a foo function:

(defn foo []
  "bar")

And run lein repl. You should get a repl command prompt for your application in the leinapp.core namespace (if you named your app leinapp). Type (foo) and you should see "bar".

To create a stand alone jar with your code (called uberjar) you can use lein uberjar. This will create a file named target\uberjar\leinapp-0.1.0-SNAPSHOT-standalone.jar. Then try java -jar target\uberjar\leinapp-0.1.0-SNAPSHOT-standalone.jar (notice I’m still on the leinapp project folder) and you’ll see the output of main!

clj

Using the clj is a more modern approach to clojure development. As I said before the official clojure page seems to be biased towards using this approach. The problem is that it seems to require Powershell to run as you can see on the clj on Windows page.

Thankfully, the good people at the clojurians slack pointed me to deps.clj project. This is an implementation of clj in clojure and can be installed natively on Windows by downloading the .zip from the releases page. This zip should contain a deps.exe file. Put that executable it in your path. You can also rename it to clj.exe if you want. Also if you have the powershell installed you can run this command from cmd.exe PowerShell -Command "iwr -useb https://raw.githubusercontent.com/borkdude/deps.clj/master/install.ps1 | iex" to install it automatically.

You can now run deps and you should get a clojure repl similar to lein repl.

To create a new project skeleton you can use the use the deps-new project. To install it run the following command from cmd.exe: deps -Ttools install io.github.seancorfield/deps-new "{:git/tag """v0.4.9"""}" :as new (please notice that there are various problems with the quoting on windows but this command should work fine).

To create a new app run: deps -Tnew app :name organization/depsapp and you’ll get your app in the depsapp folder. If you want a similar form as with lein, try deps -Tnew app :name depsapp/core :target-dir depsapp. Now the depsapp folder will contain:

  • deps.edn: The basic descriptor of your project; here you can set various attrs of your project and also add dependencies. This more or less changes the project.clj we got from leiningen.
  • src\depsapp: The source directory of your project. This is where you’ll put your code.
  • test\depsapp: Add tests here

To run the project, try: deps -M -m depsapp.core or deps  -M:run-m or deps  -X:run-x to directly run the greet function (run-m and run-x are aliases defined in deps.edn take a peek).

To start a REPL, run deps. Notice this will start on the user namespace, so you’ll need to do something like:

user=> (require 'depsapp.core)
nil
user=> (depsapp.core/foo)
"bar"

to run a (foo) function that you’ve added in the core.clj file.

To run the tests use: deps -T:build test.

To create the uberjar you’ll run: deps -T:build ci (tests must pass). Then execute it directly using java -jar target\core-0.1.0-SNAPSHOT.jar.

Also, notice that it’s really simple to create a new project with deps without the deps-new. For example, create a folder named manualapp and in this folder create a deps.edn file containing just the string {}. Then add another folder named src with a hello.clj file containing something like:

(ns hello)

(defn foo []
  "bar")

(defn run [opts]
  (println "Hello world"))

You can then open a REPL on the project using deps or run the run function using deps -X hello/run.

VSCode integration

Both leining and clj projects can easily be used with VSCode. First of all, install the calva package in your VSCode. Then, open your clojure project in VScode and press ctrl+shift+p to bring up the command pallete. Here write “Jack” (from jack-in) and select it (also this has the shortctut ctrl+alt+c ctrl+alt+j). Select the correct project type (leiningen or deps.edn). A repl will be opened to the side; you can then go to your core.clj file and run ctrl+alt+c enter to load the current file.

Then you can move to the repl on the side and run the function with (foo) or run (-main). Also you can write (foo) in your source file and press ctrl+enter to execute it and see the result; the ctrl+enter will execute the form where your cursor is. See this for more.

PDFs in Django like it’s 2022!

In a previous article I had written a very comprehensive guide on how to render PDFs in Django using tools like reportlab and xhtml2pdf. Although these tools are still valid and work fine I have decided that usually it’s way too much pain to set them up to work properly.

Another common way to generate PDFs in the Python world is using the weasyprint library. Unfortunately this library has way too many requirements and installing it on Windows is worse than putting needles in your eyes. I don’t like needles in my eyes, thank you very much.

There are various other ways to generate PDFs like using a report server like Japser or SQL Server Reporting Services but these are too “enterpris-y” for most people and require another server, a learning curve, etc.

I was actually so disappointed by the status of PDF generation today that in some recent projects instead of the PDF file I generated an HTML page with a nice pdf-printing stylesheet and instructed the users to print it as PDF (from the browser) so as to generate the PDF themselves!

However, recently I found another way to generate PDFs in my Django projects which I’d like to share with you: Using the wkhtmltopdf tool. The wkhtmltopdf is a command line program that has binaries for more or less every operating system. It’s a single binary that you can download and put it in a directory, you don’t need to run another server or any fancy installation. Only an executable. To use it? You call it like wkhtmltopdf http://google.com google.pdf and it will download the url and generate the pdf! It’s as simple as that! This tool is old and heavily used but only recently I researched its integration with Django.

Please notice that there’s actually a django-wkhtmltopdf library for integrating wkhtmltopdf with django. However I din’t have good results while trying to use it (maybe because of my Windows dev environment). Also, implementing the integration myself allowed my to more easily understand what’s happening and better debug the wkhtmltopdf. However YMMV, after you read this small post to understand how the integration works you can try django-wkhtmltopdf to see if it works in your case.

In any way, the first thing you need to do is download and install wkhtmltopdf for your platform and save its full path in your settings.py like this:

# For linux
WKHTMLTOPDF_CMD = '/usr/local/bin/wkhtmltopdf'

# or for windows
WKHTMLTOPDF_CMD = r'c:\util\wkhtmltopdf.exe'

Notice that I’m using the full path. I have observed that even if you put the binary to a directory in the system PATH it won’t be picked (at least in my case) thus I recommend using the full path.

Now, let’s suppose we’ve got a DetailView (let’s call it SampleDetailView) that we’d like to render as PDF. We can use the following CBV for that:

from subprocess import check_output
from django.template import Context, Template
from django.template.loader import get_template
from tempfile import NamedTemporaryFile
import os

class SamplePdfDetailView(SampleDetailView):
  def get_resp_from_file(self, filename, context):
      template = get_template(filename)
      resp = template.render(context)
      return resp

  def get_resp_from_string(self, template_str, context):
      template = Template(template_str)
      resp = template.render(Context(context))
      return resp

  def render_to_response(self, context):
      context['pdf'] = True
      # We can use either
      resp = self.get_resp_from_string("<h1>Hello, world! {{ object }}</h1>", context)
      # or
      # resp = self.get_resp_from_file('test_pdf.html', context)

      tempfile = NamedTemporaryFile(mode='w+b', buffering=-1,
                                    suffix='.html', prefix='tmp',
                                    dir=None, delete=False)

      tempfile.write(resp.encode('utf-8'))
      tempfile.flush()
      tempfile.close()
      cmd = [
          settings.WKHTMLTOPDF_CMD,
          '--page-size', 'A4', '--encoding', 'utf-8',
          '--footer-center', '[page] / [topage]',
          '--enable-local-file-access',  tempfile.name, '-']
      # print(" ".join(cmd))
      out = check_output(cmd)

      os.remove(tempfile.name)
      return HttpResponse(out, content_type='application/pdf')

We can put the pdf view on our url patterns right next to our DetailView i.e:

[
  ...
  path(
      "detail/<int:pk>/",
      permission_required("core.user")(
          views.SampleDetailView.as_view()
      ),
      name="detail",
  ),
  path(
      "detail/<int:pk>/pdf/",
      permission_required("core.user")(
          views.SamplePdfDetailView.as_view()
      ),
      name="detail_pdf",
  ),
  ...
]

Let’s try to understand how this works: First of all notice that we have two options, either create a PDF from an html string or from a normal template file. For the first option we pass the full html string to the get_resp_from_string and the context and we’ll get the rendered html (i.e the context will be applied to the template). For the second option we pass the filename of a django template and the context. Notice that there’s a small difference on how the template.render() method is called in the two methods.

After that we’ve got an html file saved in the resp string. We want to give this to wkhtmltopdf so as to be converted to PDF. To do that we first create a temporary file using the NamedTemporaryFile class and write the resp to it. Then we call wkhtmltopdf passing it this temporary file. Notice we use the subprocess.check_output function that will capture the output of the command and return it.

Finally we delete the temporary file and return the pdf as an HttpResponse.

We call the wkhtmltopdf like this:

c:\util\wkhtmltopdf.exe --page-size A4 --encoding utf-8 --footer-center [page] / [topage] --enable-local-file-access C:\Users\serafeim\AppData\Local\Temp\tmp_lh5r6f9.html -

The page-size can be changed to letter if you are in the US. The encoding should be utf-8. The —footer-center option adds a footer to the PDF page with the current page and the total number of pages. The —enable-local-file-access is very important since it allows wkhtmltopdf to access local files (in the filesystem) and not only remote ones. After that we’ve got the full path of our temporary file and following is the - which means that the pdf output will be on the stdout (so we’ll capture it with check_output).

Notice that there’s a commented out print command before the check_output call. If you have problems you can call this command from your command line to debug the wkhtmltopdf command (don’t forget to comment out the os.remove line to keep the temporary file). Also, wkhtmltopdf will output some stuff while rendering the command, for example something like:

Loading pages (1/6)
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done

You can pass the --quiet option to hide this output. However the output is useful to see what wkhtmltopdf is doing in case there are problems so I recommend leaving it on while developing. Let’s take a look at a problematic output:

Loading pages (1/6)
Error: Failed to load file:///static/bootstrap/css/bootstrap.min.css, with network status code 203 and http status code 0 - Error opening /static_edla/bootstrap/css/govgr_bootstrap.min.css: The system cannot find the path specified.
[...]
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done

The above output means that our template tries to load a css file that wkhtmltopdf can’t find and errors out! To understand this error, I had a line like this in my template:

<link href="{% static 'bootstrap/css/bootstrap.min.css' %}" rel="stylesheet">

which will be converted to a link like `/static/bootstrap/css/bootstrap.min.css. However notice that I tell wkhtmltopdf to render a file from my temporary directory, it doesn’t know where that link points to! Following this thing you need to be extra careful to include everything in your HTML-pdf template and not use any external links. So all styles must be inlined in the template using <style> tags and all images must be converted to data images with base64, something like:

<img src='data:image/png;base64,...>

To do that in python for a dynamic image you can use something like:

import base64

def convert_to_data(image):
  return 'data:image/xyz;base64,{}'.format(base64.b64encode(image).decode('utf-8'))

and then use that as your image src (notice I’m using image/xyz here for an arbitrary image, please use the correct image type if you know it i.e image/png or image/jpg).

If you’ve got a static image you want to include you can convert it to base64 using an online service like this, or read it with python and convert it:

import base64

with open('static/images/image.png', 'rb') as image:
  print(base64.b64encode(image.read()).decode('utf-8'))

Instead of a DetailView we could use the same approach for any kind of CBV. If you are to use the PDF response to multiple CBVs I recommend exporting the functionality to a mixin and inheriting from that also (see my CBV guide for more).

Finally, the big question in the room is why should I convert my template to a file and pass that to wkhtmltopdf, can’t I use the URL of my template, i.e pass wkhtmltopdf something like http://example.com/app/detail/321/?

By all means you can! This will also enable you to avoid using inline styles and media!! However keep in mind that the usual case is that this view will not be public but will need an authenticated user to access it; wkhtmltopdf is publicly trying to access it, it doesn’t have any rights to it so you’ll get a 404 or 403 error! Of course you can start an adventure on authenticating it somehow (and maybe doing something stupid) or you can just follow my lead and render it to a file :)

A forward and reverse proxy primer for the layman

Before some days I’d written an answer on HN where I explained as simply as possible how a forward and a reverse proxy is working and what is the difference between them. In this article I’m going to extend this answer a bit to make it a full post and clarify some things even more.

Forward and reverse proxies is an important concept that a lot of technical people aren’t familiar with. HTTP Proxying is a process of forwarding (HTTP) requests from one server to the other. So when an HTTP client issues a request to the server, the request will pass through the proxy server and be forwarded to the destination server (called the origin server). This explanation is true both for forward and reverse proxying.

Forward Proxy

A forward proxy is used when an HTTP Client (i.e a browser) wants to access resources in the internet but isn’t allowed to connect directly to the public internet so instead uses the proxy.

Usually companies don’t allow unrestricted access to the internet from their internal network. Thus the internal users would need to use a proxy to access the internet. This is the concept of the forward proxy. What happens is that when an internal user want to access an internet resource (i.e www.google.com) her client (i.e browser) will ask a specific server (the proxy server) for that resource. The client needs to be configured properly with the address of the proxy server.

So instead of http://www.google.com the browser will access http://proxy.company.com/?url=www.google.com and the proxy will fetch the results and return them to you. If the browser wants to access https://www.google.com without a configured proxy server it will get a network error.

Here’s an image that explains this:

Forward proxy

The internal client can access the internal web server directly without problems. However he cannot access the internet server directly so he needs to use the proxy to access it.

One thing that needs to be made crystal is that the fact that your browser works with the proxy does not mean that any other HTTP clients you use will also work. For example, you may want to run curl or wget to download some files from an external server; these programs will not work without setting a proxy (usually by setting the http_proxy and https_proxy environment variables or by passing a parameter). Also, the proxy only works for HTTP requests. If you are in a private network without external access you will not be able to access non-HTTP resources. For example you will not be able to access your non-company mail server (which uses either IMAP or POP3) from behind your company’s network. Typically, you’ll use a web client for accessing your mails.

So it seems that using a proxy heavily restricts the internal users usage of internet. What are the advantages of using a forward proxy?

  • Security: Since the internal computers of a company will not have internet access there’s no easy way for attackers to access these computers.
  • Content moderation: The company through the proxy can block access to various internet sites (i.e social network, gaming etc) that the users shouldn’t access during work.
  • Caching: The proxy server can have a cache so when multiple users access the same internet resource it will downloaded only once saving the company’s bandwidth.

Especially the security thing is so important that almost all corporate (or university etc) networks will use a proxy server and never allow direct access to the internet.

A well known, open source forward proxy server is Squid.

Reverse proxy

A reverse proxy is an HTTP server that “proxies” (i.e forwards) some (or all) requests it receives to a different HTTP server and returns the answer back. For example, a company may have a couple of HTTP servers in its internal network. These servers have private addresses and cannot be accessed through the internet. To allow external users to access these servers, the company will configure a reverse proxy server that will forward the requests to the internal servers as seen in the picture:

Reverse proxy

What happens is that the proxy server will forward requests that fulfill some specific criteria to other web servers. The criteria may be requests that have * a specific host (forward the requests that have a hostname of www.server1.company.com to the internal server named server1 and www.server2.company.com to the internal server named server2) * or a specific port (forward requests in the port 81 to server1 and requests in the port 82 to server2) * or even a particular path (forward requests with the path www.company.com/server1 to server1 and requests with the path www.company.com/server2 to server2)

or even other criteria that may be decided.

Let’s see some example of reverse proxying:

  • A characteristic example of reverse proxy is the well-known 3-tier architecture (web server / app server / database server). The web server is used to serve all requests but it “proxies” (forwards) some of the requests to the app server. This is used because the web server cannot serve dynamic replies but can serve static replies like for example files.
  • Offloading the SSL (https) security to a particular web server. This server will store the private key of your certificate and terminate the SSL connections. It will then forward the requests to the internal web servers using plain HTTP.
  • An HTTP load balancer will proxy the requests to a set of other servers based on some algorithm to share the load (i.e the HAProxy software load balancer or even a hardware load balancer)
  • A reverse proxy can be used to act as a security and DOS “shield” for your web servers. It will check the requests for common attack patterns and forward them to your servers only if they are safe
  • A reverse proxy can be used for caching; it will return cached versions of resources if they are available to avoid overloading the application servers
  • A CDN (content delivery network) is more or less a set of glorified reverse proxy servers that act as a first step for serving the user’s requests (based on the geographic location) also offering security protection and caching (this is what akamai or cloudflare do)

As can be seen from the previous examples there are a lot of apps that do reverse proxying, for example apache HTTP, nginx, HAProxy, varnish cache et al.

Notice that while there’s only one forward proxy, there could be a (large) chain of reverse proxies when accessing a remote server. Let’s take a look at a rather complex scenario: A user in a corporate network will access an application in another network. In this case the user’s request may pass through:

forward proxy (squid) -> security server / CDN (akamai) -> ssl termination (nginx) -> caching (varnish) -> web server (nginx again) -> app server (tomcat or gunicorn or IIS etc) as can be seen on the following image:

Reverse proxy

Notice that is this case (which is not uncommon) there are six (05) servers between your client and the application server!

One common problem with this is that unless all the intermediate servers are configured properly (by properly modifying and passing the X-Forwarded-For header) you won’t be able to retrieve the IP of the user that did the initial request.

Token Authentication for django-rest-framework

Introduction

In a a previous article I explained how to authenticate for your django-rest-framework API using the django-rest-auth package. Since then I have observed that various things have changed and most importantly that the library I used there (django-rest-auth) is not updated anymore and has been superseded by another one. Also, some of my information there is contradictory, especially the parts that deal with the session authentication and csrf protection.

Thus I’ve written this new article that betters describes a recommended authentication workflow using tokens. This workflow does not rely on sessions at all. Beyond that, is more or less the same as the previous one with some updates and clarifications where needed.

I have also updated the accompanying project of the previous article which can be found on https://github.com/spapas/rest_authenticate.

Before continuing with the tutorial, let’s take a look at what we’ll build here:

Our project

This is a single html page (styled with spectre.css) that checks if the user is logged in and either displays the login or logout button (using javascript). When you click the login you’ll get a modal in which you can enter your credentials which will be submitted through REST to the authentication endpoint and depending on the response will set a javascript variable (and a corresponding session/local storage key). Then you can use the “Test auth” button that works only on authenticated users and returns their username. Finally, notice that after you log out the “test auth” button returns a 403 access denied.

The javascript client uses token authentication so you can run the client in the same server as the server or in a completely different server (if you are using the proper CORS headers of course).

Some theory

Here I will try to explain a bunch of important concepts:

Sessions

After you log in with Django normally, your authentication information is saved to the session. The session is a bucket of information that the Django application saves about your visit — to distinguish between different visitors a cookie with a unique value named sessionid will be used. So, your web browser will send this cookie with each page request thus allowing Django to know which bucket of information is yours (and if you’ve authenticated know who are you). This is not a Django related concept but a general one (supported by most if not all HTTP frameworks) and is used to add state to an otherwise stateless medium (HTTP).

Since the sessionid cookie is sent not only with traditional but also with Ajax request it can be used to authenticate REST requests after you’ve logged in. This is what is used by default in django-rest-framework is a very good solution for most use cases: You login to django and you can go ahead and call the REST API through Ajax; the sessionid cookie will be sent along with the request and you’ll be authenticated automatically.

Now, although the session authentication is nice for using in browsers, you may need to access your API through a desktop or a mobile application where, setting the cookies yourself is not the optimal solution. Also, you may have an SPA that needs to access an API in a different domain; using using cookies for this is not easy - if possible at all.

CSRF protection

One important thing that you should be aware if you are going to use session authentication for your API is the CSRF protection. This is a mechanism that helps prevent cross-site request forgery (CSRF) attacks. A CSRF attack works like this: Let’s suppose that site A is a bank, and has a form with an email and a money amount. When the user submits the form via POST it will send this much money to the entered email using Paypal. Now, site B is a malicious site. When the user visits site B, it will automatically generate a POST request containing the malicious user’s email and the money he wants and submit it to site A. Now, if the user is authenticated with sessions on site A then site A will think that this is a valid form submission and will actually process the form as normally and send the money to the malicious user!

As you can understand this is a very serious and easy to exploit attack. To prevent this attack, the CSRF protection is used: In order to submit the form on site A, the request must contain a unique string (the CSRF token) that is generated automatically by site A. Thankfully, site B cannot access this token and thus cannot submit the form.

The CSRF situation is only related to sessions. If you are not using sessions then CSRF protection is not needed because there’s no way for site B to submit the form on site A (for example, with TokenAuthentication, site B cannot access the token that site A has).

However if you are using sessions then you must be extra careful to protect your POST views against CSRF attacks. Django does this by default so you don’t need to do anything fancy. However, when you actually want to submit a form using an API with sessions you must be careful to also include the CSRF token as explained in the Django docs about the topic (CSRF protection).

Tokens

For cases where you can’t use the session to authenticate, django-rest-framework offers a different authentication method called TokenAuthentication_. Using this method, each user of the Django application is correlated with a random string (Token) which is passed along with each request at its header thus the Django app can authenticate the user using this token. The token is retrieved when the user logs using his credentials and is saved in the browser.

One thing that may seem strange is that since both the session cookie and a token are set through HTTP Headers why all the fuss about tokens? Why not just use the session cookie and be done with it? Well, there are various reasons - here’s a rather extensive article explaining some of them. Some of the reasons are that a token can be valid forever while the session is something ephemeral - beyond authorization information, sessions may keep various other data for a web application and are expired after some time to save space. Also, since tokens are used for exactly this (authentication) they are much easier to use and reason about. Finally, as I’ve already explained, sharing cookies by multiple sites is not something you’d like to do. Actually, to make things easier for you just follow this rule: If your API will be run on a different domain than your client (i.e api.example.com and www.example.com) or your client not run on the web (i.e. is a desktop/mobile app) then you must not use session authentication. Use token authentication as proposed here or whatever else you may want that doesn’t rely on sessions.

CORS

Another thing that must concern the people that will want to use an API is the CORS situation. By default cross-origin requests are not allowed, i.e site B cannot issue Ajax requests to site A. Each server can be configured to allow cross-origin requests from other servers. This means that if you have a server api.example.com that is used as a backend and a server www.example.com that will serve your front-end, you can configure api.example.com to allow requests only from www.example.com.

By default Django does not allow any cross origin requests and you need to use the django-cors-headers package to properly configure it.

Notice that CORS protection is enforced by the Browser. For example if you have build a mobile app and are consuming an API in api.example.com then CORS protection does not apply to your http client.

Installation & configuration

The project will use django-rest-framework, dj-rest-auth and django-cors-headers.

To install django-rest-framework and dj-rest-auth just follow the instructions here i.e just add 'rest_framework', 'rest_framework.authtoken' and 'dj_rest_auth' to your INSTALLED_APPS in settings.py and run migrate.

To install django-cors-headers follow the the setup instructions: Add "corsheaders" to your INSTALLED_APPS and "django.middleware.common.CommonMiddleware" to your MIDDLEWARE in settings.py. Then you can use the CORS_ALLOWED_ORIGINS setting to configure which origins are allowed to make requests to your project. Let’s suppose that you are running your project at 127.0.0.1:8000 and you want to allow requests from a client running at 127.0.0.1:8001. You can do this by adding the following to your settings.py: CORS_ALLOWED_ORIGINS = ['http://127.0.0.1:8001', 'http://localhost:8001']. Actually, try running the project with and without that setting and see how the javascript client behaves.

Since I won’t be adding any other apps to this project (no models are actually needed), I’ve added two directories static and templates to put static files and templates there. This is configured by adding the 'DIRS' attribte to TEMPLATES, like this:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            os.path.join(BASE_DIR, 'templates'),
        ],
        // ...

and adding the STATICFILES_DIRS setting:

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
]

The remaining setting are the default as were created by django-admin startproject.

Urls

I have included the the following urls to urls.py:

urlpatterns = [
    path('admin/', admin.site.urls),
    path('test_auth/', TestAuthView.as_view(), name='test_auth', ),
    path('rest-auth/logout/', LogoutViewEx.as_view(), name='rest_logout', ),
    path('rest-auth/login/', LoginView.as_view(), name='rest_login', ),
    path('', HomeTemplateView.as_view(), name='home', ),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

These are: The django-admin, a test_auth view (that works only for authenticated users and returns their username), a view (LogoutViewEx) that overrides the rest-auth REST logout-view (I’ll explain why this is needed in a minute), the rest-auth REST login-view, the home template view (which is the only view implemented) and finally a mapping of your static files to the STATIC_URL.

The LoginView is the default provided by the dj-rest-auth project. One thing to consider is that this view will check if the credentials you pass are valid and return a valid token for your user. However, it will also optionally login the user using sessions (i.e create a new session and return a sessionid cookie). This is configured by the REST_SESSION_LOGIN option which by default is True.

To test this functionality, try logging in using this login view with a superuser and then visit the django-admin. You will see that you are already logged in. Now, logout and add (or change) REST_SESSION_LOGIN=False to your settings.py. Login again from the rest view and now if you visit the django-admin you should see that you need to login again.

Another way to test this is by checking out the response headers of the POST to rest-auth/login/ from your browser’s development tools. When you are using REST_SESSION_LOGIN=True (or you haven’t defined it since by default it is true) you’ll see the following Set-Cookie line:

sessionid=pw8rp7l7yy33lk7geuxbczaleh35w9je; expires=Wed, 08 Sep 2021 08:29:40 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax

This cookie won’t be set if you login again with REST_SESSION_LOGIN=False.

The views

I’ve defined three views in this application - the HomeTemplateView, the TestAuthView and the LogoutViewEx view that overrides the normal LogoutView of django-rest-auth.

HomeTemplateView

The HomeTemplateView is a simple TemplateView that just displays an html page and loads the client side code - we’ll talk about it later in the front-side section. This is more or less similar (without the django-stuff) with the standalone client page that can be found on client/index.html.

TestAuthView

The TestAuthView is implemented like this:

class TestAuthView(APIView):
    authentication_classes = (authentication.TokenAuthentication,)
    permission_classes = (permissions.IsAuthenticated,)

    def get(self, request, format=None):
        return Response("Hello {0}!".format(request.user))

    def post(self, request, format=None):
        return Response("Hello {0}! Posted!".format(request.user))

This is very simple however I’d like to make a few comments about the above. First of all you see that I’ve defined both a get and a post method. When you use the token authentication you’ll see that the post method will work without the need to provide a csrf token as already discussed before.

Authentication and permission

Notice that both authentication_classes and permission_classes are included in the TestAuthView. These options define:

  • which method will be used for authenticating access to the REST view i.e finding out if the user requesting access has logged in and if yes what’s his username (in our case only TokenAuthentication will be used)
  • if the user is authorized (has permission) to call this REST view (in our case only authenticated users will be allowed)

The authentication and permission classes can be set globally in your settings.py using REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] and REST_FRAMEWORK['DEFAULT_PERMISSION_CLASSES'] or defined per-class like this. If I wanted to have the same authentication and permission classes defined in my settings.py so I wouldn’t need to set these options per-class I’d add the following to my settings.py:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
    ),
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.IsAuthenticated',
    ),
}

Please keep in mind that you haven’t defined these in your views or your settings, they will have the following default values:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.BasicAuthentication'
    ),
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.AllowAny',
    ),
}

The above mean that if you don’t define authentication and permission classes anywhere then the REST views will use either session authentication (i.e the user has logged in normally using the Django login views as explained before) or HTTP basic authentication (the request provides the credentials in the header using traditional HTTP Basic authentication) and also that all users (logged in or not) will be allowed to call all APIs (this is probably not something you want).

Tokens

The TokenAuthentication that we are using for the TestAuthView means that for every request a valid token must be passed (there’s no concept of state in HTTP so you need to pass it whenever you communicate with the server).

The tokens are normal object instances of rest_framework.authtoken.models.Token and you can take a look at them (or even add one) through the Django admin (auth token - tokens). You can also even do whatever you normally would do to an object instance, for example:

>>> [ (x.user, x.key) for x in Token.objects.all()]
[(<User: root>, 'db4dcc1b9d00d1af74fb3cb41e1f9e673208485b')]

To authenticate with a token (using TokenAuthentication), you must add an extra header to your request with the format Authorization: Token token for example in the previous case root would add Authorization: Token db4dcc1b9d00d1af74fb3cb41e1f9e673208485b. To do this you’ll need something client-side code which we’ll see in the next section.

To debug your authentication with curl you can just do something like this:

curl http://127.0.0.1:8000/test_auth/ -H "Authorization:Token db4dcc1b9d00d1af74fb3cb41e1f9e673208485b"

Try it with a valid and invalid token and without providing a token at all and see the response each time.

dj-rest-auth

So, django-rest-framework provides the model (Token) and the mechanism (add the extra Authentication header) for authentication with Tokens. What it does not provide is a simple way to create/remove tokens for users: This is where the dj-rest-auth project comes to the rescue! Its login and logout REST views will automatically create (and delete) tokens for the users that are logging in.

As already described above, the login view will also authenticate the user using the session when the REST_SESSION_LOGIN is set to True (default) - this means that if a user logs in using the login REST endpoint he’ll then be logged in normally to the site and be able to access non-REST parts of the site (for example the django-admin).

Also, if the user logs in through the dj-rest-auth REST end point and if you have are using SessionAuthentication to one of your views then he’ll be able to authenticate to these views without the need to pass the token (make sure you understand why).

LogoutViewEx

Finally, let’s take a look at the LogoutViewEx:

class LogoutViewEx(LogoutView):
    authentication_classes = (authentication.TokenAuthentication,)

This class only defines the authentication_classes attribute. Is this really needed? Well, it depends on you project. If you take a look at the source code of LogoutView (https://github.com/iMerica/dj-rest-auth/blob/master/dj_rest_auth/views.py#L131) you’ll see that it does not define authentication_classes. This, as we’ve already discussed, means that it will fall-back to whatever you have defined in the settings (or the defaults of django-rest-framework).

So, if you haven’t defined anything in the settings then you’ll get the by default the SessionAuthentication and BasicAuthentication methods (hint: not the TokenAuthentication). This means that you won’t be able to logout when you pass the token (but will be able to logout from the web-app after you login - why?). So to make everything crystal and be able to reason better about the behavior I specifically define the LogoutViewEx to use the TokenAuthentication to properly log out your user. This of course means that you need to pass the token to your logout view also or else there won’t be any way to associate the request with a user to log out.

The client side scripts

I’ve included all client-side code to a home.html template that is loaded from the HomeTemplateView. Also, the same code has been included in client/index.html. This is a completely standalone javascript client that you can run in a different http server than your Django server, for example by running py -3 -m  http.server 8001 from the client folder and visiting http://127.0.0.1:8001.

The client-side code has been implemented only with jQuery because I think this is the library that most people are familiar with - and is really easy to be understood even if you are not familiar with it. It more or less consists of five sections in html:

  • A user-is-logged-in section that displays the username and the logout button
  • A user-is-not-logged-in section that displays a message and the login button
  • A test-auth section that displays a button for calling the TestAuthView with GET defined previously and outputs its response
  • A test-auth POST section that displays a button for calling the TestAuthView with POST defined previously and outputs its response
  • The login modal

Here’s the html (using spectre.css for styling):

<div class="container grid-lg">
<h2>Test</h2>
<div class="columns" id="non-logged-in">
    <div class='column col-3'>
        You have to log-in!
    </div>
    <div class='column col-3'>
        <button class="btn btn-primary"  id='loginButton'>Login</button>
    </div>
</div>
<div class="columns" id="logged-in">
    <div class='column col-3'>
        Welcome <span id='span-username'></span>!
    </div>
    <div class='column col-3'>
        <button class="btn btn-primary"  id='logoutButton'>Logout</button>
    </div>
</div>
<hr />
<div class="columns" id="test">
    <div class='column col-3'>
        <button class="btn btn-primary"  id='testAuthButton'>Test auth</button>
    </div>
    <div class='column col-9'>
        <div id='test-auth-response' ></div>
    </div>
</div>
<hr />
<div class="columns" id="test">
    <div class='column col-3'>
        <button class="btn btn-primary"  id='testAuthPostButton'>Test auth (POST)</button>
    </div>
    <div class='column col-9'>
        <div id='test-auth-post-response' ></div>
    </div>
</div>
</div>

<div class="modal" id="login-modal">
    <a href="#close" class="modal-overlay close-modal" aria-label="Close"></a>
    <div class="modal-container">
        <div class="modal-header">
            <a href="#close" class="btn btn-clear float-right close-modal" aria-label="Close"></a>
            <div class="modal-title h5">Please login</div>
        </div>
        <div class="modal-body">
            <div class="content">
                <form>
                    {% csrf_token %}
                    <div class="form-group">
                        <label class="form-label" for="input-username">Username</label>
                        <input class="form-input" type="text" id="input-username" placeholder="Name">
                    </div>
                    <div class="form-group">
                        <label class="form-label" for="input-password">Password</label>
                        <input class="form-input" type="password" id="input-password" placeholder="Password">
                    </div>
                    <div class="form-group">
                        <label class="form-checkbox" for="input-local-storage">
                            <input type="checkbox" id="input-local-storage" /> <i class="form-icon"></i>  Use local storage (remember me)
                        </label>
                    </div>
                </form>
                <div class='label label-error mt-1 d-invisible' id='modal-error'>
                    Unable to login!
                </div>
            </div>
        </div>
        <div class="modal-footer">

            <button class="btn btn-primary" id='loginOkButton' >Ok</button>
            <a href="#close" class="btn close-modal" >Close</a>
        </div>
    </div>
</div>

The html is very simple and I don’t think I need to explain much - notice that the #logged-in and #non-logged-in sections are mutually exclusive (I use $.show() and $.hide() to show and hide them) but the #test section is always displayed so you’ll be able to call the test REST API when you are and are not authenticated. For the modal to be displayed you need to add an active class to its #modal container.

For the javascript, let’s take a look at some initialization stuff:

var g_urls = {
    'login': '{% url "rest_login" %}',
    'logout': '{% url "rest_logout" %}',
    'test_auth': '{% url "test_auth" %}',
};
var g_auth = localStorage.getItem("auth");
if(g_auth == null) {
    g_auth = sessionStorage.getItem("auth");
}

if(g_auth) {
    try {
        g_auth = JSON.parse(g_auth);
    } catch(error) {
        g_auth = null;
    }
}

var initLogin = function() {
    if(g_auth) {
        $('#non-logged-in').hide();
        $('#logged-in').show();
        $('#span-username').html(g_auth.username);
        if(g_auth.remember_me) {
            localStorage.setItem("auth", JSON.stringify(g_auth));
        } else {
            sessionStorage.setItem("auth", JSON.stringify(g_auth));
        }
    } else {
        $('#non-logged-in').show();
        $('#logged-in').hide();
        $('#span-username').html('');
        localStorage.removeItem("auth");
        sessionStorage.removeItem("auth");
    }
    $('#test-auth-response').html("");
    $('#test-auth-post-response').html("");
};

First of all, I define a g_urls window/global object that will keep the required REST URLS (login/logout and test auth). These are retrieved from Django using the {% url %} template tag and are not hard-coded (in the js only client they are hard-coded of course). After that, I check to see if the user has authenticated before. Notice that because this is client-side code, I need to do that every time the page loads or else the JS won’t be initialized properly! The user login information is stored to an object named g_auth and contains three attributes: username, key (token) and remember_me.

To keep the login information I use either a key named auth to either the localStorage or the sessionStorage. The sessionStorage is used to save info for the current browser tab (not window) while the localStorage saves info for ever (until somebody deletes it). Thus, localStorage can be used for implementing a “remember me” functionality.

The final function we define here, initLogin (which is called a little later) checks to see if there is login information and hides/displays the correct things in html. It will also set the local or session storage (depending on remember me value).

After that, we have some client side code that is inside the $() function which will be called after the page has completely loaded:

$(function () {
    initLogin();

    $('#loginButton').click(function() {
        $('#login-modal').addClass('active');
    });

    $('.close-modal').click(function() {
        $('#login-modal').removeClass('active');
    });

    $('#testAuthButton').click(function() {
        $.ajax({
            url: g_urls.test_auth,
            method: "GET",
            beforeSend: function(request) {
                if(g_auth) {
                    request.setRequestHeader("Authorization", "Token " + g_auth.key);
                }
            }
        }).done(function(data) {
            $('#test-auth-response').html("<span class='label label-success'>Ok! Response: " + data);
        }).fail(function(data) {
            $('#test-auth-response').html("<span class='label label-error'>Fail! Response: " + data.responseText + " (status: " + data.status+")</span>");
        });
    });

    $('#testAuthPostButton').click(function() {
        // Same as with the GET
    });

    // continuing below ...

The first thing happening here is to call the initLogin function to properly initialize the page and then we add a couple of handlers to the click buttons of the #loginButton (which just displays the modal by adding the active class ), .close-modal class (there are multiple ways to close the modal thus I use a class which just removes that active class) and finally to the #testAuthButton and #testAuthPostButton#. These button will do a GET and POST request to the g_urls.test_auth we defined before. The important thing to notice here is that we add a beforeSend attribute to the $.ajax request which, if g_auth is defined, adds an Authorization header with the token in the form that django-rest-framework TokenAuthentication expects and as we’ve already discussed above:

beforeSend: function(request) {
    if(g_auth) {
        request.setRequestHeader("Authorization", "Token " + g_auth.key);
    }
}

If this ajax call returns without errors (the done part of the ajax call) we just add the data to a green label else if there’s an error (fail part) we add the response text and status to a red label. You can try clicking the buttons and you see that only if you’ve logged in you will succeed in this call. Also, notice that both GET and POST requests work normally without the need to also include a csrf token (I hope you understand why by now).

Let’s now take a look at the #loginOkbutton click handler (inside the modal):

$('#loginOkButton').click(function() {
    var username = $('#input-username').val();
    var password = $('#input-password').val();
    var remember_me = $('#input-local-storage').prop('checked');
    if(username && password) {
        console.log("Will try to login with ", username, password);
        $('#modal-error').addClass('d-invisible');
        $.ajax({
            url: g_urls.login,
            method: "POST",
            data: {
                username: username,
                password: password
            }
        }).done(function(data) {
            console.log("DONE: ", username, data.key);
            g_auth = {
                username: username,
                key: data.key,
                remember_me: remember_me
            };
            $('#login-modal').removeClass('active');
            initLogin();
        }).fail(function(data) {
            console.log("FAIL", data);
            $('#modal-error').removeClass('d-invisible');
        });
    } else {
        $('#modal-error').removeClass('d-invisible');
    }
});

All three user inputs (username, password, remember_me) are read from the form and if both username and password have been defined an Ajax request will be done to the g_urls.login url. We pass username and password as the request data. Now, if there’s an error (fail) I just display a generic message (by removing it’s d-invisible class) while, if the request was Ok I retrieve the key (token) from the response, initialize the g_auth object with the username, key and remember_me values and call initLogin to show the correct divs and save to the session/local storage.

Finally, here’s the code for logout (still inside the $(function () {):

    $('#logoutButton').click(function() {
        console.log("Trying to logout");
        $.ajax({
            url: g_urls.logout,
            method: "POST",
            beforeSend: function(request) {
                request.setRequestHeader("Authorization", "Token " + g_auth.key);
            }
        }).done(function(data) {
            console.log("DONE: ", data);
            g_auth = null;
            initLogin();
        }).fail(function(data) {
            console.log("FAIL: ", data);
        });
    });

}); // End of $(function () {

The code here is very simple - just do a POST to the g_urls.logout and if everything is ok delete the g_auth values and call initLogin() to show the correct divs and remove the auth key from local/session storage. Notice that when you POST to the logout REST end-point, you need to also add the Authorization header with the token or else (since we’ve defined only TokenAuthentication for the authentication_classes for the LogoutViewEx class) there won’t be any way to correlate the request with the user and log him out!

Conclusion

Using the info presented on this article you should be able to properly login and logout to Django using REST and also call REST end-points using the TokenAuthentication.

I recommend using the curl utility to try to call the rest end point with various parameters to see the response. Also, you change the LogoutViewEx with the default django-rest-auth LogoutView and then try logging out through the web-app and through curl and see what happens when you try to access the test-auth end-point.

As a final remark, a couple of thing to note:

  • You can use the django-rest-knox package to improve the functionality and security of your REST tokens (by allowing multiple tokens per user, storing them hashed in the database and configuring expiration times for the tokens)
  • If you are using Apache and mod_wsgi to run you Django project you need to set the WSGIPassAuthorization option to on in order to pass the Authorization header to your Django app.

Changing choices to a ForeignKey using Django migrations

One common requirement I’ve seen in projects is that a model will start with a choices CharField but in the future this field will need to be converted to a normal foreign key to another model. This is such a common requirement that I’ve concluded that you need to double think before using choices because there’s a high possibility that in the lifetime of your project you’ll also need to convert it to a foreign key.

For example, let’s suppose you’ve got the following model:

CATEGORY_CHOICES = [
  ('cat1', 'Category 1 name',),
  ('cat2', 'Category 2 name',),
  ('cat3', 'Category 3 name',),
  ('cat4', 'Category 4 name',),
]

class Sample(models.Model):
  name = models.CharField(max_length=100)
  category = models.CharField(max_length=100, choices=CATEGORY_CHOICES)

You will need to convert it like this

class Category(models.Model):
  name = models.CharField(max_length=100)

  def __str__(self):
    return self.name

class Sample(models.Model):
  name = models.CharField(max_length=100)
  category = models.ForeignKey('Category', on_delete=models.PROTECT)

There are various reasons that you may be forced to convert the choices field to a ForeignKey, some are:

  • Your site administrators may need to sometime change these choices themselves
  • You may want to add some properties to each choice i.e if a choice is active or not
  • The choices info is local in your django project. If for some reason you want your data to be used by a different project (for example execute a reporing query directly from the database) you’ll just get the code for each choice (and not its name) leading you to ugly case statements in your queries to display the name of each choice. Furthermore, if the choices do change you’ll need to change them in two places (your django project and your reporting queries)
  • The choice thing, although is very helpful and quick to implement leads to a non-normalized design. The name of each choice will be a string that would be duplicated to each row that has that particular choice.

In a previous article I had provided a recipe on how to properly normalize a database table containing a choices field like this using PL/pgSQL. This script should work in this case also but if you have a Django project then you should use migrations to do the conversion.

So let’s see how to convert our category choices field to a Foreign Key using django migrations!

The proper way to do it is in three distinct steps/migrations:

  1. Create the Category model and add a foreign key to it in the Sample model. You should not remove the existing choices field! So you’ll need to add another field to Sample for example named category_fk.
  2. Create a data migration to run a python script that will read the existing Sample instances and fill their category_fk field based on their category field.
  3. Remove the category field from Sample model and rename category_fk to category.

Let’s go through the steps one by one:

First we will change our initial models.py like this:

class Category(models.Model):
  name = models.CharField(max_length=100)

class Sample(models.Model):
  name = models.CharField(max_length=100)
  category = models.CharField(max_length=100, choices=CATEGORY_CHOICES)
  category_fk = models.ForeignKey('Category', on_delete=models.PROTECT, null=True)

So I’ve just added the Category model and the category_fk field to the Sample model. Notice the category choices field is still there since I need it to fill my category_fk! Also notice that I’ve added a null=True to the category_fk so it will allow the field to be added with a null value to the existing. I will fix that later. We can create and run an automatic migration now:

C:\progr\py3\migrations_tutorial>python manage.py makemigrations
Migrations for 'core':
core\migrations\0002_auto_20210715_0836.py
  - Create model Category
  - Add field category_fk to sample

C:\progr\py3\migrations_tutorial>python manage.py migrate
  Operations to perform:
    Apply all migrations: admin, auth, contenttypes, core, sessions
  Running migrations:
    Applying core.0002_auto_20210715_0836... OK

So now all my rows have an empty category_fk field.

For the second step, we will create the data migration that will fill the category_fk field. First of all let’s create an empty migration (notice my app is called core):

C:\progr\py3\migrations_tutorial>python manage.py makemigrations --empty core
Migrations for 'core':
  core\migrations\0003_auto_20210715_0844.py

Let’s take a look at what Django has created for us:

from django.db import migrations

class Migration(migrations.Migration):

  dependencies = [
      ('core', '0002_auto_20210715_0836'),
  ]

  operations = [
  ]

This is an empty migration file, it just says that it will be run after the previous migration we just created. We’ll need to add an operation to it that will do the needed work of filling the category_fk field.

This can be done like this:

from django.db import migrations

def fill_category_fk(apps, schema_editor):
  Sample = apps.get_model('core', 'Sample')
  Category = apps.get_model('core', 'Category')
  for sample in Sample.objects.all():
    sample.category_fk, created = Category.objects.get_or_create(name=sample.category)
    sample.save()

class Migration(migrations.Migration):

  dependencies = [
      ('core', '0002_auto_20210715_0836'),
  ]

  operations = [
      migrations.RunPython(fill_category_fk),
  ]

The above should be straight forward. The only thing to notice is that you should use migrations.RunPython to declare that the migration will need to run some python code. Notice that RunPython takes a second parameter with another function which will be run during the backwards migration. In our case we don’t really need it, since we omit it, it will throw an error if you try to apply this migration backwards.

The fill_category_fk uses the apps.get_model function to have access to the models it needs. You should use this instead of importing the models directly because the current state of the database models may not be the same as the state that the migration expects. I’m just using get_or_create to insert or retrieve the proper Category instance (remember that get_or_create returns an (instance, created) tuple so we need to use the first element).

Now we can try running the migration:

C:\progr\py3\migrations_tutorial>python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, core, sessions
Running migrations:
  Applying core.0003_auto_20210715_0844... OK

If any errors happened you will see the stack trace here and you will need to fix them. Don’t worry, the state of your database will not be changed until the migration finishes.

Now our database has both the (old) category and the (new) category_fk fields. Each will have the same value!

Now we need to remove the old category field and rename the existing category_fk. Let’s do it!

class Sample(models.Model):
  name = models.CharField(max_length=100)
  category = models.ForeignKey('Category', on_delete=models.PROTECT, null=True)

  def __str__(self):
      return self.name

And run the migration:

C:\progr\py3\migrations_tutorial>python manage.py makemigrations
Migrations for 'core':
  core\migrations\0004_auto_20210715_0909.py
    - Remove field category_fk from sample
    - Alter field category on sample

Uh oh! This does not seem to do what I want. Let’s take a peek at the generated migration file:

class Migration(migrations.Migration):

  dependencies = [
      ('core', '0003_auto_20210715_0844'),
  ]

  operations = [
      migrations.RemoveField(
          model_name='sample',
          name='category_fk',
      ),
      migrations.AlterField(
          model_name='sample',
          name='category',
          field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.PROTECT, to='core.Category'),
      ),
  ]

This will remove the category_fk field we just filled from our model and then try to convert the old category field to a foreign key! If you try to run the migration you’ll get an exception because the existing category field cannot be converted to a ForeignKey!

It seems that Django migrations isn’t so smart after all… To resolve that we could just create two separate migrations: One to remove the old category field and the other to rename the category_fk field to category. Django would know then that we have renamed the category_fk field. This method works fine but if you are using category in your admin (or forms) django will complain with errors like this:

<class 'core.admin.SampleAdmin'>: (admin.E108) The value of 'list_display[1]' refers to 'category', which is not a callable, an attribute of 'SampleAdmin', or an attribute or method on 'core.Sample'.

So you’ll need to rename to fix this before running the migration (and if you actually fix it you may just bite the bullet and use category_fk to avoid re-renaming it back to category).

This is rather a pain so I’ll give you another way: Edit the created migration file to do exactly what you need, i.e remove the existing category field and rename category_fk to category. Here’s the migration file:

class Migration(migrations.Migration):

  dependencies = [
      ('core', '0003_auto_20210715_0844'),
  ]

  operations = [
      migrations.RemoveField(
          model_name='sample',
          name='category',
      ),
      migrations.RenameField(
          model_name='sample',
          old_name='category_fk',
          new_name='category',
      ),
  ]

So in this migration we first remove the existing category field and then we rename the category_fk field to category. Let’s try to run it:

C:\progr\py3\migrations_tutorial>python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, core, sessions
Running migrations:
  Applying core.0004_auto_20210715_0909... OK

Success!