The Setup

We had a pretty simple architecture.

A web app built with Java and another AI app built using FastAPI and LangGraph.

The main app (Java) would send a request to the AI service. That service would process it and send the result back via a webhook. Two completely separate services. Two separate Postgres databases. Every request and response was stored in both sides.

That separation ended up saving us a lot of guesswork later. When things broke, it was obvious which side to look at.

The First Crash

It happened during peak US hours. Everything just stopped and the IP was not responding.

An alert fired. Two health checks were fine, but the EC2 reachability check failed. The instance was inaccessible.

I checked logs if there were incoming requests around that time, but none of them made it through. The database had no record of them. It was like they vanished mid-flight.

After a couple of hours, we restarted the instance. Things came back to life like nothing had happened.

Not great. But also not catastrophic. This service wasn’t user-facing in real time, so delays were acceptable. We could reprocess missed requests because each one was tied to a specific asset. So we did that and moved on.

But “restart and pray” isn’t a real solution. Something was clearly wrong.

Chasing the Obvious (and Wrong) Idea

My first thought was: memory leak. It crashed, we restarted, and everything worked again. That’s usually a dead giveaway. So I tried to reproduce it locally. Same instance type. Same specs and I spammed it with concurrent requests.

Nothing.

No crash. No slowdown. It just kept running like it had something to prove. Still, I didn’t trust it. Maybe it was too many parallel requests eating up a lot of RAM. So I added a Redis queue and limited processing to two concurrent threads.

Good change overall. Felt responsible. But, this was still not going to take us to the root of the problem.

Actually Looking at Memory

Then I realized something embarrassingly basic. We weren’t even tracking RAM usage. By default, EC2 doesn’t show memory metrics. You have to install an agent for that. Which… we hadn’t done. So I set that up. And there it was. Memory usage was climbing steadily and never dropping. Eventually, the system would choke and die.

The weird part was, It didn’t happen often. Maybe once every couple of weeks. It only occurred 3 times in total. Not frequent enough to easily debug, but frequent enough to be a real problem. At this point, it felt like trying to catch a bug that only shows up when it feels like it.

The Moment Things Clicked

The breakthrough didn’t come from staring harder at logs. It came from a random question that popped into my head while reading something unrelated: “How does this system keep track of multiple threads without mixing things up?” That had me digging.

Turns out, it uses a check pointer to store thread state. And if you don’t configure anything external, it just keeps everything in memory. Every request. Every thread. Every checkpoint.

All sitting in RAM.

And this is the important part; none of it was being cleaned up (This is a design decision that allows resuming the conversation at any point in time, although a debatable better decision would have been to default to a file based check pointer, rather than an In memory one). So with every request, memory usage ticked up a little. And then a little more. And then a little more.

Until eventually, the instance just ran out of room and gave up.

Why I Never Saw It in Dev

This also explained why I couldn’t reproduce it. In development, the service restarted all the time, every deploy, every change. Each restart wiped the memory clean. So the leak never had time to build up.

In production, though, the service stayed up for days or weeks. That’s where the problem had space to grow.

The Fix

Once the problem made sense, the fix was straightforward. We stopped storing state in memory and moved it to Postgres. Now, instead of piling everything into RAM, it gets persisted properly. Memory usage stays flat, no matter how many requests come in.

We also switched to a more appropriate instance type. Since then, no crashes. No slow memory creep. Just a stable system doing what it’s supposed to.

What I Learned (the Hard Way)

A few things I wish I had done differently:

Write things down while debugging. You will forget details. Numbers, timestamps, weird observations all of it. Don’t trust your memory. It’s unreliable when you need it most.

Don’t assume your monitoring is enough. We didn’t even have RAM metrics. That alone delayed the diagnosis way more than it should have.

Know your tools beyond the happy path. Defaults are often designed for convenience, not scale. Something that works perfectly in dev can quietly destroy you in production.

Dev and prod behave differently in subtle ways. Frequent restarts in dev were hiding the issue entirely. That gap matters more than it seems.

Sometimes the answer comes from sideways thinking. The breakthrough didn’t come from grinding harder. It came from curiosity about something loosely related. That’s often how these things work.

Looking back, none of it was very complex. But, the chasing through the invisible is what made this find stick.

• For some, the right candidate is someone who once offered them a helping hand.

• For others, it’s the representative they see regularly or feel personally connected to.

• For some, it’s the party that happened to be in power when a decision favoring them was made.

• For others, it’s about opposing someone they dislike (supporting a rival out of resentment rather than conviction (quite visible in uml-ncp supporters).

• And for some, it’s about personality; the way certain leaders present themselves.

These are thought processes I’ve observed among people in my community when asked about their voting decisions. Many of these choices feel understandable, but also incomplete.

I strongly believe that voting decisions shouldn’t be driven solely by emotion. Political rationale should play a central role. Evaluating a candidate’s policies, their track record, competence, integrity, and long-term vision. Asking, what realistic change a candidate can bring, how their policies, if in action will affect me and the community, do they have a history of delivering with accountability?
Having said so have you truly thought through to come to your conclusion? Have you convinced yourself? There's no right decision here. But, I hope you're not simply rationalizing an emotional impulse.

We must have heard how python isn’t truly parallel. Why does it happen? What actually is the Global Interpreter Lock and how it is a culprit behind the lack of true parallelism in python. I try to explain it while also explaining a few internal workings of the language with a very high level oversimplification of the actual workings.

In short, GIL is a lock, that allows only one thread at a time to execute python bytecode within the cpython interpreter.

Prerequisites

To understand GIL, we need to understand the following concepts.

1. IO Bound v/s CPU Bound

   The core reason behind need for building parallel systems is the performance bottleneck which arises from either IO wait or processor’s limit.

   • An IO bound task spends most of it’s time waiting for something outside the CPU to respond. For eg: Querying a database depends on the database, downloading a file from internet etc.

   • A task is CPU bound, if it’s speed depends entirely on how fast the processor can crunch numbers. That means, a faster processor can complete the task faster unlike the case in IO bound. Eg: image processing, mathematical calculations etc.

In IO bound task, processor speed isn’t the bottleneck while in CPU bound, it is.

2. Threads v/s Processes

   • Threads

     A thread is a lightweight unit of execution that lives inside the process. Every process starts with at least one thread. Multiple threads in a process, share the same memory, same interpreter and same GIL.

   • Process

     It is an independent running instance of a program. It has it’s own python interpreter, it’s own memory space and it’s own GIL. If we spawn a separate process for each task, then we achieve true parallelism because each task will have a GIL of it’s own. But, this isn’t an efficient approach and we’ll understand why later.

How Python manages memory?

Python uses a technique called Reference Counting. In Reference Counting, each object created in python have a reference count, that tracks the number of references that point to that object. When reference count reaches zero, the memory occupied is freed.

Python also uses Garbage collection as a way to manage memory. But the problem of GIL arises from Reference Counting. Why was reference counting preferred in the first place is a separate topic of discussion, which we will not go in this article.

Problem that GIL solves

The reference count variable is susceptible to race conditions if two threads are allowed to read or write to that value simultaneously. This can result in leaked memory that is the memory which is never released even after the end of the program. Or it can incorrectly release the memory while a reference to that object still exists.

So, GIL is a OS level lock. It is a single lock, that the thread must acquire before executing the python byte code. This way, only one thread will be able to modify the reference count. Due to this reason, even if we spawn multiple threads, only one thread can perform a CPU-bound task at a time. However, during IO, the GIL is released. During this case, the threads can work in parallel too.

How threads can acquire GIL has a different story. In the old way(before python3.2), it was based on ticks. A check occurred every 100 ticks to check if the thread should give up the lock. But modern way, is based on timed wait. A thread will run, until second thread requests for the lock. After the request is made, the second thread waits for certain time and forces the first thread to drop the lock.

We can achieve true parallelism in python if we spawn multiple processes instead of the thread. But creating a process is an expensive operation and may not necessarily provide better speed even with parallelism due to the overhead of launching a separate interpreter and memory spaces. In some cases, a single threaded program can be faster than a program using multiple processes.

What could have been an alternative solution to GIL?

• One idea would be to provide a single lock to each data structure that are shared across threads, so that can can’t be modified inconsistently. But, this solution can result in a deadlock, a condition where one lock is waiting for other to be freed in loop. This can also result in a reduced efficiency caused by repeated acquisition and release of locks.

• Another viable approach is to use a priority scheme. Penalizing the threads that use their full time slice and giving bonus to those that give up CPU voluntarily (for IO-bound tasks).

Why was GIL chosen?

The major reason was to provide thread safety when integrating c-extensions. C-extensions weren’t thread safe by default, but python gave a way to make these reliable and thread-safe. These c-extensions became one of the primary reasons for adoption of python by the dev community.

To learn more more about the technicalities of GIL, the reader can refer to this talk posted by David Breazley from Pycon’2010. : https://www.youtube.com/watch?v=Obt-vMVdM8s

Current Status of GIL

Many attempts were made to remove GIL, one of the famous ones namely Gilectomy.

As of python 3.13, an experimental free threaded mode was released. It uses advanced memory management techniques like mimalloc and biased reference counting. In spite of this, the GIL remains the current default.

Unorganized points

• The original python interpreter cpython these macro definitions are what implement the GIL.

  • Py_BEGIN_ALLOW_THREADS: Releasing the GIL by a thread.

  • Py_END_ALLOW_THREADS: To re-acquire the GIL by a thread.

As a developer, writing code isn’t enough, you need to develop skills to be able to put your work online for everyone to see. In the modern software engineering, deployment isn’t enough. The changes you make should reflect quickly as well. That’s where CI/CD comes in. Continuous Integration and Continuous deployment allows you to build, test and deploy your application and any changes without a need of manual intervention.

In this article I’ve compiled a comprehensive guide on setting up a CI/CD pipeline for a Django application, from repository setup to deployment using Docker in EC2. Most of it can be applied to other frameworks as well.

Prerequisites

• Basic knowledge of Git, Docker and Django.

• A GitHub account.

• A server (e.g., an EC2 instance or any machine accessible via SSH).

• CircleCI account (integrated with GitHub).

• Docker Hub and AWS credentials.

Background

What will we be doing in this tutorial?

The goal is to setup our project to run on docker. Test it locally, push the image to a repository(DockerHub in this case). Now, we can use this image anywhere to run our project with the same settings.

So, we will log into a server(Ec2) using ssh. Pull the docker image. And run it in the server itself. This entire process will be written in the config file. Make sure to go through it carefully. Everything else is just the setup to make sure the workflows defined in config.yml file work properly.

This tutorial is just going to help you make your hands dirty. You’ll need to understand more to deploy your own application. I will try to clarify on topics that require more exploration.

Step 1: Set Up a Github Repository

1. Create a Repository: Login to GitHub and create a new repository named django-circleci-demo

2. Clone the Repository:

git clone https://github.com/<your-username>/django-circleci-demo.git 
cd django-circleci-demo

3. Initialize a Django Project:

django-admin startproject myapp

python manage.py runserver

4. Commit and Push Code:

git add .

git commit -m "Initial commit"

git push origin main

Step 2: Configure the server with SSH

(When initially setting up a server, you’ll either download a .pem file and use it to access the server everytime, or keep it in .ssh config and create a username to login using this or other settings. With the below approach, you can generate a ssh key in your local machine, then login to your server with in browser shell or other ways, copy the public key, and then use this key to access that server from terminal in your local machine.)

1. Set Up SSH Access:

2. Generate an SSH key pair

• ssh-keygen -t rsa -b 4096 -C “your-email@example.com”

3. Copy the public key to your server in .ssh/authorized_keys.

4. Test SSH Access: ssh user@your-server-ip

5. Set Up the Server:

• Install Docker:

sudo apt update 
sudo apt install docker.io 
sudo systemctl start docker 
sudo systemctl enable docker

• Install AWS CLI:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 
unzip awscliv2.zip 
sudo ./aws/install

Step 3: CircleCI Configuration

1. Add CircleCI to the Repository: Go to CircleCI.

• Connect your GitHub repository to CircleCI.

2. Set Up Environment Variables:

• Navigate to Project Settings > Environment Variables in CircleCI.

• Add the following variables:

• DOCKER_USERNAME: Your Docker Hub username.

• DOCKER_PASSWORD: Your Docker Hub password.

• AWS_ACCESS_KEY_ID: Your AWS Access Key ID.

• AWS_SECRET_ACCESS_KEY : Your AWS Secret Access Key.

(Make sure the IAM user you’re using has enough permissions for accessing EC2 server, if your application has other components, then the user should have those permissions as well. Using ECR instead of DockerHub for pushing docker images is a common approach as well. If you’re planning to go through this approach, ECR permissions will also be required.)

• SERVER_HOST: Server IP.

• SERVER_USERNAME: Server username.

• SERVER_PASSWORD: Server password.

• BE_ENV: Environment variable specific to the project. You can base64 encode the variables when keeping it in circleci config, and decode it in the server itself to use it for enhanced security.

Step 4: Dockerize the Django App

1. Create a Dockerfile:

FROM python:3.10-slim 
WORKDIR /app 
COPY requirements.txt /app/ 
RUN pip install -r requirements.txt 
COPY . /app/ 
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

2. Add requirements.txt:

Django>=3.2,<4.0

3. Build and Test Docker image Locally:

docker build -t django-circleci-demo . 
docker run -p 8000:8000 django-circleci-demo

Step 5: Push Docker Image to Docker Hub

1. Log In to Docker Hub:

docker login -u <DOCKER_USERNAME> -p <DOCKER_PASSWORD>

2. Push Image:

docker tag django-circleci-demo <DOCKER_USERNAME>/django-circleci-demo 
docker push <DOCKER_USERNAME>/django-circleci-demo

Step 6: CircleCI Pipeline

1. Create .circleci/config.yml:

version: 2.1
jobs:
  build-and-deploy:
    docker:
      - image: circleci/python:3.10
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Install Docker
          command: |
            sudo apt-get update
            sudo apt-get install -y docker.io
      - run:
          name: Build Docker Image
          command: docker build -t django-circleci-demo .
      - run:
          name: Push to Docker Hub
          command: |
            echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
            docker tag django-circleci-demo $DOCKER_USERNAME/django-circleci-demo
            docker push $DOCKER_USERNAME/django-circleci-demo
      - run:
          name: Deploy to EC2
          command: |
            sshpass -p $SERVER_PASSWORD ssh -o StrictHostKeyChecking=no $SERVER_USERNAME@$SERVER_HOST "
            docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD &&
            docker pull $DOCKER_USERNAME/django-circleci-demo &&
            docker run -d -p 8000:8000 $DOCKER_USERNAME/django-circleci-demo"
workflows:
  version: 2
  build-and-deploy:
    jobs:
      - build-and-deploy

Step 6: Access the application

Navigate to http://<your_ec2_public_ip>:8000