🚀 Designing a Secure & Scalable Code Execution System (Like Online Compilers)

So, for the past week, I’ve been designing and building my own code execution system from scratch — similar to what online compilers do — for my application.

😅 The Wrong Way I Started With

My initial prototype had a simple idea:

  • Clients send code to a gateway endpoint.
  • The gateway calls the code execution service.
  • The code execution service spins up a new process (e.g., Python) and runs it.
  • The gateway synchronously waits for the response and returns it to the client.

❌ Sounds simple, but it was a disaster waiting to happen...

🚨 1. Scalability Nightmare

Let’s say 10,000 Python requests come in.

  • That’s 10,000 new processes created.
  • Each process might compute something heavy (e.g., big factorials).
  • CPU and RAM go 📉.
  • And worse: 10,000 clients are left waiting — even the homepage won’t load.

🔐 2. Security Disaster

When you execute user-submitted code directly on your host machine:

  • You're practically inviting hackers.
  • They can run anything — install backdoors, shut down your server, leak environment variables, etc.
  • No isolation between your service and their malicious code.

🛠️ Time to Re-Architect

I took a step back and decided to redesign the entire system. The core idea was to:

✅ Make it asynchronous

✅ Make it secure and isolated

✅ Make it scalable

⚙️ System Architecture Overview

Instead of waiting for the code to execute:

  1. The gateway receives the code and publishes a job to a RabbitMQ queue.
  2. It returns a unique Job ID to the client immediately.
  3. The code execution service listens to the queue and picks up jobs.
  4. Code is run in Docker containers (isolated environments).
  5. Output is sent back through another queue.
  6. The client connects to an SSE (Server-Sent Events) endpoint to receive output in real-time.

🧠 Container Management & Scheduling

I didn’t want to bring in Kubernetes or Docker Swarm — too heavy for my use case. So I built:

  • A custom scheduler (event-driven)
  • A pool of pre-warmed containers (5 per language)
  • A thread-safe internal queue (per language)
  • A linked-list implementation to manage the job queues

🧪 When a job comes in:

  • It’s placed into its corresponding language queue.
  • My scheduler detects available containers and assigns the job.
  • When the job completes, the output is sent to RabbitMQ and then streamed to the client via SSE.

🔐 Defense Against Attacks

Because code execution endpoints are prime targets, I added:

✅ Rate Limiting

  • Implemented a token bucket algorithm
  • Limits how many requests a user can make per second

✅ Execution Timeout

  • Every container has a 5-second timeout
  • Prevents infinite loops and long-hanging processes

✅ Isolated Environments

  • Every job runs inside a Docker container
  • Malicious code can’t touch the host machine

⚡ Advantages of This Architecture

  • 🛡️ Secure execution using isolated Docker containers
  • 🔄 Asynchronous and non-blocking — no user is left waiting
  • 🚀 Scalable — containers are reused and jobs are scheduled
  • 🔒 Resilient to attacks like infinite loops, backdoors, and DOS

🎥 I Made a YouTube Video Too

I shared the system design and architecture (not the source code).

It walks through the architecture visually so others can learn from it.

🧠 Final Thoughts

This system took me about a week to plan and build. And I took my time because…

“Unlike typical APIs where attackers have to find vulnerabilities, a code execution API is an open playground for hackers. You're basically letting them run anything.”

So I had to make sure the design was secure, scalable, and robust.

🙌 Let Me Know Your Thoughts!

If you're interested in the code behind this or want me to deep dive into:

  • 🧠 My custom scheduler algorithm
  • 🧱 Linked list queue implementation

Let me know in the comments! 💬

💻 Built with:

  • Go (Golang)
  • RabbitMQ
  • Docker
  • Custom scheduler (Golang)
  • SSE