An Intro to Docker Compose

What is Docker Compose?

Docker Compose is a tool that allows you to configure, build and run multiple containers. It's incredibly useful for building and deploying apps that use multiple services, each with their own container.

For example, let's say you have an app that uses 3 services:
- site: An ecommerce site that lets users browse and shop for items.
- database: A database that stores user, product, and order information.
- notifier: A script that monitors your database and sends email notifications to users when their orders have shipped.

Your app directory structure would look like this:

app
├── database
│   └── Dockerfile
│   └── ...
├── notifier
│   └── Dockerfile
│   └── ...
└── site
    └── Dockerfile
    └── ...

Normally, you'd have to start 3 containers yourself, either by running docker run for each image or by writing a custom script to launch them all. But with Docker Compose, you can launch all 3 containers with a single file AND configure them to boot.

How do I use Docker Compose?

Basic Overview

Let's expand on our example above. To use Docker Compose with our app, we add a docker-compose.yml file to the top level of our project directory:

app
├── docker-compose.yml
├── database
│   └── Dockerfile
│   └── ...
├── notifier
│   └── Dockerfile
│   └── ...
└── site
    └── Dockerfile
    └── ...

The contents of our docker docker-compose.yml file would be:

# line below tells docker-compose which version of the docker-compose file format to use
version: "3.8"
services:
    database:
        build: ./database
    notifier
        build: ./notifier
    site:
        build: ./site

The lines above define each service and tell Docker compose where to find their build files. To build the images for services, we'd run the command docker-compose build from the top level of the project directory. And to run them, we'd use the command docker-compose up.

Configuration Options

The example docker-compose.yml file above is very basic. But Docker compose allows you to configure your services in all sorts of ways. For example: we could add a few lines to tell docker-compose that the site and notifier services depend on the database service, and that the database service needs to build and run successfully before the others:

version: "3.8"
services:
    site:
        build: ./app
        depends_on:
            - database
    notifier
        build: ./notifier
        depends_on:
            - database
    db:
        build: ./database

Note that each of our services still has their own Dockerfile. This is useful for long, service specific configurations, when a service might have a long and/or complex Dockerfile. However, you could include all of the configuration options for a service in your docker-compose.yml file if you wanted.

For example - let's assume the database service has 2 files: database/Dockerfile and database/init.sql.

app
├── docker-compose.yml
├── database
│   └── Dockerfile
│   └── init.sql
├── notifier
│   └── Dockerfile
│   └── ...
└── site
    └── Dockerfile
    └── ...

Let's also assume the contents of database/Dockerfile are as follows:

FROM postgres:13
COPY ./init.sql /docker-entrypoint-initdb.d/init.sql

This simply tells Docker which image to use for database, and copies a single file (init.sql) to the container on build. Note that /docker-entrypoint-initdb.d is a special directory on postgres images, and that .sql files in this directory are run when postgres starts.

We could remove our database/Dockerfile entirely by changing our docker-compose.yml as below:

version: "3.8"
services:
    site:
        build: ./app
        depends_on:
            - database
    notifier
        build: ./notifier
        depends_on:
            - database
    database:
        image: "postgres:13"
        volumes:
            "./init.sql:/docker-entrypoint-initdb.d/init.sql"

This achieves the same effect - Docker Compose now knows which image to use for database, and will copy our init.sql file to the appropriate place in the container (volumes is slightly different than COPY, but I won't get into that here).

Closing Remarks

Hope that's enough to get you started using Docker compose on your own! If you want to learn more,  I highly recommend reading the Docker compose docs.

An Intro to Flask

In my last post, I presented a high level overview of what a web application is. In this post, we'll take a look at how how you can use Flask (a python based app framework) to build applications by breaking down a series of examples, culminating with the app from this official flask tutorial. I won't dive deeply into the code for that example here, as the tutorial above already does that. Instead, I will build up to it with two of my own examples, and provide a visual representation of it.

NOTE: I highly recommend the tutorial if you want a deeper dive into Flask.

What is Flask?

Flask is a Python based micro-framework for building web applications. It's "micro" simply because it's minimalist, meaning it has fewer built-in features than other frameworks.

How is an app constructed using Flask?

Consider a basic app - one that has a single view which simply shows the phrase, "Hello, World!" Using Flask, we can build this app with a single file. This file would be stored in a directory (app) with a single file (app.py):

app
|- app.py

The contents of app.py:

from flask import Flask

app = Flask(__name__)


@app.route('/')
def hello():
    return 'Hello, World!'

The code above defines an application using the Flask object (a class that represents an application). This appication serves a single view, which you can request via the route /. When you request this view, the code attached to this route is executed, and Flask responds with a simple line of text - "Hello, World!"

To run this app on a Linux or Mac machine, you would run the commands:

export FLASK_APP=app
export FLASK_ENV=development
flask run

Here's what would happen:

  1. Flask spins up a server on your local computer. This server listens for requests for the domain http://127.0.0.1:5000.
  2. The view for the route / would be returned when you visit the base domain above.

What would a more complicated example look like?

The app above is fairly straightforward. Let's try to visualize what a more complicated example would look like by expanding our app to include two more views: one that returns Foo. and one that returns Bar. We'll add these new views via a Blueprint - an object that enables you to group views. It also allows you to better organize your code by enabling you to break your views out into multiple files.

Here's a visual representation of our app:

App Overview

Note our app is a Flask object with 3 views, each defined under a route. Two of our views are grouped via a Blueprint.

We'll need to organize our files a little differently by changing app.py to __init__.py, and by nesting the app directory in a new directory flask_example:

flask_example
|- app
    |- __init__.py
    |- foobar.py

Our code for running the app will not change. However, we need to execute the commands from the flask_example directory. Why? We do this for namespace reasons - in order for our app file (now __init__.py) to import our blueprint (defined in foobar.py), we need to treat the app directory like a Python module.

Our code for __init__.py is now:

from flask import Flask
from . import foobar

app = Flask(__name__)


@app.route('/')
def hello():
    return 'Hello, World!'

app.register_blueprint(foobar.bp)

And our code for foobar.py is:

from flask import Blueprint

bp = Blueprint('foobar', __name__, url_prefix='/foobar')


@bp.route('/foo')
def foo():
    return 'Foo.'


@bp.route('/bar')
def bar():
    return 'Bar.'

What would a slightly more complicated example look like?

The example above is great, but what if you wanted your views to use HTML instead of simple text phrases? Great news! You easily do so by using templates - Jinja files you can use to construct more complex views.

Let's add a template for each of our views. Please note that I won't get into the specifics of Jinja syntax here, nor will I define the template files. I'll leave that part to you as a creative exercise ;) But if you'd like to learn more, I recommend reading the Jinja docs.

Our app will now look like this:

Flask Example 2

We'll need to add a new directory for our templates:

flask_example
|- app
    |- __init__.py
    |- foobar.py
    |- templates
        |- index.html
        |- foobar
            |- foo.html
            |- bar.html

And by slightly modifying our code for __init__.py:

from flask import Flask
from . import foobar

app = Flask(__name__)


@app.route('/')
def hello():
    render_template('index.html`)

app.register_blueprint(foobar.bp)

And foobar.py:

from flask import Blueprint

bp = Blueprint('foobar', __name__, url_prefix='/foobar')


@bp.route('/foo')
def foo():
    render_template('foobar/foo.html`)


@bp.route('/bar')
def bar():
    render_template('foobar/bar.html`)

What would an even MORE complicated example look like?

I'll cover one final example: flaskr, a simple blogging app you can build using an official Flask tutorial. You should be ready to tackle it on your own by now, but I want to give you a visual representation of the app before you start.

The app has 5 views, split between 2 blueprints:

  • blog
    • /: The site index. If a user is logged in, shows their posts. If not, shows login options.
    • /create: Allows a logged in user to create and save a new blog post.
    • /update: Allows a logged in user to edit an existing blog post.
  • auth
    • /register: Allows a user to create an account with a username and password.
    • /login: Allows a user to login to an existing account.

Note that this app also has an additional component: a mySQL database with two tables: user (which stores user data) and post (which stores post data).

With that, here's a visual overview:

flaskr Visualization

Closing Remarks

Hope you found this helpful! I know that writing this and creating the visualizations above really helped cement the Flask fundamentals for me. The next posts in this series will cover some additional backend basics, in no particular order: logging, testing and algorithm design.

An Intro to Web Apps

A quick note: this post is designed for true beginners - those who have little to no knowledge of software and web development. That disclaimer aside - let's dive in!

What is a Web Application?

A Web Application is a software application that is accessed and used via the internet, typically via a web browser. However, before we dive into what a Web Application is, we must first answer two prerequisite questions:

How does the Internet Work?

The internet is simply a network of computers that can communicate with each other. Computers communicate over the internet via the "Client-Server Model", in which a one computer (the "client") asks another computer (the "server") for some data via a request. The server then replies to this request with a response.

The specifics behind how internet communications work and are structured are vast and complex, and I won't dive into them here. For now, the model above is enough for our purposes.

What is a Software Application?

A software application, or app, is a program that enable users to view and manipulate data. At a high level, you can use the "Model/View/Controller (MVC)" framework to visualize the basic structure of most apps:


Model View Controller

In this framework, you can think of an app as consisting of 3 components, controlled by a user. The components are:

  • Model: A model that represents and stores data.
  • View: A view of the data model, which is shown to the user.
  • Controller: A component that the user uses to manipulate the data model.

Let's bring all of these concepts together and via an example you're likely familiar with: FaceBook. The data that represents your profile is an example of a model. It might be represented in a table like this:

Name Birthday Hometown ...
John Doe     3/14/1998     Norfolk, VA     ...    

A view of this model would be your profile page. And a controller that manipulates this model would be one of FaceBook's servers.

How does a Web Application Work?

While the MVC framework is useful for visualizing the purpose and function of software applications, it isn't quite accurate for web applications. Most basic web applications actually consist of these 3 components: a frontend, a backend, and a database.

  • Frontend: Renders and displays views.
  • Backend: Controls data and builds views.
  • Database: Stores models.

You, as a user, interact with the app via a view, which is shown to you via a web browser on some client (laptop, phone, etc.). Whenever you want a new view, the following happens:

  1. Your browser (aka the frontend) makes a request for a view.
  2. The request is sent to a server running the app.
  3. The server forwards this request to the backend component.
  4. The backend uses the database to manipulate and retrieve data from the model.
  5. The backend uses this data to construct the view.
  6. The server sends the view to the client as the response.
  7. Your browser renders the view and displays it.

FBDB

Please note - this is only one way to conceptualize the architecture of a basic web app. In reality, most apps rely on multiple backend components, and the frontend for some apps performs some controller functions.

An Example: FaceBook

Let's return to our FaceBook example. FaceBook is a web app that stores, and allows you to view, data from you and your friends.

It uses a PHP and C++ based backend with a MySQL database. NOTE: This isn't strictly true. FaceBook is hugely complex with dozens of components. This is my best guess at their core components based on what I could find on the interwebs.

Your profile page is an example of a view: it is a representation of the data model that represents your profile. When you open up FaceBook and go to your profile page, this is what happens:

  1. Your browser makes sends a request to facebook.com.
  2. This request is forwarded to a server running a PHP backend.
  3. The backend accesses the model representing your profile via the MySql database, and retrieves your profile information.
  4. The backend uses this information to construct a view of your profile.
  5. The server sends this view back to your browser via a response.
  6. Your browser renders the view as your profile page.

Closing Remarks

That's it! Hope it was helpful. While Web Applications are much more complicated than the simple overview I've provided here, I hope that this is at least enough to get you started on whatever projects or learning journey you have planned.

Next, we'll be diving into how you can use Flask to construct and manage basic web apps.

An Intro to Ignorance

Have you ever heard of the Dunning-Kruger effect? It's a well-known phenomenon in Psychology in which individuals with limited experience or knowledge in a domain overestimate their level of expertise. It is often visualized as a curve:

Dunning-Kruger Curve

In other words: as you learn more about a domain, the more you realize how little you actually know about it.

I write about this effect because I've realized that I am a victim of it. A friend recently approached me and asked for help building a web application. As I've helped build and deploy production level apps, I confidently agreed and set about writing a tech spec and architecting a solution. However, my engineering knowledge is largely self-taught and piecemeal, and the more I've worked on his app the more I've realized the gaps in my understanding.

Thus, I find myself on the left end of the curve above. Which, as a naturally ambitious learner, I find to be unacceptable. To rectify this, I've decided to revisit the basics by writing a series of introductory posts intended for beginners.

Over the next few weeks, we'll start with an overview of what a web application is, how you can use Flask and react.js to build one, and how you can use AWS to deploy one at scale.

Excited to get started!

An Introduction to Docker

I've been learning about Docker for use in my personal and professional projects. Though I've used it before, I've always had a vague understanding of what's actually going on behind the scenes. And so I decided to go back to the basics and build a better high-level understanding before diving into the weeds. Let's dive in:

What is Docker?

Docker is a tool for creating and running containers.

What is a container?

A container is a unit of software that packages up code and all its dependencies (code, system tools, system libraries and settings). When a container is running, you can consider it to be an isolated environment (separate from your local machine, or host) in which your code can run.

A container is similar to a virtual machine, but instead of requiring a separate operating system and a hypervisor program to divy up the host's resources between the host and the virtual machine, a container uses the host's operating system kernel and shares resources with it.

I won't go into the details here. But there are a multitude of blog posts on this topic, such as this one.

The bottom line: containers are isolated environments in which you can package code and run it. And they require fewer resources than virtual machines do.

How does Docker work?

So, how do you use docker to create a container? And what happens when you do?

Docker containers are built using images. An image is a blueprint for a container - it's a set of dependencies and configuration options that are installed on your image when it's created.

Note that Docker images live on Dockerhub - a repository for both user-generated and officially maintained Docker images.

When you create and run a container using Docker, you need to tell Docker several things:

  1. What image to use for your container.
  2. Where the files for your container will live on your local machine/host.
  3. Any additional files (aside from the image files) to copy to your container.
  4. Any commands you want run on the container after it starts.

You can specify these items via the command line, or you can specify them in a special file a Dockerfile.

Here's an example Dockerfile:

# Use the official image as a parent image.
FROM node:current-slim

# Set the working directory.
WORKDIR /usr/src/app

# Copy the file from your host to your current location.
COPY package.json .

# Run the command inside your image filesystem.
RUN npm install

# Inform Docker that the container is listening on the specified port at runtime.
EXPOSE 8080

# Run the specified command within the container.
CMD [ "npm", "start" ]

# Copy the rest of your app's source code from your host to your image filesystem.
COPY . .

Visual Example

Let's say you have an app you want to run in a container. Let's also assume you have a project directory with some source code for the app and a Dockerfile with the instructions on how to setup your container. When you use the command docker run , Docker will do the following:

  1. Fetch the image files from Dockerhub.
  2. Create a working directory for your container - space on your host that contains the root filesystem for your container.
  3. Install the dependencies for your app and start your container.
  4. Copy your app's source code to the container's working directory.

Here's a visual example:

Conclusion

That's all for now! Hope this helped you get an idea of how Docker works. The best resource to read for this is likely the official Docker orientation, but I wanted to write my own verison to get the ideas down in my mind.