Composing Projects Using Docker-Compose

📙
The Docker Handbook
Search…
📙
The Docker Handbook
Table of Contents
Introduction to Containerization and Docker
Installing Docker
Hello World in Docker
Container Manipulation Basics
Image Manipulation Basics
Containerizing a JavaScript Application
Network Manipulation Basics
Containerizing a Multi-Container JavaScript Application
Composing Projects Using Docker-Compose
Conclusion
Composing Projects Using Docker-Compose
In the previous section, you've learned about managing a multi-container project and the difficulties of it. Instead of writing so many commands, there is an easier way to manage multi-container projects, a tool called Docker Compose.
According to the Docker documentation - "Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration."
Although Compose works in all environments, it's more focused on development and testing. Using Compose on a production environment is not recommended at all.
Compose Basics
Go the directory where you've cloned the repository that came with this article. Go inside the notes-api/api directory and create a Dockerfile.dev file. Put the following code in it:
1
# stage one
2
FROM node:lts-alpine as builder
3
​
4
# install dependencies for node-gyp
5
RUN apk add --no-cache python make g++
6
​
7
WORKDIR /app
8
​
9
COPY ./package.json .
10
RUN npm install
11
​
12
# stage two
13
FROM node:lts-alpine
14
​
15
ENV NODE_ENV=development
16
​
17
USER node
18
RUN mkdir -p /home/node/app
19
WORKDIR /home/node/app
20
​
21
COPY . .
22
COPY --from=builder /app/node_modules /home/node/app/node_modules
23
​
24
CMD [ "./node_modules/.bin/nodemon", "--config", "nodemon.json", "bin/www" ]
Copied!
The code is almost identical to the Dockerfile that you worked with in the previous section. The three differences in this file are as follows:
On line 10, we run npm install instead of npm run install --only=prod because we want the development dependencies also.
On line 15, we set the NODE_ENV environment variable to development instead of production.
On line 24, we use a tool called nodemon to get hot-reload feature for the API.
You already know that this project has two containers:
notes-db - A database server powered by PostgreSQL.
notes-api - A REST API powered by Express.js
In the world of compose, each container that makes up the application is known as a service and the first step to composing a multi-container project is to define these services.
Just like the Docker daemon uses a Dockerfile for building images, Docker Compose uses docker-compose.yaml file to read service definitions from.
Head to the notes-api directory and create a new docker-compose.yaml file. Put following code into the newly created file:
1
version: "3.8"
2
​
3
services: 
4
    db:
5
        image: postgres:12
6
        container_name: notes-db-dev
7
        volumes: 
8
            - db-data:/var/lib/postgresql/data
9
        environment:
10
            POSTGRES_DB: notesdb
11
            POSTGRES_PASSWORD: secret
12
    api:
13
        build:
14
            context: ./api
15
            dockerfile: Dockerfile.dev
16
        image: notes-api:dev
17
        container_name: notes-api-dev
18
        environment: 
19
            DB_HOST: db ## same as the database service name
20
            DB_DATABASE: notesdb
21
            DB_PASSWORD: secret
22
        volumes: 
23
            - /home/node/app/node_modules
24
            - ./api:/home/node/app
25
        ports: 
26
            - 3000:3000
27
​
28
volumes:
29
    db-data:
30
        name: notes-db-dev-data
Copied!
Every valid docker-compose.yaml file starts by defining the file version. At the time of writing, 3.8 is the latest version. You can look up the latest version here.
Blocks in an YAML file are defined by indentation. I will go through each of the blocks and will explain what they do.
The services block holds the definitions for each of the services or containers in the application. db and api are the two services that comprise this project.
The db block defines a new service in the application and holds necessary information to start the container. Every service requires either a pre-built image or a Dockerfile to run a container. For the db service we're using the official PostgreSQL image.
Unlike the db service, a pre-built image for the api service doesn't exist. Hence, we use the Dockerfile.dev file.
The volumes block defines any name volume needed by any of the services. At the time it only enlists db-data volume used by the db service.
Now that we have a high level overview of the docker-compose.yaml file, lets have a closer look at the individual services.
Definition code for the db service is as follows:
1
db:
2
    image: postgres:12
3
    container_name: notes-db-dev
4
    volumes: 
5
        - db-data:/var/lib/postgresql/data
6
    environment:
7
        POSTGRES_DB: notesdb
8
        POSTGRES_PASSWORD: secret
Copied!
The image key holds the image repository and tag used for this container. We're using the postgres:12 image for running the database container.
The container_name indicates the name of the container. By default containers are named following <project directory name>_<service name> syntax. You can override that using container_name.
The volumes array holds the volume mappings for the service and supports named volumes, anonymous volumes, bind mounts. The syntax <source>:<destination> is identical to what you've seen before. 
The environment map holds the values of the various environment variables needed for the service.
Definition code for the api service is as follows:
1
api:
2
    build:
3
        context: ./api
4
        dockerfile: Dockerfile.dev
5
    image: notes-api:dev
6
    container_name: notes-api-dev
7
    environment: 
8
        DB_HOST: db ## same as the database service name
9
        DB_DATABASE: notesdb
10
        DB_PASSWORD: secret
11
    volumes: 
12
        - /home/node/app/node_modules
13
        - ./api:/home/node/app
14
    ports: 
15
        - 3000:3000
Copied!
The api service doesn't come with a pre-built image instead what it has is a build configuration. Under the build block we define the context and the name of the Dockerfile for building an image. You should have an understanding of context and Dockerfile by now so I won't spend time explaining those.
The image key holds the name of the image to be built. If not assigned the image will be named following the <project directory name>_<service name> syntax.
Inside the environment map, the DB_HOST variable demonstrates a feature of Compose. That is, you can refer to another service in the same application by using its name. So the db here, will be replaced by the IP address of the api service container. The DB_DATABASE and DB_PASSWORD variables have to match up with POSTGRES_DB and POSTGRES_PASSWORD respectively from the db service definition.
In the volumes map, you can see an anonymous volume and a bind mount described. The syntax is identical to what you've seen in previous sections.
The ports map defines any port mapping. The syntax, <host port>:<container port> is identical to the --publish option you used before.
Finally, code for the volumes is as follows:
1
volumes:
2
    db-data:
3
        name: notes-db-dev-data
Copied!
Any named volume used in any of the services has to be defined here. If you don't define a name, the volume will be named following the <project directory name>_<volume key> syntax and the key here is db-data. You can learn about the different options for volume configuration in the official docs.
What About the Network Bridge?
You may have noticed that there is no network bridge creation section in this YAML file. While the docker-compose specification supports adding network configuration to the file, docker-compose has a helpful feature that automatically creates a bridge network for the composed project, and assigns a name based on the directory name (or project-name, if defined with the --project-name switch or the COMPOSE_PROJECT_NAME environment variable). So, since this file doesn't include that information, you can look for a "notes-api_default" network after bringing up the composed project below.
Starting Services
There are a few ways of starting services defined in a YAML file. The first command that you'll learn about is the up command. The up command builds any missing images, creates containers and starts them in one go.
Before you execute the command though, make sure you've opened your terminal in the same directory where the docker-compose.yaml file is. This is very important for every docker-compose command you execute.
1
docker-compose --file docker-compose.yaml up --detach
2
​
3
# Creating network "notes-api_default" with the default driver
4
# Creating volume "notes-db-dev-data" with default driver
5
# Building api
6
# Sending build context to Docker daemon  37.38kB
7
#
8
# Step 1/13 : FROM node:lts-alpine as builder
9
#  ---> 471e8b4eb0b2
10
# Step 2/13 : RUN apk add --no-cache python make g++
11
#  ---> Running in 197056ec1964
12
### LONG INSTALLATION STUFF GOES HERE ###
13
# Removing intermediate container 197056ec1964
14
#  ---> 6609935fe50b
15
# Step 3/13 : WORKDIR /app
16
#  ---> Running in 17010f65c5e7
17
# Removing intermediate container 17010f65c5e7
18
#  ---> b10d12e676ad
19
# Step 4/13 : COPY ./package.json .
20
#  ---> 600d31d9362e
21
# Step 5/13 : RUN npm install
22
#  ---> Running in a14afc8c0743
23
### LONG INSTALLATION STUFF GOES HERE ###
24
#  Removing intermediate container a14afc8c0743
25
#  ---> 952d5d86e361
26
# Step 6/13 : FROM node:lts-alpine
27
#  ---> 471e8b4eb0b2
28
# Step 7/13 : ENV NODE_ENV=development
29
#  ---> Running in 0d5376a9e78a
30
# Removing intermediate container 0d5376a9e78a
31
#  ---> 910c081ce5f5
32
# Step 8/13 : USER node
33
#  ---> Running in cfaefceb1eff
34
# Removing intermediate container cfaefceb1eff
35
#  ---> 1480176a1058
36
# Step 9/13 : RUN mkdir -p /home/node/app
37
#  ---> Running in 3ae30e6fb8b8
38
# Removing intermediate container 3ae30e6fb8b8
39
#  ---> c391cee4b92c
40
# Step 10/13 : WORKDIR /home/node/app
41
#  ---> Running in 6aa27f6b50c1
42
# Removing intermediate container 6aa27f6b50c1
43
#  ---> 761a7435dbca
44
# Step 11/13 : COPY . .
45
#  ---> b5d5c5bdf3a6
46
# Step 12/13 : COPY --from=builder /app/node_modules /home/node/app/node_modules
47
#  ---> 9e1a19960420
48
# Step 13/13 : CMD [ "./node_modules/.bin/nodemon", "--config", "nodemon.json", "bin/www" ]
49
#  ---> Running in 5bdd62236994
50
# Removing intermediate container 5bdd62236994
51
#  ---> 548e178f1386
52
# Successfully built 548e178f1386
53
# Successfully tagged notes-api:dev
54
# Creating notes-api-dev ... done
55
# Creating notes-db-dev  ... done
Copied!
The --detach or -d option here functions same as the one you've seen before. The --file or -f option is only needed if the YAML file is not named docker-compose.yaml but I've used here for demonstration purpose.
Apart from the the up command there is the start command. The main difference between these two is the start command doesn't create missing containers, only starts existing containers. It's basically same as the container start command.
The --build option for the up command forces a rebuild of the images. There are some other options for the up command that you can see on official docs.
Listing Services
Although service containers started by Compose can be listed using the container ls command, there is the ps command for listing containers defined in the YAML only.
1
docker-compose ps
2
​
3
#     Name                   Command               State           Ports         
4
# -------------------------------------------------------------------------------
5
# notes-api-dev   docker-entrypoint.sh ./nod ...   Up      0.0.0.0:3000->3000/tcp
6
# notes-db-dev    docker-entrypoint.sh postgres    Up      5432/tcp
Copied!
It's not as informative as the container ls output, but useful when you have tons of containers running simultaneously.
Executing Commands Inside a Running Service
I hope you remember from the previous section that you have to run some migration scripts to create the database tables for this API. Just like the container exec command, there is an exec command for docker-compose. Generic syntax for the command is as follows:
1
docker-compose exec <service name> <command>
Copied!
To execute the npm run db:migrate command inside the api service, you can execute the following command:
1
docker-compose exec api npm run db:migrate
2
​
3
# > [email protected] db:migrate /home/node/app
4
# > knex migrate:latest
5
# 
6
# Using environment: development
7
# Batch 1 run: 1 migrations
Copied!
Unlike the container exec command, you don't need to pass the -it flag for interactive sessions. docker-compose does that automatically.
Accessing Logs From a Running Service
You can also use the logs command to retrieve logs from a running service. The generic syntax for the command is as follows:
1
docker-compose logs <service name>
Copied!
To access the logs from the api service execute the following command:
1
docker-compose logs api
2
​
3
# Attaching to notes-api-dev
4
# notes-api-dev | [nodemon] 2.0.7
5
# notes-api-dev | [nodemon] reading config ./nodemon.json
6
# notes-api-dev | [nodemon] to restart at any time, enter `rs`
7
# notes-api-dev | [nodemon] or send SIGHUP to 1 to restart
8
# notes-api-dev | [nodemon] ignoring: *.test.js
9
# notes-api-dev | [nodemon] watching path(s): *.*
10
# notes-api-dev | [nodemon] watching extensions: js,mjs,json
11
# notes-api-dev | [nodemon] starting `node bin/www`
12
# notes-api-dev | [nodemon] forking
13
# notes-api-dev | [nodemon] child pid: 19
14
# notes-api-dev | [nodemon] watching 18 files
15
# notes-api-dev | app running -> http://127.0.0.1:3000
Copied!
This is just a portion from the log output. You can kind of hook into the output stream of the service and get the logs in real-time by using the -f or --follow option. Any later log will show up instantly in the terminal as long as you don't exit by pressing ctrl + c key combination or closing the window. The container will keep running even if you exit out of the log window.
Stopping Services
For stopping services, there are two approaches that you can take. The first one is the down command. The down command stops all running containers and removes them from the system. It also removes any networks:
1
docker-compose down --volumes
2
​
3
# Stopping notes-api-dev ... done
4
# Stopping notes-db-dev  ... done
5
# Removing notes-api-dev ... done
6
# Removing notes-db-dev  ... done
7
# Removing network notes-api_default
8
# Removing volume notes-db-dev-data
Copied!
The --volumes option indicates that you want to remove any named volume defined in the volumes block. You can learn about the additional options for the down command in the official docs.
Another command for stopping services is the stop command which functions identically to the container stop command. It stops all the containers for the application and keeps the contianers. These containers can later be started with the start or up command.
Composing a Full-stack Application
In this sub-section, we'll be adding a front-end to our notes API and turn it into a complete fullstack application. I won't be explaining any of the Dockerfile.dev files in this sub-section (except the one for the nginx service) as they are identical to some of the others you've already seen in previous sub-sections.
If you've cloned the project code repository, then go inside the fullstack-notes-application directory. Each directory inside the project root contains the code for each services and the corresponding Dockerfile.
Before we start with the docker-compose.yaml file let's look at a diagram of how the application is going to work:
Instead of accepting requests directly like we previously did, in this application, all the requests will be first received by a NGINX (lets call it router) service. The router will then see if the requested end-point has /api in it. If yes, the router will route the request to the back-end or if not, the router will route the request to the front-end.
The reason behind doing this is that when you run a front-end application it doesn't run inside a container. It runs on the browser, served from a container. As a result, Compose networking doesn't work as expected and the front-end application fails to find the api service.
NGINX on the other hand runs inside a container and can communicate with the different services across the entire application.
I will not get into the configuration of NGINX here. That topic is kinda out of scope of this article. But if you want to have a look at it, go ahead and checkout the /notes-api/nginx/development.conf and /notes-api/nginx/production.conf files. Code for the /notes-api/nginx/Dockerfile.dev is as follows:
1
FROM nginx:stable-alpine
2
​
3
COPY ./development.conf /etc/nginx/conf.d/default.conf
Copied!
All it does is copy the configuration file to /etc/nginx/conf.d/default.conf inside the container.
Let's start writing the docker-compose.yaml file. Apart from the api and db services there will be the client and nginx services. There will also be some network definitions that I'll get into shortly.
1
version: "3.8"
2
​
3
services: 
4
    db:
5
        image: postgres:12
6
        container_name: notes-db-dev
7
        volumes: 
8
            - db-data:/var/lib/postgresql/data
9
        environment:
10
            POSTGRES_DB: notesdb
11
            POSTGRES_PASSWORD: secret
12
        networks:
13
            - backend
14
    api:
15
        build: 
16
            context: ./api
17
            dockerfile: Dockerfile.dev
18
        image: notes-api:dev
19
        container_name: notes-api-dev
20
        volumes: 
21
            - /home/node/app/node_modules
22
            - ./api:/home/node/app
23
        environment: 
24
            DB_HOST: db ## same as the database service name
25
            DB_PORT: 5432
26
            DB_USER: postgres
27
            DB_DATABASE: notesdb
28
            DB_PASSWORD: secret
29
        networks:
30
            - backend
31
    client:
32
        build:
33
            context: ./client
34
            dockerfile: Dockerfile.dev
35
        image: notes-client:dev
36
        container_name: notes-client-dev
37
        volumes: 
38
            - /home/node/app/node_modules
39
            - ./client:/home/node/app
40
        networks:
41
            - frontend
42
    nginx:
43
        build:
44
            context: ./nginx
45
            dockerfile: Dockerfile.dev
46
        image: notes-router:dev
47
        container_name: notes-router-dev
48
        restart: unless-stopped
49
        ports: 
50
            - 8080:80
51
        networks:
52
            - backend
53
            - frontend
54
​
55
volumes:
56
    db-data:
57
        name: notes-db-dev-data
58
​
59
networks: 
60
    frontend:
61
        name: fullstack-notes-application-network-frontend
62
        driver: bridge
63
    backend:
64
        name: fullstack-notes-application-network-backend
65
        driver: bridge
Copied!
The file is almost identical to the previous one you worked with. Only thing that needs some explanation is the network configuration. Code for the networks block is as follows:
1
networks: 
2
    frontend:
3
        name: fullstack-notes-application-network-frontend
4
        driver: bridge
5
    backend:
6
        name: fullstack-notes-application-network-backend
7
        driver: bridge
Copied!
I've defined two bridge networks. By default Compose creates a bridge network and attaches all containers to that. In this project however, I wanted proper network isolation. So I defined two networks, one for the front-end services and one for the back-end services.
I've also added networks block in each of the service definitions. This way the the api and db service will be attached to one network and the client service will be attached to a separate network. The nginx service however will be attached to both the networks so that it can perform as router between the front-end and back-end services.
Start all the services by executing following command:
1
docker-compose --file docker-compose.yaml up --detach
2
​
3
# Creating network "fullstack-notes-application-network-backend" with driver "bridge"
4
# Creating network "fullstack-notes-application-network-frontend" with driver "bridge"
5
# Creating volume "notes-db-dev-data" with default driver
6
# Building api
7
# Sending build context to Docker daemon  37.38kB
8
# 
9
# Step 1/13 : FROM node:lts-alpine as builder
10
#  ---> 471e8b4eb0b2
11
# Step 2/13 : RUN apk add --no-cache python make g++
12
#  ---> Running in 8a4485388fd3
13
### LONG INSTALLATION STUFF GOES HERE ###
14
# Removing intermediate container 8a4485388fd3
15
#  ---> 47fb1ab07cc0
16
# Step 3/13 : WORKDIR /app
17
#  ---> Running in bc76cc41f1da
18
# Removing intermediate container bc76cc41f1da
19
#  ---> 8c03fdb920f9
20
# Step 4/13 : COPY ./package.json .
21
#  ---> a1d5715db999
22
# Step 5/13 : RUN npm install
23
#  ---> Running in fabd33cc0986
24
### LONG INSTALLATION STUFF GOES HERE ###
25
# Removing intermediate container fabd33cc0986
26
#  ---> e09913debbd1
27
# Step 6/13 : FROM node:lts-alpine
28
#  ---> 471e8b4eb0b2
29
# Step 7/13 : ENV NODE_ENV=development
30
#  ---> Using cache
31
#  ---> b7c12361b3e5
32
# Step 8/13 : USER node
33
#  ---> Using cache
34
#  ---> f5ac66ca07a4
35
# Step 9/13 : RUN mkdir -p /home/node/app
36
#  ---> Using cache
37
#  ---> 60094b9a6183
38
# Step 10/13 : WORKDIR /home/node/app
39
#  ---> Using cache
40
#  ---> 316a252e6e3e
41
# Step 11/13 : COPY . .
42
#  ---> Using cache
43
#  ---> 3a083622b753
44
# Step 12/13 : COPY --from=builder /app/node_modules /home/node/app/node_modules
45
#  ---> Using cache
46
#  ---> 707979b3371c
47
# Step 13/13 : CMD [ "./node_modules/.bin/nodemon", "--config", "nodemon.json", "bin/www" ]
48
#  ---> Using cache
49
#  ---> f2da08a5f59b
50
# Successfully built f2da08a5f59b
51
# Successfully tagged notes-api:dev
52
# Building client
53
# Sending build context to Docker daemon  43.01kB
54
# 
55
# Step 1/7 : FROM node:lts-alpine
56
#  ---> 471e8b4eb0b2
57
# Step 2/7 : USER node
58
#  ---> Using cache
59
#  ---> 4be5fb31f862
60
# Step 3/7 : RUN mkdir -p /home/node/app
61
#  ---> Using cache
62
#  ---> 1fefc7412723
63
# Step 4/7 : WORKDIR /home/node/app
64
#  ---> Using cache
65
#  ---> d1470d878aa7
66
# Step 5/7 : COPY ./package.json .
67
#  ---> Using cache
68
#  ---> bbcc49475077
69
# Step 6/7 : RUN npm install
70
#  ---> Using cache
71
#  ---> 860a4a2af447
72
# Step 7/7 : CMD [ "npm", "run", "serve" ]
73
#  ---> Using cache
74
#  ---> 11db51d5bee7
75
# Successfully built 11db51d5bee7
76
# Successfully tagged notes-client:dev
77
# Building nginx
78
# Sending build context to Docker daemon   5.12kB
79
# 
80
# Step 1/2 : FROM nginx:stable-alpine
81
#  ---> f2343e2e2507
82
# Step 2/2 : COPY ./development.conf /etc/nginx/conf.d/default.conf
83
#  ---> Using cache
84
#  ---> 02a55d005a98
85
# Successfully built 02a55d005a98
86
# Successfully tagged notes-router:dev
87
# Creating notes-client-dev ... done
88
# Creating notes-api-dev    ... done
89
# Creating notes-router-dev ... done
90
# Creating notes-db-dev     ... done
Copied!
Now visit http://localhost:8080 and voilà!
Try adding and deleting notes to see if the application works properly or not. The project also comes with shell scripts and a Makefile. Explore them to see how you can run this project without the help of docker-compose like you did in the previous section.
Containerizing a Multi-Container JavaScript Application
Conclusion
Last modified 1yr ago
Copy link
Contents
Compose Basics
What About the Network Bridge?
Starting Services
Listing Services
Executing Commands Inside a Running Service
Accessing Logs From a Running Service
Stopping Services
Composing a Full-stack Application