← Back to Note

Docker and Docker-Compose Advanced

Prakash Pun

Prakash Pun - December 22, 2022

12 min read

Continue on Docker and Docker-Compose: A Comprehensive Guide

Basic Docker Commands and Container Management

docker ps -l --format=$FORMAT # List the most recent container with custom format docker tag docker_id image_name # Name containers for better organization docker commit test test1 # Create a new image from container changes docker run --rm -ti ubuntu sleep 5 ## Remove after exit (terminal interactive) docker run -ti ubuntu bash -c "sleep 3; echo all done" # Chain commands docker run -d -ti ubuntu bash # d -> detach (run in background)

Running Things in a Container

Attaching to Containers with docker attach

  • Works with detached containers
  • Connects to interactive containers
  • To detach without stopping: use Control-p Control-q
  • List running containers with docker ps
  • Connect with docker attach container_name

Running Additional Processes with docker exec

  • Starts another process in an existing container
  • Excellent tool for debugging and database administration
  • Cannot add ports, volumes, or other configurations
  • Example: docker exec -ti container_name bash

Monitoring Container Output with docker logs

  • Captures and stores container output
  • Access with docker logs container_name
  • Named containers are easier to track: docker run --name example -d ubuntu bash -c "lose /etc/password"
  • Best practice: Monitor log size to prevent performance issues

Container Lifecycle Management

Stopping and Removing Containers

  • Kill running containers: docker kill container_name
  • Remove stopped containers: docker rm container_name
  • View last exited container: docker ps -l

Resource Constraints for Better Performance

  • Memory limits:
    • docker run --memory maximum-allowed-memory image-name command
  • CPU Limits:
    • docker run --cpu-shares (relative to other containers)
    • docker run --cpu-quota (absolute limits)
  • Resource limiting is essential for orchestration environments

Lessons from Production Environments

  • Important: Don't let containers fetch dependencies at startup
  • Always save important data from stopped containers
  • Use proper tagging and naming conventions

Container Networking

  • Containers are isolated from the internet by default
  • Group containers into private networks for security
  • Explicitly configure connectivity between containers
  • Expose specific ports to allow external connections
  • Create private networks for inter-container communication

Exposing Specific Ports

  • Maps exact ports between container and host
  • Exposes as many ports as needed for your application
  • Requires coordination when running multiple containers
  • Makes port discovery straightforward
  • Example:
    docker run --rm -ti -p 45678:45678 -p 45679:45679 --name echo-server ubuntu:14.04 bash # Inside container: nc -lp 45678 | nc -lp 45679 # On host: nc localhost 45678 nc localhost 45679 # From another container: docker run --rm -ti ubuntu:14.04 nc host.docker.internal 45678 # Use host.docker.internal or actual IP on Windows

Dynamic Port Exposure

  • Fixed internal port, dynamic external port
  • Allows multiple containers with same internal ports
  • Works well with service discovery tools
  • Example:
    docker run --rm -ti -p 45678 -p 45679 --name echo-server ubuntu:14.04 bash docker port echo-server # Shows port mappings

UDP Port Configuration

  • Specify protocol with /udp suffix
  • Example:
    docker run -p 1234:1234/udp # UDP port mapping docker run --rm -ti -p 45678/udp --name echo-server ubuntu:14.04 bash nc -ulp 45678 # UDP listener inside container docker port echo-server # Check port mappings docker run --rm -ti ubuntu:14.04 bash nc -u host.docker.internal 34545 # Connect to UDP port

Creating Custom Container Networks

  • List networks: docker network ls
  • Create a network: docker network create learning
  • Connect containers to networks:
    docker run --rm -ti --net learning --name catserver ubuntu:14.04 bash # Inside container: ping catserver # DNS resolution works automatically docker run --rm -ti --net learning --name dogserver ubuntu:14.04 bash nc -lp 1234 # Start a listener # On catserver: nc dogserver 1234 # Connect to dogserver
  • Multiple network connections:
    docker network create catsonly docker network connect catsonly catserver docker run --rm -ti --net catsonly --name bobcatserver ubuntu:14.04 bash

Legacy Linking (Still Useful for Simple Setups)

  • One-way port linking
  • Environment variables shared one-way
  • Depends on startup order
  • Example:
    docker run --rm -ti -e SECRET=theinternetlovescat --name catserver ubuntu:14.04 bash docker run --rm -ti --link catserver --name dogserver ubuntu:14.04 bash # Check: nc -lp 4321 nc dogserver 4321 nc catserver 4321 env # View shared environment variables

Docker Images: Creating and Managing

Tagging Images for Organization

  • Give meaningful names to images
  • docker commit automatically tags images
  • Example: docker commit container_id name_you_want_to_give
  • Standard naming convention:
    registry.example.com:port/organization/image-name:version-tag
  • Use what you need: organization/image-name:tag is often sufficient

Managing Images

  • docker pull - Download images (automatic with docker run)
  • Very useful for offline work and faster deployments
  • docker push - Upload images to registries

Cleaning Up Unused Images

  • Images can consume significant disk space
  • Remove with:
    docker rmi image-name:tag docker rmi image-id

Working with Docker Volumes

  • Shared storage for data persistence and sharing
  • Two main types:
    • Persistent volumes: Data survives container lifecycle
    • Ephemeral volumes: Temporary, removed when unused
  • Not included in image layers (keeps images smaller)

Sharing Data with the Host

  • Mount host folders and files into containers
  • Important: Files must exist before container start
  • Example:
    docker run -ti -v D:\\Projects\\example:/shared-folder ubuntu bash

Sharing Data Between Containers

  • Use volumes-from to share container volumes
  • Example:
    docker run -ti -v /shared-data ubuntu bash docker run -ti --volumes-from nervous_galois ubuntu bash

Docker Registries and Image Distribution

  • Registries manage and distribute container images
  • Docker Hub provides free public registries
  • Organizations can run private registries for security

Finding and Managing Images

  • Search for images: docker search image-name
  • Pull specific images: docker pull debian:sid
  • Tag images for your registry: docker tag debian:sid prakashpun7/test-image:v99.9
  • Push to registries: docker push prakashpun7/test-image:v99.9
  • Best practice: Regularly clean up unused images

Dockerfile: Building Custom Images

What is a Dockerfile?

  • A set of instructions to create a Docker image
  • Build with:
    docker build -t name-of-result . # '.' is the build context path
  • Result is stored in your local Docker registry

Layer-Based Image Building

  • Each line creates a new image layer
  • Previous layers remain unchanged
  • Avoid spanning large files across multiple lines (bloats the image)
  • Layer caching speeds up builds significantly

Caching Considerations

  • Docker skips unchanged layers during builds
  • Watch for "using cache" in build output
  • Changing parts should be at the end of Dockerfile
  • Package installation early, code changes later

Dockerfile vs. Shell Scripts

  • Dockerfile looks like shell scripts but differs fundamentally
  • Each line runs in isolation
  • Processes started on one line won't run on the next line
  • Environment variables persist between lines
  • Example:
    # FROM debian:sid # RUN apt-get -y update # RUN apt-get -y install nano # CMD ["bin/nano", "/tmp/notes"] FROM example/nanoer ADD notes.txt /notes.txt CMD ["bin/nano", "/notes.txt"]

Common Dockerfile Instructions

FROM Statement

  • Specifies the base image
  • Must be the first command
  • Example: FROM java:8

MAINTAINER Statement

RUN Statement

  • Executes commands and saves results
  • Example:
    RUN unzip install.zip /opt/install/ RUN echo Hello

ADD Statement

  • Copies files from build context or URLs
  • Automatically extracts archives
  • Examples:
    ADD run.sh /run.sh ADD project.tar.gz /install/ ADD https://project.example.com/download/1.0/project.rpm /project/

ENV Statement

  • Sets environment variables during build and runtime
  • Example:
    ENV DB_HOST=db.production.example.com ENV DB_PORT=5432

ENTRYPOINT and CMD Statements

  • ENTRYPOINT: Start of the command
  • CMD: Complete command or arguments to ENTRYPOINT
  • Combined when both exist
  • Use ENTRYPOINT for command-line-like containers
  • Default to CMD when unsure

Shell Form vs. Exec Form

  • Shell form: nano notes.txt
  • Exec form: ["/bin/nano", "notes.txt"]
  • Both valid for ENTRYPOINT, RUN, and CMD

EXPOSE Statement

  • Maps ports into the container
  • Example: EXPOSE 8080

VOLUME Statement

  • Defines shared or ephemeral volumes
  • Examples:
    VOLUME ["/host/path/", "/container/path/"] VOLUME ["/shared-data"]
  • Avoid defining shared folders in production Dockerfiles

WORKDIR Statement

  • Sets container's starting directory
  • Example: WORKDIR /install/

USER Statement

  • Sets container's runtime user
  • Examples:
    USER arthur USER 100

Multi-Stage Dockerfile Examples

Standard Approach (Larger Image)

FROM ubuntu:16.04 RUN apt-get update RUN apt-get -y install curl RUN curl https://google.com | wc -c > google-size ENTRYPOINT echo google is this big; cat google-size

Multi-Stage Build (Smaller Image)

FROM ubuntu:16.04 as builder RUN apt-get update RUN apt-get -y install curl RUN curl https://google.com | wc -c > google-size FROM alpine COPY --from=builder /google-size /google-size ENTRYPOINT echo google is this big; cat google-size

Preventing the Golden Image Problem

  • Include installers in your project source
  • Create a canonical build process from scratch
  • Tag builds with git commit hashes
  • Use lightweight base images like Alpine
  • Build public images from Dockerfiles
  • Never leave passwords in layers - delete files in the same layer

Docker Architecture

Kernel Functions

  • Responds to hardware messages
  • Starts and schedules programs
  • Controls storage resources
  • Passes messages between programs
  • Allocates system resources
  • Creates containers through kernel configuration

Docker Implementation Details

  • Written in Go programming language
  • Manages Linux kernel features:
    • cgroups: Process containment
    • namespaces: Network isolation
    • copy-on-write filesystems: Efficient storage

Docker's Primary Purpose

  • Simplifies distributed systems deployment
  • Standardizes application packaging

Docker Client-Server Architecture

  • Docker has separate client and server components
  • Server receives commands over network or file socket
  • Client can run inside Docker containers

Networking Concepts

Networking Layers

  • Ethernet: Frame transport on physical media
  • IP layer: Packet movement on local networks
  • Routing: Forwards packets between networks
  • Ports: Address specific applications

Bridge Networking

  • Docker creates virtual networks using bridge devices
  • Software switches control the Ethernet layer
  • Example:
    docker run -ti --rm --net=host ubuntu:16.04 bash apt-get update && apt-get install bridge-utils brctl show docker network create my-network
  • Host network mode: docker run --net=host options image-name command

Network Routing

  • Uses firewall rules for inter-network communication
  • Employs NAT (Network Address Translation)
  • Rewrites source address on outbound packets
  • Rewrites destination address on inbound packets
  • Example:
    docker run -ti --rm --net=host --privileged=true ubuntu bash apt-get update && apt-get install iptables iptables -n -L -t nat
  • Port exposure creates port forwarding rules

Network Namespaces

  • Attaches processes to private network segments
  • Bridges private networks to shared network
  • Provides virtual network interfaces
  • Each container gets its own networking stack

Linux Process Management

  • Process hierarchy (parent-child relationship)
  • Child processes return exit codes to parents
  • Process Zero (init) starts all others
  • Docker containers start with an init process
  • Containers terminate when the init process exits
  • Example:
    docker inspect --format '{{.State.Pid}}' container-name kill 7538 # Where 7538 is the process ID

Resource Limiting Mechanisms

  • CPU time scheduling
  • Memory allocation limits
  • Inherited limitations and quotas
  • Process hierarchy cannot escape resource limits

Unix Storage Concepts

  • Physical storage devices
  • Logical storage devices
  • Filesystem organization
  • FUSE and network filesystems
  • Copy-on-write for efficiency

Layer Transfer

  • Layers move between containers as gzip files
  • Container independence from storage engines
  • Most containers work on any compatible system
  • Some storage engines have layer count limits

Volume and Bind Mounting

  • Linux Virtual File System (VFS)
  • Device mounting to the VFS
  • Directory mounting to the VFS
  • Example: mount -o bind other-work work
  • Mount order considerations
  • Host filesystem always mounts over guest

Docker Registry Operations

  • Maintains image index and tags
  • Handles authentication and authorization
  • Stores and distributes images
  • Example:
    docker save -o my-images.tar.gz debian:sid busybox ubuntu:14.04 docker rmi debian:sid busybox ubuntu:14.04 docker load -i my-images.tar.gz

Container Migration

  • Save containers: docker save
  • Load containers: docker load
  • Migrate between storage types
  • Offline distribution (high-bandwidth sneakernet)

Container Orchestration

  • Multiple systems manage Docker containers
  • Core functions:
    • Container lifecycle management
    • Service discovery
    • Resource allocation

Docker Compose

  • Single-machine container coordination
  • Optimized for development and testing
  • One command deploys all containers, volumes, and networks

Kubernetes

  • Containers run applications
  • Pods group related containers
  • Services expose pods to the network
  • Labels enable advanced service discovery

Kubernetes Advantages

  • Scriptable operations with kubectl
  • Flexible overlay networking
  • Works on-premises or in cloud environments
  • Built-in service discovery
  • Learn more at

AWS EC2 Container Service (ECS)

ECS Components

  • Task definitions: Container groups
  • Tasks: Container execution
  • Services: Persistent tasks with network exposure

ECS Advantages

  • Integrates with Elastic Load Balancers
  • Host instance creation in AWS
  • Agent-based cluster joining
  • Docker control socket access
  • Docker repository integration
  • CloudFormation stack integration
  • More information at

Practical Example: Node.js Application

FROM node:12 WORKDIR /app COPY package*.json ./ RUN npm install COPY . . ENV PORT=4000 EXPOSE 4000 CMD ["npm", "start"]

Build and run:

docker build -t prakashpun/demoapp:1.0 . docker run -p 5000:4000 container_id

Core Docker Concepts

Dockerfile => Blueprint for building docker images Image => Template for running docker containers Container => Running process with isolated environment