Continue on Docker and Docker-Compose: A Comprehensive Guide

Basic Docker Commands and Container Management

docker ps -l --format=$FORMAT  # List the most recent container with custom format
docker tag docker_id image_name # Name containers for better organization

docker commit test test1  # Create a new image from container changes
docker run --rm -ti ubuntu sleep 5 ## Remove after exit (terminal interactive)
docker run -ti ubuntu bash -c "sleep 3; echo all done"  # Chain commands

docker run -d -ti ubuntu bash # d -> detach (run in background)

Running Things in a Container

Attaching to Containers with docker attach

• Works with detached containers
• Connects to interactive containers
• To detach without stopping: use Control-p Control-q
• List running containers with docker ps
• Connect with docker attach container_name

Running Additional Processes with docker exec

• Starts another process in an existing container
• Excellent tool for debugging and database administration
• Cannot add ports, volumes, or other configurations
• Example: docker exec -ti container_name bash

Monitoring Container Output with docker logs

• Captures and stores container output
• Access with docker logs container_name
• Named containers are easier to track: docker run --name example -d ubuntu bash -c "lose /etc/password"
• Best practice: Monitor log size to prevent performance issues

Container Lifecycle Management

Stopping and Removing Containers

• Kill running containers: docker kill container_name
• Remove stopped containers: docker rm container_name
• View last exited container: docker ps -l

Resource Constraints for Better Performance

• Memory limits:
  • docker run --memory maximum-allowed-memory image-name command
• CPU Limits:
  • docker run --cpu-shares (relative to other containers)
  • docker run --cpu-quota (absolute limits)
• Resource limiting is essential for orchestration environments

Lessons from Production Environments

• Important: Don't let containers fetch dependencies at startup
• Always save important data from stopped containers
• Use proper tagging and naming conventions

Container Networking

• Containers are isolated from the internet by default
• Group containers into private networks for security
• Explicitly configure connectivity between containers
• Expose specific ports to allow external connections
• Create private networks for inter-container communication

Exposing Specific Ports

• Maps exact ports between container and host
• Exposes as many ports as needed for your application
• Requires coordination when running multiple containers
• Makes port discovery straightforward
• Example:

  docker run --rm -ti -p 45678:45678 -p 45679:45679 --name echo-server ubuntu:14.04 bash
  # Inside container:
  nc -lp 45678 | nc -lp 45679
  # On host:
  nc localhost 45678
  nc localhost 45679
  # From another container:
  docker run --rm -ti ubuntu:14.04
  nc host.docker.internal 45678  # Use host.docker.internal or actual IP on Windows
  

Dynamic Port Exposure

• Fixed internal port, dynamic external port
• Allows multiple containers with same internal ports
• Works well with service discovery tools
• Example:

  docker run --rm -ti -p 45678 -p 45679 --name echo-server ubuntu:14.04 bash
  docker port echo-server  # Shows port mappings
  

UDP Port Configuration

• Specify protocol with /udp suffix
• Example:

  docker run -p 1234:1234/udp  # UDP port mapping
  docker run --rm -ti -p 45678/udp --name echo-server ubuntu:14.04 bash
  nc -ulp 45678  # UDP listener inside container
  docker port echo-server  # Check port mappings
  docker run --rm -ti ubuntu:14.04 bash
  nc -u host.docker.internal 34545  # Connect to UDP port
  

Creating Custom Container Networks

• List networks: docker network ls
• Create a network: docker network create learning
• Connect containers to networks:

  docker run --rm -ti --net learning --name catserver ubuntu:14.04 bash
  # Inside container:
  ping catserver  # DNS resolution works automatically
  
  docker run --rm -ti --net learning --name dogserver ubuntu:14.04 bash
  nc -lp 1234  # Start a listener
  # On catserver:
  nc dogserver 1234  # Connect to dogserver
  

• Multiple network connections:

  docker network create catsonly
  docker network connect catsonly catserver
  docker run --rm -ti --net catsonly --name bobcatserver ubuntu:14.04 bash
  

Legacy Linking (Still Useful for Simple Setups)

• One-way port linking
• Environment variables shared one-way
• Depends on startup order
• Example:

  docker run --rm -ti -e SECRET=theinternetlovescat --name catserver ubuntu:14.04 bash
  docker run --rm -ti --link catserver --name dogserver ubuntu:14.04 bash
  # Check:
  nc -lp 4321
  nc dogserver 4321
  nc catserver 4321
  env  # View shared environment variables
  

Docker Images: Creating and Managing

Tagging Images for Organization

• Give meaningful names to images
• docker commit automatically tags images
• Example: docker commit container_id name_you_want_to_give
• Standard naming convention:

  registry.example.com:port/organization/image-name:version-tag
  

• Use what you need: organization/image-name:tag is often sufficient

Managing Images

• docker pull - Download images (automatic with docker run)
• Very useful for offline work and faster deployments
• docker push - Upload images to registries

Cleaning Up Unused Images

• Images can consume significant disk space
• Remove with:

  docker rmi image-name:tag
  docker rmi image-id
  

Working with Docker Volumes

• Shared storage for data persistence and sharing
• Two main types:
  • Persistent volumes: Data survives container lifecycle
  • Ephemeral volumes: Temporary, removed when unused
• Not included in image layers (keeps images smaller)

Sharing Data with the Host

• Mount host folders and files into containers
• Important: Files must exist before container start
• Example:

  docker run -ti -v D:\\Projects\\example:/shared-folder ubuntu bash
  

Sharing Data Between Containers

• Use volumes-from to share container volumes
• Example:

  docker run -ti -v /shared-data ubuntu bash
  docker run -ti --volumes-from nervous_galois ubuntu bash
  

Docker Registries and Image Distribution

• Registries manage and distribute container images
• Docker Hub provides free public registries
• Organizations can run private registries for security

Finding and Managing Images

• Search for images: docker search image-name
• Pull specific images: docker pull debian:sid
• Tag images for your registry: docker tag debian:sid prakashpun7/test-image:v99.9
• Push to registries: docker push prakashpun7/test-image:v99.9
• Best practice: Regularly clean up unused images

Dockerfile: Building Custom Images

What is a Dockerfile?

• A set of instructions to create a Docker image
• Build with:

  docker build -t name-of-result .  # '.' is the build context path
  

• Result is stored in your local Docker registry

Layer-Based Image Building

• Each line creates a new image layer
• Previous layers remain unchanged
• Avoid spanning large files across multiple lines (bloats the image)
• Layer caching speeds up builds significantly

Caching Considerations

• Docker skips unchanged layers during builds
• Watch for "using cache" in build output
• Changing parts should be at the end of Dockerfile
• Package installation early, code changes later

Dockerfile vs. Shell Scripts

• Dockerfile looks like shell scripts but differs fundamentally
• Each line runs in isolation
• Processes started on one line won't run on the next line
• Environment variables persist between lines
• Example:

  # FROM debian:sid
  # RUN apt-get -y update
  # RUN apt-get -y install nano
  # CMD ["bin/nano", "/tmp/notes"]
  FROM example/nanoer
  ADD notes.txt /notes.txt
  CMD ["bin/nano", "/notes.txt"]
  

Common Dockerfile Instructions

FROM Statement

• Specifies the base image
• Must be the first command
• Example: FROM java:8

MAINTAINER Statement

• Documents the author
• Example: MAINTAINER Firstname Lastname <[email protected]>

RUN Statement

• Executes commands and saves results
• Example:

  RUN unzip install.zip /opt/install/
  RUN echo Hello
  

ADD Statement

• Copies files from build context or URLs
• Automatically extracts archives
• Examples:

  ADD run.sh /run.sh
  ADD project.tar.gz /install/
  ADD https://project.example.com/download/1.0/project.rpm /project/
  

ENV Statement

• Sets environment variables during build and runtime
• Example:

  ENV DB_HOST=db.production.example.com
  ENV DB_PORT=5432
  

ENTRYPOINT and CMD Statements

• ENTRYPOINT: Start of the command
• CMD: Complete command or arguments to ENTRYPOINT
• Combined when both exist
• Use ENTRYPOINT for command-line-like containers
• Default to CMD when unsure

Shell Form vs. Exec Form

• Shell form: nano notes.txt
• Exec form: ["/bin/nano", "notes.txt"]
• Both valid for ENTRYPOINT, RUN, and CMD

EXPOSE Statement

• Maps ports into the container
• Example: EXPOSE 8080

VOLUME Statement

• Defines shared or ephemeral volumes
• Examples:

  VOLUME ["/host/path/", "/container/path/"]
  VOLUME ["/shared-data"]
  

• Avoid defining shared folders in production Dockerfiles

WORKDIR Statement

• Sets container's starting directory
• Example: WORKDIR /install/

USER Statement

• Sets container's runtime user
• Examples:

  USER arthur
  USER 100
  

Multi-Stage Dockerfile Examples

Standard Approach (Larger Image)

FROM ubuntu:16.04
RUN apt-get update
RUN apt-get -y install curl
RUN curl https://google.com | wc -c > google-size
ENTRYPOINT echo google is this big; cat google-size

Multi-Stage Build (Smaller Image)

FROM ubuntu:16.04 as builder
RUN apt-get update
RUN apt-get -y install curl
RUN curl https://google.com | wc -c > google-size

FROM alpine
COPY --from=builder /google-size /google-size
ENTRYPOINT echo google is this big; cat google-size

Preventing the Golden Image Problem

• Include installers in your project source
• Create a canonical build process from scratch
• Tag builds with git commit hashes
• Use lightweight base images like Alpine
• Build public images from Dockerfiles
• Never leave passwords in layers - delete files in the same layer

Docker Architecture

Kernel Functions

• Responds to hardware messages
• Starts and schedules programs
• Controls storage resources
• Passes messages between programs
• Allocates system resources
• Creates containers through kernel configuration

Docker Implementation Details

• Written in Go programming language
• Manages Linux kernel features:
  • cgroups: Process containment
  • namespaces: Network isolation
  • copy-on-write filesystems: Efficient storage

Docker's Primary Purpose

• Simplifies distributed systems deployment
• Standardizes application packaging

Docker Client-Server Architecture

• Docker has separate client and server components
• Server receives commands over network or file socket
• Client can run inside Docker containers

Networking Concepts

Networking Layers

• Ethernet: Frame transport on physical media
• IP layer: Packet movement on local networks
• Routing: Forwards packets between networks
• Ports: Address specific applications

Bridge Networking

• Docker creates virtual networks using bridge devices
• Software switches control the Ethernet layer
• Example:

  docker run -ti --rm --net=host ubuntu:16.04 bash
  apt-get update && apt-get install bridge-utils
  brctl show
  docker network create my-network
  

• Host network mode: docker run --net=host options image-name command

Network Routing

• Uses firewall rules for inter-network communication
• Employs NAT (Network Address Translation)
• Rewrites source address on outbound packets
• Rewrites destination address on inbound packets
• Example:

  docker run -ti --rm --net=host --privileged=true ubuntu bash
  apt-get update && apt-get install iptables
  iptables -n -L -t nat
  

• Port exposure creates port forwarding rules

Network Namespaces

• Attaches processes to private network segments
• Bridges private networks to shared network
• Provides virtual network interfaces
• Each container gets its own networking stack

Linux Process Management

• Process hierarchy (parent-child relationship)
• Child processes return exit codes to parents
• Process Zero (init) starts all others
• Docker containers start with an init process
• Containers terminate when the init process exits
• Example:

  docker inspect --format '{{.State.Pid}}' container-name
  kill 7538  # Where 7538 is the process ID
  

Resource Limiting Mechanisms

• CPU time scheduling
• Memory allocation limits
• Inherited limitations and quotas
• Process hierarchy cannot escape resource limits

Unix Storage Concepts

• Physical storage devices
• Logical storage devices
• Filesystem organization
• FUSE and network filesystems
• Copy-on-write for efficiency

Layer Transfer

• Layers move between containers as gzip files
• Container independence from storage engines
• Most containers work on any compatible system
• Some storage engines have layer count limits

Volume and Bind Mounting

• Linux Virtual File System (VFS)
• Device mounting to the VFS
• Directory mounting to the VFS
• Example: mount -o bind other-work work
• Mount order considerations
• Host filesystem always mounts over guest

Docker Registry Operations

• Maintains image index and tags
• Handles authentication and authorization
• Stores and distributes images
• Example:

  docker save -o my-images.tar.gz debian:sid busybox ubuntu:14.04
  docker rmi debian:sid busybox ubuntu:14.04
  docker load -i my-images.tar.gz
  

Container Migration

• Save containers: docker save
• Load containers: docker load
• Migrate between storage types
• Offline distribution (high-bandwidth sneakernet)

Container Orchestration

• Multiple systems manage Docker containers
• Core functions:
  • Container lifecycle management
  • Service discovery
  • Resource allocation

Docker Compose

• Single-machine container coordination
• Optimized for development and testing
• One command deploys all containers, volumes, and networks

Kubernetes

• Containers run applications
• Pods group related containers
• Services expose pods to the network
• Labels enable advanced service discovery

Kubernetes Advantages

• Scriptable operations with kubectl
• Flexible overlay networking
• Works on-premises or in cloud environments
• Built-in service discovery
• Learn more at kubernetes.io

AWS EC2 Container Service (ECS)

ECS Components

• Task definitions: Container groups
• Tasks: Container execution
• Services: Persistent tasks with network exposure

ECS Advantages

• Integrates with Elastic Load Balancers
• Host instance creation in AWS
• Agent-based cluster joining
• Docker control socket access
• Docker repository integration
• CloudFormation stack integration
• More information at aws.amazon.com/ecs

Practical Example: Node.js Application

FROM node:12
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
ENV PORT=4000
EXPOSE 4000
CMD ["npm", "start"]

Build and run:

docker build -t prakashpun/demoapp:1.0 .
docker run -p 5000:4000 container_id

Core Docker Concepts

Dockerfile => Blueprint for building docker images
Image => Template for running docker containers
Container => Running process with isolated environment