Continue on Docker and Docker-Compose

Copied!
docker ps -l --format=$FORMAT
docker tag docker_id image_name # name the containers

docker commit test test1
docker run --rm -ti ubuntu sleep 5 ## remove after exit (terminal interactive)
docker run -ti ubuntu bash -c "sleep 3; echo all done"

docker run -d -ti ubuntu bash # d -> detach

Running things running in a container

docker attach

• detached containers
• interactive containers
• detach with Control-p control-q
• docker ps
• docker attach container_name

Running more things in a container

docker exec

• Starts another process in an existing container
• great for debugging and DB administration
• can't add ports, volumes, and so on

Looking at container output

docker logs

• keep the output of containers
• View with docker logs container_name
• docker run --name example -d ubuntu bash -c "lose /etc/password" (--name to name the container to example and detach to run in the background, give process bash)
• don't let the output get too large

Stopping and removing containers

• killing and removing containers
• docker kill container_name (stop container)
• docker rm container_name (remove the container)
• docker ps -l (last container to exit)

Resource constraints

• memory limits
  • docker run --memory maximum-allowed-memory-image-name command
• CPU Limits
  • docker run --cpu-shares relative to other containers
  • docker run --cpu-quota to limit it in general
• Orchestration
  • Generally requires resource limiting

Lessons from the field

• Don't let your containers fetch dependencies when they start
• Don't leave important things in unnamed stopped containers

Container Networking

• Programs in containers are isolated from the internet by default
• You can group your containers into "private" networks
• You explicitly choose who can connect to whom
• expose ports to let connections in
• Private networks to connect between containers

Exposing specific ports

• Explicitly specifies the port inside the container and outside
• Exposes as many ports as you want
• Requires coordination between containers
• Makes it easy to find the exposed ports
• docker run --rm -ti -p 45678:45678 -p 45679:45679 --name echo-server ubuntu:14.04 bash (publish 45678 to 45678)
  • nc -lp 45678 | nc -lp 45679
  • nc localhost 45678
  • nc localhost 45679
  • docker run --rm -ti ubuntu:14.04
  • nc host.docker.internal 45678 (inside the running container) (put the ip address for windows for host.docker.internal)

Exposing Ports Dynamically

• The port inside the container is fixed
• The port on the host is chosen from the unused ports
• This allows many containers to run programs with fixed ports
• This often is used with a service discovery program
• without port docker run --rm -ti -p 45678 -p 45679 --name echo-server ubuntu:14.04 bash
• docker port echo-server

Exposing UDP Ports

• docker run -p outside-port:inside-port/protocol (tcp/udp)
• docker run -p 1234:1234/upd
• docker run --rm -ti -p 45678/udp --name echo-server ubuntu:14.04 bash
• nc -ulp 45678
• docker port echo-server
• docker run --rm -ti ubuntu:14.04 bash
• nc -u host.docker.internal 34545

Container Networking

• docker network ls
• docker network create learning create a network
• docker run --rm -ti --net learning --name catserver ubuntu:14.04 bash
  • ping catserver
• docker run --rm -ti --net learning --name dogserver ubuntu:14.04 bash
  • nc -lp 1234
  • on catserver container nc dogserver 1234
• docker network create catsonly
• docker network connect catsonly catserver
• docker run --rm -ti --net catsonly --name bobcatserver ubuntu:14.04 bash

Legacy Linking

• Links all ports, though only one way
• Secret environment variables are shared only one way
• Depends on the startup order
• Restarts only sometimes breaks the links
• docker run --rm -ti -e SECRET=theinternetlovescat --name catserver ubuntu:14.04 bash
• docker run --rm -ti --link catserver --name dogserver ubuntu:14.04 bash
  • nc -lp 4321
  • nc dogserver 4321
  • nc catserver 4321
  • env

Images

Tagging Images

• Tagging gives images names
• docker commit tags images for you
• docker commit container_id name_yo_like_to_give
• docker commit ea4fd18fc19d my-image:v2.1
• This is an example of the name structure:
  • registry.example.com:port/organ-ization/image-name:version-tag
• You can leave out the parts you don't need
• Organization/image-name is often enough
• docker pull
• Run automatically by docker run
• Useful for offline work
• Opposite docker push

Cleaning UP

• Images can accumulate quickly
  • docker rmi image-name:tag
  • docker rmi image-id

Volumes

• Shared folder
• Virtual 'discs' to store and share data
• Two main varieties
  • Persistent (data will be still there after the container is away)
  • Ephemeral (temporary, no container is not using them it goes away)
• Not part of images

Sharing Data with the host

• "Shared folders" with the host
• Sharing a "single file" into a container
• Note that a file must exist before you start the container, or it will be assumed to be a directory
• docker run -ti -v D:\\Projects\\example:/shared-folder ubuntu bash

Sharing data between containers

• volumes-from
• Shared "discs" that exist only as long as they are being used
• Can be shared between containers
• docker run -ti -v /shared-data ubuntu bash
• docker run -ti --volumes-from nervous_galois ubuntu bash

Docker Registries

• Registries manage and distribute images
• Docker (the company) offers these for free
• You can run your own, as well

Finding images

• Docker search commands
• docker pull debian:sid
• docker tag debian:sid prakashpun7/test-image:v99.9
• docker push prakashpun7/test-image:v99.9
• Clean up your images regularly

What is a Dockerfile

• This is a small "program" to create an image
• You run this program with
  • docker build -t name-of-result . Note that. here (where to search for Dockerfile)
• When it finishes, the result will be in your local docker registry

Producing the Next Image with Each Step

• Each line takes the image from the previous line and makes another image
• The previous image is unchanged
• It does not edit the state from the previous line
• You don't want large files to span lines or your image will be huge

Caching with each step

• This is important; watch the build output for "using cache"
• Docker skips lines that have not changed since the last build
• If your first line is "download the latest file", it may not always run
• This caching saves huge amounts of time
• The parts that change the most belong at the end of the Dockerfile

Dockerfile are not Shell Scripts

• Dockerfiles look like shell scripts
• Dockerfiles are not shell scripts
• processes you start on one line will not be running on the next line
• Environment variables you set will be set on the next line
  • if you use the ENV command, remember that each line is its own call to docker run

Copied!
# FROM debian:sid
# RUN apt-get -y update
# RUN apt-get -y install nano
# CMD ["bin/nano", "/tmp/notes"]
FROM example/nanoer
ADD notes.txt /notes.txt
CMD ["bin/nano", "/notes.txt"]

The FROM Statement

• Which image to download and start from
• Must be the first command in your Dockerfile
  • FROM java:8

The MAINTAINER Statement

• Defines the author of this Dockerfile
  • MAINTAINER Firstname Lastname 

The RUN Statement

• Runs the comand line, waits for it to finish, and saves the result
  • RUN unzip install.zip /opt/install/
  • RUN echo Hello

The ADD Statement

• Adds local files
  • Add
    /run.sh
• Add the contents of tar archives
  • ADD project.tar.gz /install/
• Works with URLs as well
• ADD <https://project.example.com/download> /1.0/project.rpm /project/

The ENV Statement

• Sets environment variables
• Both during the build and when running the result
  • ENV DB_HOSt=
  • ENV DB_PORT=5432

The ENTRYPOINT and CMD Statements

• ENTRYPOINT specifies the start of the command to run
• CMD specifies the whole command to run
• If you have both ENTRYPOINT and CMD, they are combined together
• If your container acts like a command-line program, you can use ENTRYPOINT
• If you are unsure, you can use CMD

Shell From vs.Exec From

• ENTRYPOINT RUN and CMD can use either form.
• Shell form looks like this:
  • nano notes.txt
• Exec form looks like this:
  • ["/bin/nano", "notes.txt"]

The EXPOSE Statement

• Maps a port into the container
  • EXPOSE 8080

The VOLUME Statment

• Defines shared or ephemeral volumes
  • VOLUME ["/host/path/", "/container/path/"]
  • VOLUME ["/shared-data"]
• Avoid defining shared folders in DockerFiles

The WORKDIR Statement

• Sets the directory the container starts in
  • WORKDIR /install/

The USER Statment

• Sets which user the container will run as
  • USER arthur
  • USER 100

Multi-project Docker files

• docker build -t google-size .

Copied!
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get -y install curl
RUN curl <https://google.com> | wc -c > google-size
ENTRYPOINT echo google is this big; cat google-size

Copied!
FROM ubuntu:16.04 as builder
RUN apt-get update
RUN apt-get -y install curl
RUN curl <https://google.com> | wc -c > google-size

FROM alpine
COPY --from=builder /google-size /google-size

ENTRYPOINT echo google is this big; cat google-size

Preventing the Golder Image Problem

• Include installers in your project
• Have a canonical build that builds everything completely from scratch
• Tag your builds with the git hash of the code that build it
• Use small images, such as Alpine
• Build images you shae publicly from Dockerfiles, always
• Don't ever leave passwords in layers; delete files inthe same step!

What Kernels Do

• Respond to message from the hardware
• Start and schedule programs
• Control and organize storage
• Pass messages between programs
• Allocate resources, memory, CPU, network, and so on.
• Create containers by Docker configuring the kernel

What Docker does

• Program is written in Go -- upcoming systems language
• Manages kernel features
  • Uses "cgroups" to contain process
  • Uses "namespaces" to contain networks
  • Uses "copy-on-wirte" filesystems to build images

What Docker Really Does

• Makes scripting distributed systems "easy"*
  • For a very peculiar definition of "easy"

The Docker Control Socket

• Docker is two programs: a client and a server
• The server receives commands over a socket (either over a network or through a "file")
• The client can even run inside the docker itself

Networking in Brief

• Ethernet: moves "frames" on a wire (or Wi-Fi)
• IP layer: moves packets on a local network
• Routing: forwards packets between networks
• Ports: address particular programs on a computer

Bridging

• Docker uses bridges to create virtual networks in your computer
• These are software switches
• They control the Ethernet layer
• docker run -ti --rm --net=host ubuntu:16.04 bash
• apt-get update && apt-get install bridge-utils
• brctl show
• docker network create my-network
• You can turn off this protection with
  • docker run --net=host options image-name command

Routing

• Creates "firewall" rules to move packets between networks
• NAT (Network Address Translation)
• Change the source address on the way out
• Change the destination address on the way back in
• docker run -ti --rm --net=host --privileged=true ubuntu bash
• apt-get update && apt-get install iptables
• iptables -n -L -t nat
• "Exposing" a port --really "Port forwarding"

Namespaces

• They allow processes to be attached to private network segments
• These private networks are bridged into a shared network with the rest of the containers
• Containers have virtual network "cards"
• Containers get their own copy of the networking stack

Primer on Linux Processes

• Processes come from other processes--parent-child relationship
• When a child process exits, it returns an exit code to its parent
• Process Zero is special; called init, the process that starts the rest
• In Docker, your container starts with an init process and vanishes when that process exits
• docker inspect --format '{{.State.Pid}}' container-name
• kill 7538 => 7538 = pid

Resource Limiting

• Scheduling CPU time
• Memory allocation limits
• Inherited limitations and quotas
• Can't escape your limits by starting more process

Unix Storage in Brief

• Actual storage devices
• Logical storage devices
• Filesystems
• FUSE filesystems and network filesystems
• COW;s => copy on write

Moving Cows

• The contents of layers are moved between containers in gzip files
• Containers are independent of the storage engine
• Any containers can be loaded (almost) anywhere
• It is possible to run out of layers on some of the storage engines

Volumes and Bind Mounting

• The Linux VFS (Virtual File System)
• Mounting devices on the VFS
• Mounting directories on the VFS
• mount -o bind other-work work
• Getting the mount order correct
• Mounting volumes --always mounts the host's filesystem over the guest

What is a Docker Registry?

• Maintains an index and searches tags
• Authorizes and authenticates connections (sometimes)
• docker save -o my-images.tar.gz debian:sid busybox ubuntu:14.04
• docker rmi debian:sid busybox ubuntu:14.04
• docker load -i my-images.tar.gz

Saving and Loading Containers

• docker save
• docker load
• Migrating between storage types
• Shipping images on disks (or never underestimate the bandwidth of a thumb drive in a get liner)

Intro to orchestration (One Container = One Hand Clapping)

• Many orchestration systems of Docker
• Start containers--and restart them if they fail
• Service discovery--allow them to find each other
• Resource allocation--match containers to computers

Docker Compose

• Single machine coordination
• Designed for testing and development
• Brings up all your containers, volumes, networks, etc., with one command

Kubernetes

• Containers run programs
• Pods group containers together
• Services make pods available to others.
• Labels are used for very advanced service discovery

Advantages of Kubernetes

• Make scripting large operations possible with the *kubect1* command
• Very flexible overlay networking
• Runs equally well on your hardware or a cloud provider
• Built-in service discovery
• Get started at 

EC2 Container Service (ECS)

• Task definitions
  • Define a set of containers that always run together
• Tasks
  • Actually makes a container run right now
• Services and exposes it to the Net
  • Ensures that a task is running all the time

Advantages of ECS

• Connects load balancers (ELBs) to services
• Can create your own host instances in AWS
• Make your instances start the agent and join the cluster
• Pass the docker control socket into the agent
• Provides docker repos--and it's easy to run your own repo
• Note that containers (tasks) can be part of CloudFormation stacks!
• Get started at 

Copied!
FROM node:12
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
ENV PORT=4000
EXPOSE 4000
CMD ["npm", "start"]

docker build -t prakashpun/demoapp:1.0 . docker run -p 5000:4000 container_id

Copied!
=> docker file 
=> blueprint for building docker image image 
=> template for running docker container container 
=> running process

Thank You 😁😁😁