SwarmKit Architecture

Author : Rajdeep Dua
Last Updated : June 20 2016

Introduction

In this article we look at the overall Architecture of Swarmkit. Swarmkit is a distributed resource manager. This can be bundled to run Docker tasks or other types of Tasks.

Main components of Swarmkit:

Swarmkit is composed two types of Nodes

  1. Managers : Responsible for Assigning Tasks to Workers
  2. Workers : Place where actual tasks run

Following entities are scheduled on Workerss

  • Tasks is the unit of work performed by a Worker.
  • Service is a bundle of Tasks which are run as a single unit and monitored by the Manager

Node

Node struct implements the node functionality for a member of a swarm cluster. Node handles workloads (as a worker) and may also run as a manager.

type Node struct {
        sync.RWMutex
        config        *NodeConfig
        remotes       *persistentRemotes
        role          string
        conn          *grpc.ClientConn
        connCond      *sync.Cond
        nodeID        string
        ....
        err           error
        agent         *Agent
        manager       *manager.Manager
        roleChangeReq chan api.NodeRole
}

Every node has a reference to a Manager instance and Agent instance.

Node can have four states

  • started
  • stopped
  • ready
  • closed

Manager

Manager has following components at Runtime

  • Dispatcher
  • Allocator
  • Scheduler
  • GRPC Server
  • GRPC Local Server
  • Global Orchestrator
../_images/swarmkit-diagram-manager.png

golang struct for the manager is listed below

type Manager struct {
        config    *Config
        listeners map[string]net.Listener
        caserver               *ca.Server
        Dispatcher             *dispatcher.Dispatcher
        replicatedOrchestrator *orchestrator.ReplicatedOrchestrator
        globalOrchestrator     *orchestrator.GlobalOrchestrator
        taskReaper             *orchestrator.TaskReaper
        scheduler              *scheduler.Scheduler
        allocator              *allocator.Allocator
        keyManager             *keymanager.KeyManager
        server                 *grpc.Server
        localserver            *grpc.Server
        RaftNode               *raft.Node
}

Each Manager listens on a IP Address and port for GRPC messages. Two separate RPC servers run for listening to local and remote connections.

Dispatcher

Dispatcher is responsible for dispatching tasks and tracking agent health.

It has reference to the following elements.

  • List of managers

    lastSeenManagers     []*api.WeightedPeer
    
    This is the last where tasks are dispatched
    
  • Raft cluster :Cluster from where dispatcher gets list of managers

  • Manager Queue where messages are published

  • Each dispatcher has reference to list of Nodes

  • Nodes are registered in the dispatcher using following call

    func (d *Dispatcher) register(ctx context.Context, nodeID string,
     description *api.NodeDescription) (string, string, error) {
            ...
    
            rn := d.nodes.Add(node, expireFunc)
            ...
    }
    
  • Dispatches Tasks to Nodes

Scheduler

Assigns tasks to the Nodes. Following datastructure represents a scheduler. It has reference to following tasks

  • UnAssigned Tasks unassignedTasks
  • PreAssigned Tasks preassignedTasks
  • All Tasks allTasks
type Scheduler struct {
    store           *store.MemoryStore
    unassignedTasks *list.List
    preassignedTasks map[string]*api.Task
    nodeHeap         nodeHeap
    allTasks         map[string]*api.Task
}

Allocator

Controls how allocation in manager is handled. It is called from the run() function of the manager. It is used to allocate Network resources.

RaftNode

Represents a Node data struct. It connects to other nodes using GRPC.

Cluster

This represents the active state of a Cluster, All the nodes are members of this cluster. Every cluster has a unique id.

type Cluster struct {
        id uint64
        members map[uint64]*Member
}

Agent

Agent implements primary node functionality for a member of a swarm cluster. The primary functionality of this Agent is to run on the Node and provide status of the tasks.

../_images/swarmkit-diagram1.png

Components of a Agent instance are

  • Reporter
  • Controller
  • Task
  • Worker
  • Storage
  • Node
  • exec.Controller
  • exec.container.Controller

Agent is represented by the following datastructure

type Agent struct {
        config *Config
        node *api.Node
        keys []*api.EncryptionKey
        sessionq chan sessionOperation
        worker   Worker
        ...
}

Worker

Worker implements the core task management logic and persistence. It coordinates the set of assignments with the executor. Each Agent has a single worker which has reference to taskManager Map: map[string]*taskManager and Executor : exec.Executor. Each worker also has a set of Listeners ( map[*statusReporterKey]struct{} ) giving status of the tasks.

type worker struct {
        db        *bolt.DB
        executor  exec.Executor
        listeners map[*statusReporterKey]struct{}

        taskManagers map[string]*taskManager
}

TaskManager

Manages all the tasks for an agent. Has reference to exec.Controller

type taskManager struct {
        task     *api.Task
        ctlr     exec.Controller
        reporter StatusReporter

        updateq chan *api.Task

        shutdown chan struct{}
        closed   chan struct{}
}

Reporter

Reports status of the Tasks. Maintains a map of task ids and the Task Object

type statusReporter struct {
        reporter StatusReporter
        statuses map[string]*api.TaskStatus
        ...
}

Controller

Interface for managing lifecycle of a Task. Makes the call to underlying implementation which talks to Docker APIs

Update(ctx context.Context, t *api.Task) error

Prepare(ctx context.Context) error

Start(ctx context.Context) error

Wait(ctx context.Context) error

Shutdown(ctx context.Context) error

Terminate(ctx context.Context) error

Remove(ctx context.Context) error

Close() error

ContainerConfig

Reference to the underlying Docker APIs for making the actual calls

Task

Task which needs to be executed on the container

Storage

Agents use BoltDB to store information about the task