SwarmKit Architecture¶

Author : Rajdeep Dua
Last Updated : June 20 2016

Introduction
- Node
- Manager
  - Dispatcher
  - Scheduler
  - Allocator
  - RaftNode
  - Cluster
Agent
- Worker
- TaskManager
- Reporter
- Controller
- ContainerConfig
- Task
- Storage

Introduction ¶

In this article we look at the overall Architecture of Swarmkit. Swarmkit is a distributed resource manager. This can be bundled to run Docker tasks or other types of Tasks.

Main components of Swarmkit:

Swarmkit is composed two types of Nodes

Managers : Responsible for Assigning Tasks to Workers
Workers : Place where actual tasks run

Following entities are scheduled on Workerss

Tasks is the unit of work performed by a Worker.
Service is a bundle of Tasks which are run as a single unit and monitored by the Manager

Node ¶

Node struct implements the node functionality for a member of a swarm cluster. Node handles workloads (as a worker) and may also run as a manager.

type Node struct {
        sync.RWMutex
        config        *NodeConfig
        remotes       *persistentRemotes
        role          string
        conn          *grpc.ClientConn
        connCond      *sync.Cond
        nodeID        string
        ....
        err           error
        agent         *Agent
        manager       *manager.Manager
        roleChangeReq chan api.NodeRole
}

Every node has a reference to a Manager instance and Agent instance.

Node can have four states

started
stopped
ready
closed

Manager ¶

Manager has following components at Runtime

Dispatcher
Allocator
Scheduler
GRPC Server
GRPC Local Server
Global Orchestrator

golang struct for the manager is listed below

type Manager struct {
        config    *Config
        listeners map[string]net.Listener
        caserver               *ca.Server
        Dispatcher             *dispatcher.Dispatcher
        replicatedOrchestrator *orchestrator.ReplicatedOrchestrator
        globalOrchestrator     *orchestrator.GlobalOrchestrator
        taskReaper             *orchestrator.TaskReaper
        scheduler              *scheduler.Scheduler
        allocator              *allocator.Allocator
        keyManager             *keymanager.KeyManager
        server                 *grpc.Server
        localserver            *grpc.Server
        RaftNode               *raft.Node
}

Each Manager listens on a IP Address and port for GRPC messages. Two separate RPC servers run for listening to local and remote connections.

Dispatcher ¶

Dispatcher is responsible for dispatching tasks and tracking agent health.

It has reference to the following elements.
List of managers
lastSeenManagers     []*api.WeightedPeer

This is the last where tasks are dispatched
Raft cluster :Cluster from where dispatcher gets list of managers

Manager Queue where messages are published

Each dispatcher has reference to list of Nodes
Nodes are registered in the dispatcher using following call
func (d *Dispatcher) register(ctx context.Context, nodeID string,
 description *api.NodeDescription) (string, string, error) {
        ...

        rn := d.nodes.Add(node, expireFunc)
        ...
}
Dispatches Tasks to Nodes

Scheduler ¶

Assigns tasks to the Nodes. Following datastructure represents a scheduler. It has reference to following tasks

UnAssigned Tasks unassignedTasks
PreAssigned Tasks preassignedTasks
All Tasks allTasks

type Scheduler struct {
    store           *store.MemoryStore
    unassignedTasks *list.List
    preassignedTasks map[string]*api.Task
    nodeHeap         nodeHeap
    allTasks         map[string]*api.Task
}

Allocator ¶

Controls how allocation in manager is handled. It is called from the run() function of the manager. It is used to allocate Network resources.

RaftNode ¶

Represents a Node data struct. It connects to other nodes using GRPC.

Cluster ¶

This represents the active state of a Cluster, All the nodes are members of this cluster. Every cluster has a unique id.

type Cluster struct {
        id uint64
        members map[uint64]*Member
}

Agent ¶

Agent implements primary node functionality for a member of a swarm cluster. The primary functionality of this Agent is to run on the Node and provide status of the tasks.

Components of a Agent instance are

Reporter
Controller
Task
Worker
Storage
Node
exec.Controller
exec.container.Controller

Agent is represented by the following datastructure

type Agent struct {
        config *Config
        node *api.Node
        keys []*api.EncryptionKey
        sessionq chan sessionOperation
        worker   Worker
        ...
}

Worker implements the core task management logic and persistence. It coordinates the set of assignments with the executor. Each Agent has a single worker which has reference to taskManager Map: map[string]*taskManager and Executor : exec.Executor. Each worker also has a set of Listeners ( map[*statusReporterKey]struct{} ) giving status of the tasks.

type worker struct {
        db        *bolt.DB
        executor  exec.Executor
        listeners map[*statusReporterKey]struct{}

        taskManagers map[string]*taskManager
}

TaskManager ¶

Manages all the tasks for an agent. Has reference to exec.Controller

type taskManager struct {
        task     *api.Task
        ctlr     exec.Controller
        reporter StatusReporter

        updateq chan *api.Task

        shutdown chan struct{}
        closed   chan struct{}
}

Reporter ¶

Reports status of the Tasks. Maintains a map of task ids and the Task Object

type statusReporter struct {
        reporter StatusReporter
        statuses map[string]*api.TaskStatus
        ...
}

Controller ¶

Interface for managing lifecycle of a Task. Makes the call to underlying implementation which talks to Docker APIs

Update(ctx context.Context, t *api.Task) error

Prepare(ctx context.Context) error

Start(ctx context.Context) error

Wait(ctx context.Context) error

Shutdown(ctx context.Context) error

Terminate(ctx context.Context) error

Remove(ctx context.Context) error

Close() error

Container Tutorials