org.apache.storm.scheduler

Class Cluster

java.lang.Object

org.apache.storm.scheduler.Cluster

public class Cluster
extends Object

Constructor Summary

Constructors

Constructor and Description
Cluster(Cluster src) Copy constructor
Cluster(INimbus nimbus, Map<String,SupervisorDetails> supervisors, Map<String,SchedulerAssignmentImpl> assignments, Map storm_conf)

Method Summary

Modifier and Type	Method and Description
void	assign(WorkerSlot slot, String topologyId, Collection<ExecutorDetails> executors) Assign the slot to the executors for this topology.
void	blacklistHost(String host)
void	freeSlot(WorkerSlot slot) Free the specified slot.
void	freeSlots(Collection<WorkerSlot> slots) free the slots.
Set<Integer>	getAssignablePorts(SupervisorDetails supervisor)
List<WorkerSlot>	getAssignableSlots()
List<WorkerSlot>	getAssignableSlots(SupervisorDetails supervisor)
static Double	getAssignedMemoryForSlot(Map<String,Object> topConf) Get heap memory usage for a worker’s main process and logwriter process.
int	getAssignedNumWorkers(TopologyDetails topology)
SchedulerAssignment	getAssignmentById(String topologyId) get the current assignment for the topology.
Map<String,SchedulerAssignment>	getAssignments() Get all the assignments.
Set<Integer>	getAvailablePorts(SupervisorDetails supervisor) Return the available ports of this supervisor.
List<WorkerSlot>	getAvailableSlots()
List<WorkerSlot>	getAvailableSlots(SupervisorDetails supervisor) Return all the available slots on this supervisor.
Set<String>	getBlacklistedHosts()
double	getClusterTotalCPUResource() Get the total amount of CPU resources in cluster
double	getClusterTotalMemoryResource() Get the total amount of memory resources in cluster
Map	getConf()
String	getHost(String supervisorId)
INimbus	getINimbus()
Map<String,List<ExecutorDetails>>	getNeedsSchedulingComponentToExecutors(TopologyDetails topology)
Map<ExecutorDetails,String>	getNeedsSchedulingExecutorToComponents(TopologyDetails topology)
Map<String,List<String>>	getNetworkTopography()
Map<String,String>	getStatusMap() Get all topology scheduler statuses
SupervisorDetails	getSupervisorById(String nodeId) Get a specific supervisor with the nodeId
Map<String,SupervisorDetails>	getSupervisors() Get all the supervisors.
List<SupervisorDetails>	getSupervisorsByHost(String host) Get all the supervisors on the specified host.
Map<String,Double[]>	getSupervisorsResourcesMap() Get the amount of used and free resources on a supervisor.
Map<String,Double[]>	getTopologyResourcesMap() Get the amount of resources used by topologies.
Collection<ExecutorDetails>	getUnassignedExecutors(TopologyDetails topology) get the unassigned executors of the topology.
Set<Integer>	getUsedPorts(SupervisorDetails supervisor) Get all the used ports of this supervisor.
Collection<WorkerSlot>	getUsedSlots()
Collection<WorkerSlot>	getUsedSlotsByTopologyId(String topologyId) get slots used by a topology
Map<String,Map<WorkerSlot,Double[]>>	getWorkerResourcesMap() Gets the reference to the full topology->worker resource map.
boolean	isBlackListed(String supervisorId)
boolean	isBlacklistedHost(String host)
boolean	isSlotOccupied(WorkerSlot slot)
boolean	needsScheduling(TopologyDetails topology) Does the topology need scheduling?
List<TopologyDetails>	needsSchedulingTopologies(Topologies topologies)
void	setAssignments(Map<String,SchedulerAssignment> newAssignments) set assignments for cluster
void	setBlacklistedHosts(Set<String> hosts)
void	setNetworkTopography(Map<String,List<String>> networkTopography)
void	setStatus(String topologyId, String status) set scheduler status for a topology
void	setStatusMap(Map<String,String> statusMap) set scheduler status map
void	setSupervisorResources(String supervisorId, Double[] resources)
void	setSupervisorsResourcesMap(Map<String,Double[]> supervisorResources) Sets the amount of used and free resources on a supervisor.
void	setTopologyResources(String topologyId, Double[] resources) Set the amount of resources used used by a topology.
void	setTopologyResourcesMap(Map<String,Double[]> topologyResources) Set the amount of resources used used by a topology.
void	setWorkerResources(String topologyId, Map<WorkerSlot,Double[]> resources) Set the worker resources map for a specific topologyId
void	setWorkerResourcesMap(Map<String,Map<WorkerSlot,Double[]>> resources) Set the worker resources map for all topologies in source
void	updateAssignedCpuForTopologyAndSupervisor(Topologies topologies) Update CPU usage for each topology and each supervisor node
void	updateAssignedMemoryForTopologyAndSupervisor(Topologies topologies)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Cluster

public Cluster(INimbus nimbus,
               Map<String,SupervisorDetails> supervisors,
               Map<String,SchedulerAssignmentImpl> assignments,
               Map storm_conf)

Cluster

public Cluster(Cluster src)

Copy constructor

Method Detail

setBlacklistedHosts

public void setBlacklistedHosts(Set<String> hosts)

getBlacklistedHosts

public Set<String> getBlacklistedHosts()

blacklistHost

public void blacklistHost(String host)

isBlackListed

public boolean isBlackListed(String supervisorId)

isBlacklistedHost

public boolean isBlacklistedHost(String host)

getHost

public String getHost(String supervisorId)

needsSchedulingTopologies

public List<TopologyDetails> needsSchedulingTopologies(Topologies topologies)

Returns:

all the topologies which needs scheduling.

needsScheduling

public boolean needsScheduling(TopologyDetails topology)

Does the topology need scheduling?

A topology needs scheduling if one of the following conditions holds:

• Although the topology is assigned slots, but is squeezed. i.e. the topology is assigned less slots than desired.
• There are unassigned executors in this topology

getNeedsSchedulingExecutorToComponents

public Map<ExecutorDetails,String> getNeedsSchedulingExecutorToComponents(TopologyDetails topology)

Parameters:

topology -

Returns:

a executor -> component-id map which needs scheduling in this topology.

getNeedsSchedulingComponentToExecutors

public Map<String,List<ExecutorDetails>> getNeedsSchedulingComponentToExecutors(TopologyDetails topology)

Parameters:

topology -

Returns:

a component-id -> executors map which needs scheduling in this topology.

getUsedPorts

public Set<Integer> getUsedPorts(SupervisorDetails supervisor)

Get all the used ports of this supervisor.

getAvailablePorts

public Set<Integer> getAvailablePorts(SupervisorDetails supervisor)

Return the available ports of this supervisor.

getAssignablePorts

public Set<Integer> getAssignablePorts(SupervisorDetails supervisor)

getAvailableSlots

public List<WorkerSlot> getAvailableSlots(SupervisorDetails supervisor)

Return all the available slots on this supervisor.

getAssignableSlots

public List<WorkerSlot> getAssignableSlots(SupervisorDetails supervisor)

getUnassignedExecutors

public Collection<ExecutorDetails> getUnassignedExecutors(TopologyDetails topology)

get the unassigned executors of the topology.

getAssignedNumWorkers

public int getAssignedNumWorkers(TopologyDetails topology)

Parameters:

topology -

Returns:

the number of workers assigned to this topology.

assign

public void assign(WorkerSlot slot,
                   String topologyId,
                   Collection<ExecutorDetails> executors)

Assign the slot to the executors for this topology.

Throws:

RuntimeException - if the specified slot is already occupied.

getAvailableSlots

public List<WorkerSlot> getAvailableSlots()

Returns:

all the available worker slots in the cluster.

getAssignableSlots

public List<WorkerSlot> getAssignableSlots()

freeSlot

public void freeSlot(WorkerSlot slot)

Free the specified slot.

Parameters:

slot -

freeSlots

public void freeSlots(Collection<WorkerSlot> slots)

free the slots.

Parameters:

slots -

isSlotOccupied

public boolean isSlotOccupied(WorkerSlot slot)

Parameters:

slot - the slot be to checked.

Returns:

true if the specified slot is occupied.

getAssignmentById

public SchedulerAssignment getAssignmentById(String topologyId)

get the current assignment for the topology.

getUsedSlotsByTopologyId

public Collection<WorkerSlot> getUsedSlotsByTopologyId(String topologyId)

get slots used by a topology

getSupervisorById

public SupervisorDetails getSupervisorById(String nodeId)

Get a specific supervisor with the nodeId

getUsedSlots

public Collection<WorkerSlot> getUsedSlots()

getSupervisorsByHost

public List<SupervisorDetails> getSupervisorsByHost(String host)

Get all the supervisors on the specified host.

Parameters:

host - hostname of the supervisor

Returns:

the SupervisorDetails object.

getAssignments

public Map<String,SchedulerAssignment> getAssignments()

Get all the assignments.

setAssignments

public void setAssignments(Map<String,SchedulerAssignment> newAssignments)

set assignments for cluster

getSupervisors

public Map<String,SupervisorDetails> getSupervisors()

Get all the supervisors.

getClusterTotalCPUResource

public double getClusterTotalCPUResource()

Get the total amount of CPU resources in cluster

getClusterTotalMemoryResource

public double getClusterTotalMemoryResource()

Get the total amount of memory resources in cluster

getNetworkTopography

public Map<String,List<String>> getNetworkTopography()

setNetworkTopography

public void setNetworkTopography(Map<String,List<String>> networkTopography)

getAssignedMemoryForSlot

public static Double getAssignedMemoryForSlot(Map<String,Object> topConf)

Get heap memory usage for a worker’s main process and logwriter process.

Parameters:

topConf -

• the topology config

Returns:

the assigned memory (in MB)

updateAssignedMemoryForTopologyAndSupervisor

public void updateAssignedMemoryForTopologyAndSupervisor(Topologies topologies)

updateAssignedCpuForTopologyAndSupervisor

public void updateAssignedCpuForTopologyAndSupervisor(Topologies topologies)

Update CPU usage for each topology and each supervisor node

setStatus

public void setStatus(String topologyId,
                      String status)

set scheduler status for a topology

getStatusMap

public Map<String,String> getStatusMap()

Get all topology scheduler statuses

setStatusMap

public void setStatusMap(Map<String,String> statusMap)

set scheduler status map

setTopologyResources

public void setTopologyResources(String topologyId,
                                 Double[] resources)

Set the amount of resources used used by a topology. Used for displaying resource information on the UI

Parameters:

topologyId -

resources - describes the resources requested and assigned to topology in the following format in an array: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}

setTopologyResourcesMap

public void setTopologyResourcesMap(Map<String,Double[]> topologyResources)

Set the amount of resources used used by a topology. Used for displaying resource information on the UI

Parameters:

topologyResources - a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}

getTopologyResourcesMap

public Map<String,Double[]> getTopologyResourcesMap()

Get the amount of resources used by topologies. Used for displaying resource information on the UI

Returns:

a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}

setSupervisorResources

public void setSupervisorResources(String supervisorId,
                                   Double[] resources)

setSupervisorsResourcesMap

public void setSupervisorsResourcesMap(Map<String,Double[]> supervisorResources)

Sets the amount of used and free resources on a supervisor. Used for displaying resource information on the UI

Parameters:

supervisorResources - a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}

getSupervisorsResourcesMap

public Map<String,Double[]> getSupervisorsResourcesMap()

Get the amount of used and free resources on a supervisor. Used for displaying resource information on the UI

Returns:

a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}

getWorkerResourcesMap

public Map<String,Map<WorkerSlot,Double[]>> getWorkerResourcesMap()

Gets the reference to the full topology->worker resource map.

Returns:

map of topology -> map of worker slot ->resources for that worker

setWorkerResourcesMap

public void setWorkerResourcesMap(Map<String,Map<WorkerSlot,Double[]>> resources)

Set the worker resources map for all topologies in source

Parameters:

resources - map

setWorkerResources

public void setWorkerResources(String topologyId,
                               Map<WorkerSlot,Double[]> resources)

Set the worker resources map for a specific topologyId

Parameters:

topologyId - the id of the topology

resources - map for the topology

getINimbus

public INimbus getINimbus()

getConf

public Map getConf()