Package org.apache.storm.scheduler
Class Cluster
java.lang.Object
org.apache.storm.scheduler.Cluster
- All Implemented Interfaces:
ISchedulingState
- Direct Known Subclasses:
SingleTopologyCluster
The current state of the storm cluster. Cluster is not currently thread safe.
-
Constructor Summary
ConstructorDescriptionCopy constructor.Cluster
(Cluster src, Topologies topologies) Testing Constructor that takes an existing cluster and replaces the topologies in it.Cluster
(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String, SupervisorDetails> supervisors, Map<String, ? extends SchedulerAssignment> assignments, Topologies topologies, Map<String, Object> conf) -
Method Summary
Modifier and TypeMethodDescriptionprotected void
assertValidTopologyForModification
(String topologyId) Check if the given topology is allowed for modification right now.void
assign
(SchedulerAssignment assignment, boolean ignoreSingleExceptions) Assign everything for the given topology.void
assign
(WorkerSlot slot, String topologyId, Collection<ExecutorDetails> executors) Assign the slot to the executors for this topology.void
blacklistHost
(String host) void
freeSlot
(WorkerSlot slot) Free the specified slot.void
freeSlots
(Collection<WorkerSlot> slots) free the slots.Get all scheduled resources for node.getAssignablePorts
(SupervisorDetails supervisor) Get the ports that are not blacklisted.Get all non-blacklisted slots in the cluster.getAssignableSlots
(SupervisorDetails supervisor) Return all non-blacklisted slots on this supervisor.static double
getAssignedMemoryForSlot
(Map<String, Object> topConf) Get heap memory usage for a worker's main process and logwriter process.int
getAssignedNumWorkers
(TopologyDetails topology) Get the number of workers assigned to a topology.getAssignmentById
(String topologyId) get the current assignment for the topology.Get all the assignments.getAvailablePorts
(SupervisorDetails supervisor) Return the available ports of this supervisor.Get the resources on the supervisor that are available to be scheduled.Get all the available worker slots in the cluster.getAvailableSlots
(SupervisorDetails supervisor) Return all the available slots on this supervisor.Get all of the hosts that are blacklisted.double
Get the total amount of CPU resources in cluster.Get the total amount of generic resources (excluding CPU and memory) in cluster.double
Get the total amount of memory resources in cluster.getConf()
Get the nimbus configuration.Map a supervisor to a given host.double
Get the component name to executor list for executors that need to be scheduled.Get the executor to component name map for executors that need to be scheduled.Get the network topography (rackId -> nodes in the rack).getNonBlacklistedAvailableSlots
(List<String> blacklistedSupervisorIds) Get all the available worker slots in the cluster, that are not blacklisted.getNonBlacklistedClusterAvailableResources
(Collection<String> blacklistedSupervisorIds) Get the resources in the cluster that are available for scheduling.double
getScheduledCpuForNode
(String nodeId) Get the total cpu currently scheduled on a node.double
getScheduledMemoryForNode
(String nodeId) Get the total memory currently scheduled on a node.Get all topology scheduler statuses.getSupervisorById
(String nodeId) Get a specific supervisor with thenodeId
.Get all the supervisors.getSupervisorsByHost
(String host) Get all the supervisors on the specifiedhost
.Get the amount of used and free resources on a supervisor.Get all of the topologies.Get the amount of resources used by topologies.getUnassignedExecutors
(TopologyDetails topology) get the unassigned executors of the topology.getUsedPorts
(SupervisorDetails supervisor) Get all the used ports of this supervisor.Get all currently occupied slots.getUsedSlotsByTopologyId
(String topologyId) get slots used by a topology.Get the resources for a given slot.Gets the reference to the full topology->worker resource map.boolean
isBlackListed
(String supervisorId) Check is a given supervisor is on a blacklisted host.boolean
isBlacklistedHost
(String host) Check if a given host is blacklisted.boolean
isSlotOccupied
(WorkerSlot slot) Check if a slot is occupied or not.boolean
needsScheduling
(TopologyDetails topology) Does the topology need scheduling.boolean
needsSchedulingRas
(TopologyDetails topology) LikeISchedulingState.needsScheduling(TopologyDetails)
but does not take into account the number of workers requested.Get all of the topologies that need scheduling.void
setAssignments
(Map<String, ? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions) Set assignments for cluster.void
setBlacklistedHosts
(Set<String> hosts) Set the list of hosts that are blacklisted.void
setGreyListedSupervisors
(Set<String> greyListedSupervisors) void
setNetworkTopography
(Map<String, List<String>> networkTopography) void
set scheduler status for a topology.void
setStatus
(TopologyDetails td, String statusMessage) set scheduler status for a topology.void
setStatusIfAbsent
(String topologyId, String statusMessage) void
setStatusMap
(Map<String, String> statusMap) set scheduler status map.void
Unassign everything for the given topology id.void
updateFrom
(Cluster other) Update the assignments and status from the other cluster.boolean
wouldFit
(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap) Would scheduling exec on ws fit?Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.storm.scheduler.ISchedulingState
getAssignedRacks, getHostToRack
-
Constructor Details
-
Cluster
public Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String, SupervisorDetails> supervisors, Map<String, ? extends SchedulerAssignment> assignments, Topologies topologies, Map<String, Object> conf) -
Cluster
Copy constructor. -
Cluster
Testing Constructor that takes an existing cluster and replaces the topologies in it.- Parameters:
src
- the original clustertopologies
- the new topolgoies to use
-
-
Method Details
-
getAssignedMemoryForSlot
Get heap memory usage for a worker's main process and logwriter process.- Parameters:
topConf
- - the topology config- Returns:
- the assigned memory (in MB)
-
assertValidTopologyForModification
Check if the given topology is allowed for modification right now. If not throw an IllegalArgumentException else go on.- Parameters:
topologyId
- the id of the topology to check
-
getTopologies
Description copied from interface:ISchedulingState
Get all of the topologies.- Specified by:
getTopologies
in interfaceISchedulingState
- Returns:
- all of the topologies that are a part of the cluster.
-
getBlacklistedHosts
Description copied from interface:ISchedulingState
Get all of the hosts that are blacklisted.- Specified by:
getBlacklistedHosts
in interfaceISchedulingState
- Returns:
- all of the hosts that are blacklisted
-
setBlacklistedHosts
Set the list of hosts that are blacklisted.- Parameters:
hosts
- the new hosts that are blacklisted.
-
blacklistHost
-
isBlackListed
Description copied from interface:ISchedulingState
Check is a given supervisor is on a blacklisted host.- Specified by:
isBlackListed
in interfaceISchedulingState
- Parameters:
supervisorId
- the id of the supervisor- Returns:
- true if it is else false
-
isBlacklistedHost
Description copied from interface:ISchedulingState
Check if a given host is blacklisted.- Specified by:
isBlacklistedHost
in interfaceISchedulingState
- Parameters:
host
- the name of the host- Returns:
- true if it is else false.
-
getHost
Description copied from interface:ISchedulingState
Map a supervisor to a given host.- Specified by:
getHost
in interfaceISchedulingState
- Parameters:
supervisorId
- the id of the supervisor- Returns:
- the actual host name the supervisor is on
-
needsSchedulingTopologies
Description copied from interface:ISchedulingState
Get all of the topologies that need scheduling.- Specified by:
needsSchedulingTopologies
in interfaceISchedulingState
- Returns:
- all of the topologies that are not fully scheduled.
-
needsScheduling
Description copied from interface:ISchedulingState
Does the topology need scheduling.A topology needs scheduling if one of the following conditions holds:
- Although the topology is assigned slots, but is squeezed. i.e. the topology is assigned less slots than desired.
- There are unassigned executors in this topology
- Specified by:
needsScheduling
in interfaceISchedulingState
-
needsSchedulingRas
Description copied from interface:ISchedulingState
LikeISchedulingState.needsScheduling(TopologyDetails)
but does not take into account the number of workers requested. This is because the number of workers is ignored in RAS- Specified by:
needsSchedulingRas
in interfaceISchedulingState
- Parameters:
topology
- the topology to check- Returns:
- true if the topology needs scheduling else false.
-
getNeedsSchedulingExecutorToComponents
Description copied from interface:ISchedulingState
Get the executor to component name map for executors that need to be scheduled.- Specified by:
getNeedsSchedulingExecutorToComponents
in interfaceISchedulingState
- Parameters:
topology
- the topology this is for- Returns:
- a executor -> component-id map which needs scheduling in this topology.
-
getNeedsSchedulingComponentToExecutors
public Map<String,List<ExecutorDetails>> getNeedsSchedulingComponentToExecutors(TopologyDetails topology) Description copied from interface:ISchedulingState
Get the component name to executor list for executors that need to be scheduled.- Specified by:
getNeedsSchedulingComponentToExecutors
in interfaceISchedulingState
- Parameters:
topology
- the topology this is for- Returns:
- a component-id -> executors map which needs scheduling in this topology.
-
getUsedPorts
Description copied from interface:ISchedulingState
Get all the used ports of this supervisor.- Specified by:
getUsedPorts
in interfaceISchedulingState
-
getAvailablePorts
Description copied from interface:ISchedulingState
Return the available ports of this supervisor.- Specified by:
getAvailablePorts
in interfaceISchedulingState
-
getAssignablePorts
Description copied from interface:ISchedulingState
Get the ports that are not blacklisted.- Specified by:
getAssignablePorts
in interfaceISchedulingState
- Parameters:
supervisor
- the supervisor- Returns:
- the ports that are not blacklisted
-
getNonBlacklistedAvailableSlots
Description copied from interface:ISchedulingState
Get all the available worker slots in the cluster, that are not blacklisted.- Specified by:
getNonBlacklistedAvailableSlots
in interfaceISchedulingState
- Parameters:
blacklistedSupervisorIds
- list of supervisor ids that should also be considered blacklisted.
-
getAvailableSlots
Description copied from interface:ISchedulingState
Get all the available worker slots in the cluster.- Specified by:
getAvailableSlots
in interfaceISchedulingState
-
getAvailableSlots
Description copied from interface:ISchedulingState
Return all the available slots on this supervisor.- Specified by:
getAvailableSlots
in interfaceISchedulingState
-
getAssignableSlots
Description copied from interface:ISchedulingState
Return all non-blacklisted slots on this supervisor.- Specified by:
getAssignableSlots
in interfaceISchedulingState
- Parameters:
supervisor
- the supervisor- Returns:
- the non-blacklisted slots
-
getAssignableSlots
Description copied from interface:ISchedulingState
Get all non-blacklisted slots in the cluster.- Specified by:
getAssignableSlots
in interfaceISchedulingState
-
getUnassignedExecutors
Description copied from interface:ISchedulingState
get the unassigned executors of the topology.- Specified by:
getUnassignedExecutors
in interfaceISchedulingState
- Parameters:
topology
- the topology to check- Returns:
- the unassigned executors of the topology.
-
getAssignedNumWorkers
Description copied from interface:ISchedulingState
Get the number of workers assigned to a topology.- Specified by:
getAssignedNumWorkers
in interfaceISchedulingState
- Parameters:
topology
- the topology this is for- Returns:
- the number of workers assigned to this topology.
-
getAvailableResources
Description copied from interface:ISchedulingState
Get the resources on the supervisor that are available to be scheduled.- Specified by:
getAvailableResources
in interfaceISchedulingState
- Parameters:
sd
- the supervisor.- Returns:
- the resources available to be scheduled.
-
wouldFit
public boolean wouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap) Description copied from interface:ISchedulingState
Would scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.- Specified by:
wouldFit
in interfaceISchedulingState
- Parameters:
ws
- the slot to put it inexec
- the executor to investigatetd
- the topology detains for this executorresourcesAvailable
- all the available resourcesmaxHeap
- the maximum heap size for ws- Returns:
- true it fits else false
-
assign
Assign the slot to the executors for this topology.- Throws:
RuntimeException
- if the specified slot is already occupied.
-
assign
Assign everything for the given topology.- Parameters:
assignment
- the new assignment to make
-
freeSlot
Free the specified slot.- Parameters:
slot
- the slot to free
-
freeSlots
free the slots.- Parameters:
slots
- multiple slots to free
-
isSlotOccupied
Description copied from interface:ISchedulingState
Check if a slot is occupied or not.- Specified by:
isSlotOccupied
in interfaceISchedulingState
- Parameters:
slot
- the slot be to checked.- Returns:
- true if the specified slot is occupied.
-
getAssignmentById
Description copied from interface:ISchedulingState
get the current assignment for the topology.- Specified by:
getAssignmentById
in interfaceISchedulingState
-
getUsedSlotsByTopologyId
Description copied from interface:ISchedulingState
get slots used by a topology.- Specified by:
getUsedSlotsByTopologyId
in interfaceISchedulingState
-
getSupervisorById
Description copied from interface:ISchedulingState
Get a specific supervisor with thenodeId
.- Specified by:
getSupervisorById
in interfaceISchedulingState
-
getUsedSlots
Description copied from interface:ISchedulingState
Get all currently occupied slots.- Specified by:
getUsedSlots
in interfaceISchedulingState
-
getSupervisorsByHost
Description copied from interface:ISchedulingState
Get all the supervisors on the specifiedhost
.- Specified by:
getSupervisorsByHost
in interfaceISchedulingState
- Parameters:
host
- hostname of the supervisor- Returns:
- the
SupervisorDetails
object.
-
getAssignments
Description copied from interface:ISchedulingState
Get all the assignments.- Specified by:
getAssignments
in interfaceISchedulingState
-
setAssignments
public void setAssignments(Map<String, ? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions) Set assignments for cluster. -
getSupervisors
Description copied from interface:ISchedulingState
Get all the supervisors.- Specified by:
getSupervisors
in interfaceISchedulingState
-
getNonBlacklistedClusterAvailableResources
public NormalizedResourceOffer getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds) Description copied from interface:ISchedulingState
Get the resources in the cluster that are available for scheduling.- Specified by:
getNonBlacklistedClusterAvailableResources
in interfaceISchedulingState
- Parameters:
blacklistedSupervisorIds
- other ids that are tentatively blacklisted.
-
getClusterTotalCpuResource
public double getClusterTotalCpuResource()Description copied from interface:ISchedulingState
Get the total amount of CPU resources in cluster.- Specified by:
getClusterTotalCpuResource
in interfaceISchedulingState
-
getClusterTotalMemoryResource
public double getClusterTotalMemoryResource()Description copied from interface:ISchedulingState
Get the total amount of memory resources in cluster.- Specified by:
getClusterTotalMemoryResource
in interfaceISchedulingState
-
getClusterTotalGenericResources
Description copied from interface:ISchedulingState
Get the total amount of generic resources (excluding CPU and memory) in cluster.- Specified by:
getClusterTotalGenericResources
in interfaceISchedulingState
-
getNetworkTopography
Description copied from interface:ISchedulingState
Get the network topography (rackId -> nodes in the rack).- Specified by:
getNetworkTopography
in interfaceISchedulingState
-
setNetworkTopography
-
setStatus
set scheduler status for a topology. -
setStatus
set scheduler status for a topology. -
setStatusIfAbsent
-
getStatusMap
Description copied from interface:ISchedulingState
Get all topology scheduler statuses.- Specified by:
getStatusMap
in interfaceISchedulingState
-
setStatusMap
set scheduler status map. -
getStatus
-
getTopologyResourcesMap
Description copied from interface:ISchedulingState
Get the amount of resources used by topologies. Used for displaying resource information on the UI.- Specified by:
getTopologyResourcesMap
in interfaceISchedulingState
- Returns:
- a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
-
getSupervisorsResourcesMap
Description copied from interface:ISchedulingState
Get the amount of used and free resources on a supervisor. Used for displaying resource information on the UI- Specified by:
getSupervisorsResourcesMap
in interfaceISchedulingState
- Returns:
- a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}
-
getWorkerResourcesMap
Description copied from interface:ISchedulingState
Gets the reference to the full topology->worker resource map.- Specified by:
getWorkerResourcesMap
in interfaceISchedulingState
- Returns:
- map of topology -> map of worker slot ->resources for that worker
-
getWorkerResources
Description copied from interface:ISchedulingState
Get the resources for a given slot.- Specified by:
getWorkerResources
in interfaceISchedulingState
- Parameters:
ws
- the slot- Returns:
- the resources currently assigned
-
getResourceMetrics
-
getAllScheduledResourcesForNode
Description copied from interface:ISchedulingState
Get all scheduled resources for node.- Specified by:
getAllScheduledResourcesForNode
in interfaceISchedulingState
-
getScheduledMemoryForNode
Description copied from interface:ISchedulingState
Get the total memory currently scheduled on a node.- Specified by:
getScheduledMemoryForNode
in interfaceISchedulingState
- Parameters:
nodeId
- the id of the node- Returns:
- the total memory currently scheduled on the node
-
getScheduledCpuForNode
Description copied from interface:ISchedulingState
Get the total cpu currently scheduled on a node.- Specified by:
getScheduledCpuForNode
in interfaceISchedulingState
- Parameters:
nodeId
- the id of the node- Returns:
- the total cpu currently scheduled on the node
-
getINimbus
-
getConf
Description copied from interface:ISchedulingState
Get the nimbus configuration.- Specified by:
getConf
in interfaceISchedulingState
-
unassign
Unassign everything for the given topology id.- Parameters:
topoId
- the is of the topology to unassign
-
updateFrom
Update the assignments and status from the other cluster.- Parameters:
other
- the cluster to get the assignments and status from
-
getMinWorkerCpu
public double getMinWorkerCpu() -
getGreyListedSupervisors
-
setGreyListedSupervisors
-