org.apache.storm.scheduler.resource.strategies.scheduling

Class BaseResourceAwareStrategy

java.lang.Object

org.apache.storm.scheduler.resource.strategies.scheduling.BaseResourceAwareStrategy

All Implemented Interfaces:

IStrategy

Direct Known Subclasses:

ConstraintSolverStrategy, DefaultResourceAwareStrategy, GenericResourceAwareStrategy

public abstract class BaseResourceAwareStrategy
extends Object
implements IStrategy

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BaseResourceAwareStrategy.AllResources a class to contain individual object resources as well as cumulative stats.
protected static interface	BaseResourceAwareStrategy.ExistingScheduleFunc interface for calculating the number of existing executors scheduled on a object (rack or node).
protected static class	BaseResourceAwareStrategy.ObjectResources class to keep track of resources on a rack or node.

Field Summary

Fields

Modifier and Type	Field and Description
protected Cluster	cluster
protected RAS_Nodes	nodes

Constructor Summary

Constructors

Constructor and Description
BaseResourceAwareStrategy()

Method Summary

Modifier and Type	Method and Description
protected WorkerSlot	findWorkerForExec(ExecutorDetails exec, TopologyDetails td, Iterable<String> sortedNodes) Find a worker to schedule executor exec on.
protected List<Component>	getSpouts(TopologyDetails td) Get a list of all the spouts in the topology.
List<RAS_Node>	hostnameToNodes(String hostname) hostname to Ids.
RAS_Node	idToNode(String id) Find RAS_Node for specified node id.
protected List<String>	makeHostToNodeIds(List<String> hosts)
protected SchedulingResult	mkNotEnoughResources(TopologyDetails td)
protected String	nodeToRack(RAS_Node node) Get the rack on which a node is a part of.
protected List<ExecutorDetails>	orderExecutors(TopologyDetails td, Collection<ExecutorDetails> unassignedExecutors) Order executors based on how many in and out connections it will potentially need to make, in descending order.
void	prepare(Map<String,Object> config) Prepare the Strategy for scheduling.
protected boolean	scheduleExecutor(ExecutorDetails exec, TopologyDetails td, Collection<ExecutorDetails> scheduledTasks, Iterable<String> sortedNodes) Schedule executor exec from topology td.
protected Iterable<String>	sortAllNodes(TopologyDetails td, ExecutorDetails exec, List<String> favoredNodeIds, List<String> unFavoredNodeIds)
protected TreeSet<BaseResourceAwareStrategy.ObjectResources>	sortNodes(List<RAS_Node> availNodes, ExecutorDetails exec, TopologyDetails topologyDetails, String rackId, Map<String,AtomicInteger> scheduledCount) Nodes are sorted by two criteria.
protected abstract TreeSet<BaseResourceAwareStrategy.ObjectResources>	sortObjectResources(BaseResourceAwareStrategy.AllResources allResources, ExecutorDetails exec, TopologyDetails topologyDetails, BaseResourceAwareStrategy.ExistingScheduleFunc existingScheduleFunc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.scheduler.resource.strategies.scheduling.IStrategy

schedule

Field Detail

cluster

protected Cluster cluster

nodes

protected RAS_Nodes nodes

Constructor Detail

BaseResourceAwareStrategy

public BaseResourceAwareStrategy()

Method Detail

prepare

public void prepare(Map<String,Object> config)

Description copied from interface: IStrategy

Prepare the Strategy for scheduling.

Specified by:

prepare in interface IStrategy

Parameters:

config - the cluster configuration

mkNotEnoughResources

protected SchedulingResult mkNotEnoughResources(TopologyDetails td)

scheduleExecutor

protected boolean scheduleExecutor(ExecutorDetails exec,
                                   TopologyDetails td,
                                   Collection<ExecutorDetails> scheduledTasks,
                                   Iterable<String> sortedNodes)

Schedule executor exec from topology td.

Parameters:

exec - the executor to schedule

td - the topology executor exec is a part of

scheduledTasks - executors that have been scheduled

Returns:

true if scheduled successfully, else false.

sortObjectResources

protected abstract TreeSet<BaseResourceAwareStrategy.ObjectResources> sortObjectResources(BaseResourceAwareStrategy.AllResources allResources,
                                                                                          ExecutorDetails exec,
                                                                                          TopologyDetails topologyDetails,
                                                                                          BaseResourceAwareStrategy.ExistingScheduleFunc existingScheduleFunc)

findWorkerForExec

protected WorkerSlot findWorkerForExec(ExecutorDetails exec,
                                       TopologyDetails td,
                                       Iterable<String> sortedNodes)

Find a worker to schedule executor exec on.

Parameters:

exec - the executor to schedule

td - the topology that the executor is a part of

Returns:

a worker to assign exec on. Returns null if a worker cannot be successfully found in cluster

sortNodes

protected TreeSet<BaseResourceAwareStrategy.ObjectResources> sortNodes(List<RAS_Node> availNodes,
                                                                       ExecutorDetails exec,
                                                                       TopologyDetails topologyDetails,
                                                                       String rackId,
                                                                       Map<String,AtomicInteger> scheduledCount)

Nodes are sorted by two criteria.

1) the number executors of the topology that needs to be scheduled is already on the node in descending order. The reasoning to sort based on criterion 1 is so we schedule the rest of a topology on the same node as the existing executors of the topology.

2) the subordinate/subservient resource availability percentage of a node in descending order We calculate the resource availability percentage by dividing the resource availability that have exhausted or little of one of the resources mentioned above will be ranked after on the node by the resource availability of the entire rack By doing this calculation, nodes nodes that have more balanced resource availability. So we will be less likely to pick a node that have a lot of one resource but a low amount of another.

Parameters:

availNodes - a list of all the nodes we want to sort

rackId - the rack id availNodes are a part of

Returns:

a sorted list of nodes.

makeHostToNodeIds

protected List<String> makeHostToNodeIds(List<String> hosts)

sortAllNodes

protected Iterable<String> sortAllNodes(TopologyDetails td,
                                        ExecutorDetails exec,
                                        List<String> favoredNodeIds,
                                        List<String> unFavoredNodeIds)

nodeToRack

protected String nodeToRack(RAS_Node node)

Get the rack on which a node is a part of.

Parameters:

node - the node to find out which rack its on

Returns:

the rack id

orderExecutors

protected List<ExecutorDetails> orderExecutors(TopologyDetails td,
                                               Collection<ExecutorDetails> unassignedExecutors)

Order executors based on how many in and out connections it will potentially need to make, in descending order. First order components by the number of in and out connections it will have. Then iterate through the sorted list of components. For each component sort the neighbors of that component by how many connections it will have to make with that component. Add an executor from this component and then from each neighboring component in sorted order. Do this until there is nothing left to schedule.

Parameters:

td - The topology the executors belong to

unassignedExecutors - a collection of unassigned executors that need to be unassigned. Should only try to assign executors from this list

Returns:

a list of executors in sorted order

getSpouts

protected List<Component> getSpouts(TopologyDetails td)

Get a list of all the spouts in the topology.

Parameters:

td - topology to get spouts from

Returns:

a list of spouts

hostnameToNodes

public List<RAS_Node> hostnameToNodes(String hostname)

hostname to Ids.

Parameters:

hostname - the hostname.

Returns:

the ids n that node.

idToNode

public RAS_Node idToNode(String id)

Find RAS_Node for specified node id.

Parameters:

id - the node/supervisor id to lookup

Returns:

a RAS_Node object