org.apache.storm.scheduler.resource.strategies.scheduling

Class BaseResourceAwareStrategy

java.lang.Object

org.apache.storm.scheduler.resource.strategies.scheduling.BaseResourceAwareStrategy

All Implemented Interfaces:

IStrategy

Direct Known Subclasses:

ConstraintSolverStrategy, DefaultResourceAwareStrategy, GenericResourceAwareStrategy

public abstract class BaseResourceAwareStrategy
extends Object
implements IStrategy

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BaseResourceAwareStrategy.AllResources a class to contain individual object resources as well as cumulative stats.
protected static interface	BaseResourceAwareStrategy.ExistingScheduleFunc interface for calculating the number of existing executors scheduled on a object (rack or node).
protected static class	BaseResourceAwareStrategy.ObjectResources class to keep track of resources on a rack or node.

Field Summary

Fields

Modifier and Type	Field and Description
protected Cluster	cluster
protected RasNodes	nodes

Constructor Summary

Constructors

Constructor and Description
BaseResourceAwareStrategy()

Method Summary

Modifier and Type	Method and Description
protected WorkerSlot	findWorkerForExec(ExecutorDetails exec, TopologyDetails td, Iterable<String> sortedNodes) Find a worker to schedule executor exec on.
protected List<Component>	getSpouts(TopologyDetails td) Get a list of all the spouts in the topology.
List<RasNode>	hostnameToNodes(String hostname) hostname to Ids.
RasNode	idToNode(String id) Find RASNode for specified node id.
protected List<String>	makeHostToNodeIds(List<String> hosts)
protected SchedulingResult	mkNotEnoughResources(TopologyDetails td)
protected String	nodeToRack(RasNode node) Get the rack on which a node is a part of.
protected List<ExecutorDetails>	orderExecutors(TopologyDetails td, Collection<ExecutorDetails> unassignedExecutors)
void	prepare(Map<String,Object> config) Prepare the Strategy for scheduling.
protected boolean	scheduleExecutor(ExecutorDetails exec, TopologyDetails td, Collection<ExecutorDetails> scheduledTasks, Iterable<String> sortedNodes) Schedule executor exec from topology td.
protected Iterable<String>	sortAllNodes(TopologyDetails td, ExecutorDetails exec, List<String> favoredNodeIds, List<String> unFavoredNodeIds)
protected TreeSet<BaseResourceAwareStrategy.ObjectResources>	sortNodes(List<RasNode> availNodes, ExecutorDetails exec, TopologyDetails topologyDetails, String rackId, Map<String,AtomicInteger> scheduledCount) Nodes are sorted by two criteria.
protected abstract TreeSet<BaseResourceAwareStrategy.ObjectResources>	sortObjectResources(BaseResourceAwareStrategy.AllResources allResources, ExecutorDetails exec, TopologyDetails topologyDetails, BaseResourceAwareStrategy.ExistingScheduleFunc existingScheduleFunc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.scheduler.resource.strategies.scheduling.IStrategy

schedule

Field Detail

cluster

protected Cluster cluster

nodes

protected RasNodes nodes

Constructor Detail

BaseResourceAwareStrategy

public BaseResourceAwareStrategy()

Method Detail

prepare

public void prepare(Map<String,Object> config)

Description copied from interface: IStrategy

Prepare the Strategy for scheduling.

Specified by:

prepare in interface IStrategy

Parameters:

config - the cluster configuration

mkNotEnoughResources

protected SchedulingResult mkNotEnoughResources(TopologyDetails td)

scheduleExecutor

protected boolean scheduleExecutor(ExecutorDetails exec,
                                   TopologyDetails td,
                                   Collection<ExecutorDetails> scheduledTasks,
                                   Iterable<String> sortedNodes)

Schedule executor exec from topology td.

Parameters:

exec - the executor to schedule

td - the topology executor exec is a part of

scheduledTasks - executors that have been scheduled

Returns:

true if scheduled successfully, else false.

sortObjectResources

protected abstract TreeSet<BaseResourceAwareStrategy.ObjectResources> sortObjectResources(BaseResourceAwareStrategy.AllResources allResources,
                                                                                          ExecutorDetails exec,
                                                                                          TopologyDetails topologyDetails,
                                                                                          BaseResourceAwareStrategy.ExistingScheduleFunc existingScheduleFunc)

findWorkerForExec

protected WorkerSlot findWorkerForExec(ExecutorDetails exec,
                                       TopologyDetails td,
                                       Iterable<String> sortedNodes)

Find a worker to schedule executor exec on.

Parameters:

exec - the executor to schedule

td - the topology that the executor is a part of

Returns:

a worker to assign exec on. Returns null if a worker cannot be successfully found in cluster

sortNodes

protected TreeSet<BaseResourceAwareStrategy.ObjectResources> sortNodes(List<RasNode> availNodes,
                                                                       ExecutorDetails exec,
                                                                       TopologyDetails topologyDetails,
                                                                       String rackId,
                                                                       Map<String,AtomicInteger> scheduledCount)

Nodes are sorted by two criteria.

1) the number executors of the topology that needs to be scheduled is already on the node in descending order. The reasoning to sort based on criterion 1 is so we schedule the rest of a topology on the same node as the existing executors of the topology.

2) the subordinate/subservient resource availability percentage of a node in descending order We calculate the resource availability percentage by dividing the resource availability that have exhausted or little of one of the resources mentioned above will be ranked after on the node by the resource availability of the entire rack By doing this calculation, nodes nodes that have more balanced resource availability. So we will be less likely to pick a node that have a lot of one resource but a low amount of another.

Parameters:

availNodes - a list of all the nodes we want to sort

rackId - the rack id availNodes are a part of

Returns:

a sorted list of nodes.

makeHostToNodeIds

protected List<String> makeHostToNodeIds(List<String> hosts)

sortAllNodes

protected Iterable<String> sortAllNodes(TopologyDetails td,
                                        ExecutorDetails exec,
                                        List<String> favoredNodeIds,
                                        List<String> unFavoredNodeIds)

nodeToRack

protected String nodeToRack(RasNode node)

Get the rack on which a node is a part of.

Parameters:

node - the node to find out which rack its on

Returns:

the rack id

orderExecutors

protected List<ExecutorDetails> orderExecutors(TopologyDetails td,
                                               Collection<ExecutorDetails> unassignedExecutors)

getSpouts

protected List<Component> getSpouts(TopologyDetails td)

Get a list of all the spouts in the topology.

Parameters:

td - topology to get spouts from

Returns:

a list of spouts

hostnameToNodes

public List<RasNode> hostnameToNodes(String hostname)

hostname to Ids.

Parameters:

hostname - the hostname.

Returns:

the ids n that node.

idToNode

public RasNode idToNode(String id)

Find RASNode for specified node id.

Parameters:

id - the node/supervisor id to lookup

Returns:

a RASNode object