org.apache.storm.container

Class DefaultResourceIsolationManager

java.lang.Object

org.apache.storm.container.DefaultResourceIsolationManager

All Implemented Interfaces:

ResourceIsolationInterface

Direct Known Subclasses:

CgroupManager

public class DefaultResourceIsolationManager
extends Object
implements ResourceIsolationInterface

This is the default class to manage worker processes, including launching, killing, profiling and etc.

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,Object>	conf
protected boolean	runAsUser

Constructor Summary

Constructors

Constructor and Description
DefaultResourceIsolationManager()

Method Summary

Modifier and Type	Method and Description
boolean	areAllProcessesDead(String user, String workerId) Check if all the processes are dead.
void	cleanup(String user, String workerId, int port) This function will be called when the worker needs to shutdown.
void	forceKill(String user, String workerId) Kill the given worker forcefully.
protected Set<Long>	getAllPids(String workerId) Get all the pids that are a part of the container.
long	getMemoryUsage(String user, String workerId, int port) Get the current memory usage of the a given worker.
long	getSystemFreeMemoryMb() Get the amount of free memory in MB.
boolean	isResourceManaged() This class doesn’t really manage resources.
void	kill(String user, String workerId) Kill the given worker.
void	launchWorkerProcess(String user, String topologyId, Map<String,Object> topoConf, int port, String workerId, List<String> command, Map<String,String> env, String logPrefix, ExitCodeCallback processExitCallback, File targetDir) After reserving resources for the worker (i.e.
void	prepare(Map<String,Object> conf) Called when starting up.
void	reserveResourcesForWorker(String workerId, Integer workerMemory, Integer workerCpu, String numaId) This function should be used prior to starting the worker to reserve resources for the worker.
boolean	runProfilingCommand(String user, String workerId, List<String> command, Map<String,String> env, String logPrefix, File targetDir) Run profiling command.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

conf

protected Map<String,Object> conf

runAsUser

protected boolean runAsUser

Constructor Detail

DefaultResourceIsolationManager

public DefaultResourceIsolationManager()

Method Detail

prepare

public void prepare(Map<String,Object> conf)
             throws IOException

Description copied from interface: ResourceIsolationInterface

Called when starting up.

Specified by:

prepare in interface ResourceIsolationInterface

Parameters:

conf - the cluster config

Throws:

IOException - on any error.

reserveResourcesForWorker

public void reserveResourcesForWorker(String workerId,
                                      Integer workerMemory,
                                      Integer workerCpu,
                                      String numaId)

Description copied from interface: ResourceIsolationInterface

This function should be used prior to starting the worker to reserve resources for the worker.

Specified by:

reserveResourcesForWorker in interface ResourceIsolationInterface

Parameters:

workerId - worker id of the worker to start

workerMemory - the amount of memory for the worker or null if not enforced

workerCpu - the amount of cpu for the worker or null if not enforced

numaId - NUMA zone if applicable the worker should be bound to

cleanup

public void cleanup(String user,
                    String workerId,
                    int port)
             throws IOException

Description copied from interface: ResourceIsolationInterface

This function will be called when the worker needs to shutdown. This function should include logic to clean up after a worker is shutdown.

Specified by:

cleanup in interface ResourceIsolationInterface

Parameters:

user - the user of the worker

workerId - worker id to shutdown and clean up after

port - the port of the worker

Throws:

IOException

launchWorkerProcess

public void launchWorkerProcess(String user,
                                String topologyId,
                                Map<String,Object> topoConf,
                                int port,
                                String workerId,
                                List<String> command,
                                Map<String,String> env,
                                String logPrefix,
                                ExitCodeCallback processExitCallback,
                                File targetDir)
                         throws IOException

Description copied from interface: ResourceIsolationInterface

After reserving resources for the worker (i.e. calling reserveResourcesForWorker), this function can be used to launch worker process.

Specified by:

launchWorkerProcess in interface ResourceIsolationInterface

Parameters:

user - the user who runs the command as

topologyId - the Id of the topology

topoConf - the topology configuration

port - the port where the worker is on

workerId - the Id of the worker

command - the command to run

env - the environment to run the command

logPrefix - the prefix to include in the logs

processExitCallback - a callback for when the process exits

targetDir - the working directory to run the command in

Throws:

IOException - on I/O exception

getMemoryUsage

public long getMemoryUsage(String user,
                           String workerId,
                           int port)
                    throws IOException

Description copied from interface: ResourceIsolationInterface

Get the current memory usage of the a given worker.

Specified by:

getMemoryUsage in interface ResourceIsolationInterface

Parameters:

user - the user that the worker is running as

workerId - the id of the worker

port - the port of the worker

Returns:

the amount of memory the worker is using in bytes or -1 if not supported

Throws:

IOException - on I/O exception

getSystemFreeMemoryMb

public long getSystemFreeMemoryMb()
                           throws IOException

Description copied from interface: ResourceIsolationInterface

Get the amount of free memory in MB. This might not be the entire box, it might be within a parent resource group.

Specified by:

getSystemFreeMemoryMb in interface ResourceIsolationInterface

Returns:

The amount of memory in MB that are free on the system.

Throws:

IOException - on I/O exception

kill

public void kill(String user,
                 String workerId)
          throws IOException

Description copied from interface: ResourceIsolationInterface

Kill the given worker.

Specified by:

kill in interface ResourceIsolationInterface

Parameters:

user - the user that the worker is running as

workerId - the id of the worker to kill

Throws:

IOException - on I/O exception

forceKill

public void forceKill(String user,
                      String workerId)
               throws IOException

Description copied from interface: ResourceIsolationInterface

Kill the given worker forcefully.

Specified by:

forceKill in interface ResourceIsolationInterface

Parameters:

user - the user that the worker is running as

workerId - the id of the worker to kill

Throws:

IOException - on I/O exception

getAllPids

protected Set<Long> getAllPids(String workerId)
                        throws IOException

Get all the pids that are a part of the container.

Returns:

all of the pids that are a part of this container

Throws:

IOException

areAllProcessesDead

public boolean areAllProcessesDead(String user,
                                   String workerId)
                            throws IOException

Description copied from interface: ResourceIsolationInterface

Check if all the processes are dead.

Specified by:

areAllProcessesDead in interface ResourceIsolationInterface

Parameters:

user - the user that the processes are running as

workerId - the id of the worker to kill

Returns:

true if all the processed are dead; false otherwise

Throws:

IOException - on I/O exception

runProfilingCommand

public boolean runProfilingCommand(String user,
                                   String workerId,
                                   List<String> command,
                                   Map<String,String> env,
                                   String logPrefix,
                                   File targetDir)
                            throws IOException,
                                   InterruptedException

Description copied from interface: ResourceIsolationInterface

Run profiling command.

Specified by:

runProfilingCommand in interface ResourceIsolationInterface

Parameters:

user - the user that the worker is running as

workerId - the id of the worker

command - the command to run

env - the environment to run the command

logPrefix - the prefix to include in the logs

targetDir - the working directory to run the command in

Returns:

true if succeeds; false otherwise

Throws:

IOException - on I/O exception

InterruptedException - if interrupted

isResourceManaged

public boolean isResourceManaged()

This class doesn’t really manage resources.

Specified by:

isResourceManaged in interface ResourceIsolationInterface

Returns:

false