org.apache.storm.localizer

Class Localizer

java.lang.Object

org.apache.storm.localizer.Localizer

public class Localizer
extends Object

Class to download and manage files from the blobstore. It uses an LRU cache to determine which files to keep so they can be reused and which files to delete.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ARCHIVESDIR
static String	FILECACHE
static String	FILESDIR
static org.slf4j.Logger	LOG
static String	USERCACHE

Constructor Summary

Constructors

Constructor and Description
Localizer(Map<String,Object> conf, String baseDir)

Method Summary

Modifier and Type	Method and Description
protected void	addLocalizedResourceInDir(String dir, LocalizedResourceSet lrsrcSet, boolean uncompress)
void	addReferences(List<LocalResource> localresource, String user, String topo)
LocalizedResource	getBlob(LocalResource localResource, String user, String topo, File userFileDir) This function either returns the blob in the existing cache or if it doesn’t exist in the cache, it will download the blob and will block until the download is complete.
List<LocalizedResource>	getBlobs(List<LocalResource> localResources, String user, String topo, File userFileDir) This function either returns the blobs in the existing cache or if they don’t exist in the cache, it downloads them in parallel (up to SUPERVISOR_BLOBSTORE_DOWNLOAD_THREAD_COUNT) and will block until all of them have been downloaded
protected File	getCacheDirForArchives(File dir)
protected File	getCacheDirForFiles(File dir)
protected ClientBlobStore	getClientBlobStore()
protected File	getLocalUserDir(String userName)
File	getLocalUserFileCacheDir(String userName)
protected File	getUserCacheDir()
void	handleCacheCleanup()
protected boolean	isLocalizedResourceDownloaded(LocalizedResource lrsrc)
protected boolean	isLocalizedResourceUpToDate(LocalizedResource lrsrc, ClientBlobStore blobstore)
protected File[]	readCurrentBlobs(String location)
protected File[]	readDirContents(String location)
protected void	reconstructLocalizedResources()
void	removeBlobReference(String key, String user, String topo, boolean uncompress)
void	setBlobPermissions(Map conf, String user, String path)
protected void	setTargetCacheSize(long size)
void	shutdown()
void	startCleaner()
List<LocalizedResource>	updateBlobs(List<LocalResource> localResources, String user) This function updates blobs on the supervisor.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

USERCACHE

public static final String USERCACHE

See Also:

Constant Field Values

FILECACHE

public static final String FILECACHE

See Also:

Constant Field Values

FILESDIR

public static final String FILESDIR

See Also:

Constant Field Values

ARCHIVESDIR

public static final String ARCHIVESDIR

See Also:

Constant Field Values

Constructor Detail

Localizer

public Localizer(Map<String,Object> conf,
                 String baseDir)

Method Detail

setTargetCacheSize

protected void setTargetCacheSize(long size)

startCleaner

public void startCleaner()

shutdown

public void shutdown()

getUserCacheDir

protected File getUserCacheDir()

getLocalUserDir

protected File getLocalUserDir(String userName)

getLocalUserFileCacheDir

public File getLocalUserFileCacheDir(String userName)

getCacheDirForFiles

protected File getCacheDirForFiles(File dir)

getCacheDirForArchives

protected File getCacheDirForArchives(File dir)

addLocalizedResourceInDir

protected void addLocalizedResourceInDir(String dir,
                                         LocalizedResourceSet lrsrcSet,
                                         boolean uncompress)

readDirContents

protected File[] readDirContents(String location)

readCurrentBlobs

protected File[] readCurrentBlobs(String location)

reconstructLocalizedResources

protected void reconstructLocalizedResources()

removeBlobReference

public void removeBlobReference(String key,
                                String user,
                                String topo,
                                boolean uncompress)
                         throws AuthorizationException,
                                KeyNotFoundException

Throws:

AuthorizationException

KeyNotFoundException

addReferences

public void addReferences(List<LocalResource> localresource,
                          String user,
                          String topo)

getBlob

public LocalizedResource getBlob(LocalResource localResource,
                                 String user,
                                 String topo,
                                 File userFileDir)
                          throws AuthorizationException,
                                 KeyNotFoundException,
                                 IOException

This function either returns the blob in the existing cache or if it doesn’t exist in the cache, it will download the blob and will block until the download is complete.

Throws:

AuthorizationException

KeyNotFoundException

IOException

isLocalizedResourceDownloaded

protected boolean isLocalizedResourceDownloaded(LocalizedResource lrsrc)

isLocalizedResourceUpToDate

protected boolean isLocalizedResourceUpToDate(LocalizedResource lrsrc,
                                              ClientBlobStore blobstore)
                                       throws AuthorizationException,
                                              KeyNotFoundException

Throws:

AuthorizationException

KeyNotFoundException

getClientBlobStore

protected ClientBlobStore getClientBlobStore()

updateBlobs

public List<LocalizedResource> updateBlobs(List<LocalResource> localResources,
                                           String user)
                                    throws AuthorizationException,
                                           KeyNotFoundException,
                                           IOException

This function updates blobs on the supervisor. It uses a separate thread pool and runs asynchronously of the download and delete.

Throws:

AuthorizationException

KeyNotFoundException

IOException

getBlobs

public List<LocalizedResource> getBlobs(List<LocalResource> localResources,
                                        String user,
                                        String topo,
                                        File userFileDir)
                                 throws AuthorizationException,
                                        KeyNotFoundException,
                                        IOException

This function either returns the blobs in the existing cache or if they don’t exist in the cache, it downloads them in parallel (up to SUPERVISOR_BLOBSTORE_DOWNLOAD_THREAD_COUNT) and will block until all of them have been downloaded

Throws:

AuthorizationException

KeyNotFoundException

IOException

setBlobPermissions

public void setBlobPermissions(Map conf,
                               String user,
                               String path)
                        throws IOException

Throws:

IOException

handleCacheCleanup

public void handleCacheCleanup()