org.apache.storm.blobstore

Class BlobStore

java.lang.Object

org.apache.storm.blobstore.BlobStore

All Implemented Interfaces:

Shutdownable

Direct Known Subclasses:

HdfsBlobStore, LocalFsBlobStore

public abstract class BlobStore
extends Object
implements Shutdownable

Provides a way to store blobs that can be downloaded. Blobs must be able to be uploaded and listed from Nimbus, and downloaded from the Supervisors. It is a key value based store. Key being a string and value being the blob data.

ACL checking must take place against the provided subject. If the blob store does not support Security it must validate that all ACLs set are always WORLD, everything.

The users can upload their blobs through the blob store command line. The command line also allows us to update and delete blobs.

Modifying the replication factor only works for HdfsBlobStore as for the LocalFsBlobStore the replication is dependent on the number of Nimbodes available.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	BlobStore.BlobStoreFileInputStream Input stream implementation used for writing both the metadata containing the acl information and the blob data.
protected class	BlobStore.BlobStoreFileOutputStream Output stream implementation used for reading the metadata and data information.
static class	BlobStore.KeyTranslationIterator Blob store implements its own version of iterator to list the blobs

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	BASE_BLOBS_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
BlobStore()

Method Summary

Modifier and Type	Method and Description
void	createBlob(String key, byte[] data, SettableBlobMeta meta, Subject who) Wrapper called to create the blob which contains the byte data
void	createBlob(String key, InputStream in, SettableBlobMeta meta, Subject who) Wrapper called to create the blob which contains the byte data
abstract AtomicOutputStream	createBlob(String key, SettableBlobMeta meta, Subject who) Creates the blob.
abstract void	deleteBlob(String key, Subject who) Deletes the blob data and metadata.
<R> Set<R>	filterAndListKeys(KeyFilter<R> filter) Filters keys based on the KeyFilter passed as the argument.
abstract InputStreamWithMeta	getBlob(String key, Subject who) Gets the InputStream to read the blob details
abstract ReadableBlobMeta	getBlobMeta(String key, Subject who) Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.
abstract int	getBlobReplication(String key, Subject who) Gets the replication factor of the blob.
abstract Iterator<String>	listKeys() Returns an iterator with all the list of keys currently available on the blob store.
abstract void	prepare(Map conf, String baseDir, NimbusInfo nimbusInfo) Allows us to initialize the blob store
byte[]	readBlob(String key, Subject who) Wrapper around readBlobTo which returns a ByteArray output stream.
void	readBlobTo(String key, OutputStream out, Subject who) Reads the blob from the blob store and writes it into the output stream.
abstract void	setBlobMeta(String key, SettableBlobMeta meta, Subject who) Sets the metadata with renewed acls for the blob.
abstract AtomicOutputStream	updateBlob(String key, Subject who) Updates the blob data.
abstract int	updateBlobReplication(String key, int replication, Subject who) Modifies the replication factor of the blob.
static void	validateKey(String key) Validates key checking for potentially harmful patterns

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.daemon.Shutdownable

shutdown

Field Detail

BASE_BLOBS_DIR_NAME

protected static final String BASE_BLOBS_DIR_NAME

See Also:

Constant Field Values

Constructor Detail

BlobStore

public BlobStore()

Method Detail

prepare

public abstract void prepare(Map conf,
                             String baseDir,
                             NimbusInfo nimbusInfo)

Allows us to initialize the blob store

Parameters:

conf - The storm configuration

baseDir - The directory path to store the blobs

nimbusInfo - Contains the nimbus host, port and leadership information.

createBlob

public abstract AtomicOutputStream createBlob(String key,
                                              SettableBlobMeta meta,
                                              Subject who)
                                       throws AuthorizationException,
                                              KeyAlreadyExistsException

Creates the blob.

Parameters:

key - Key for the blob.

meta - Metadata which contains the acls information

who - Is the subject creating the blob.

Returns:

AtomicOutputStream returns a stream into which the data can be written.

Throws:

AuthorizationException

KeyAlreadyExistsException

updateBlob

public abstract AtomicOutputStream updateBlob(String key,
                                              Subject who)
                                       throws AuthorizationException,
                                              KeyNotFoundException

Updates the blob data.

Parameters:

key - Key for the blob.

who - Is the subject having the write privilege for the blob.

Returns:

AtomicOutputStream returns a stream into which the data can be written.

Throws:

AuthorizationException

KeyNotFoundException

getBlobMeta

public abstract ReadableBlobMeta getBlobMeta(String key,
                                             Subject who)
                                      throws AuthorizationException,
                                             KeyNotFoundException

Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.

Parameters:

key - Key for the blob.

who - Is the subject having the read privilege for the blob.

Returns:

AtomicOutputStream returns a stream into which the data can be written.

Throws:

AuthorizationException

KeyNotFoundException

setBlobMeta

public abstract void setBlobMeta(String key,
                                 SettableBlobMeta meta,
                                 Subject who)
                          throws AuthorizationException,
                                 KeyNotFoundException

Sets the metadata with renewed acls for the blob.

Parameters:

key - Key for the blob.

meta - Metadata which contains the updated acls information.

who - Is the subject having the write privilege for the blob.

Throws:

AuthorizationException

KeyNotFoundException

deleteBlob

public abstract void deleteBlob(String key,
                                Subject who)
                         throws AuthorizationException,
                                KeyNotFoundException

Deletes the blob data and metadata.

Parameters:

key - Key for the blob.

who - Is the subject having write privilege for the blob.

Throws:

AuthorizationException

KeyNotFoundException

getBlob

public abstract InputStreamWithMeta getBlob(String key,
                                            Subject who)
                                     throws AuthorizationException,
                                            KeyNotFoundException

Gets the InputStream to read the blob details

Parameters:

key - Key for the blob.

who - Is the subject having the read privilege for the blob.

Returns:

InputStreamWithMeta has the additional file length and version information.

Throws:

AuthorizationException

KeyNotFoundException

listKeys

public abstract Iterator<String> listKeys()

Returns an iterator with all the list of keys currently available on the blob store.

Returns:

Iterator

getBlobReplication

public abstract int getBlobReplication(String key,
                                       Subject who)
                                throws Exception

Gets the replication factor of the blob.

Parameters:

key - Key for the blob.

who - Is the subject having the read privilege for the blob.

Returns:

BlobReplication object containing the replication factor for the blob.

Throws:

Exception

updateBlobReplication

public abstract int updateBlobReplication(String key,
                                          int replication,
                                          Subject who)
                                   throws AuthorizationException,
                                          KeyNotFoundException,
                                          IOException

Modifies the replication factor of the blob.

Parameters:

key - Key for the blob.

replication - The replication factor the blob has to be set.

who - Is the subject having the update privilege for the blob

Returns:

BlobReplication object containing the updated replication factor for the blob.

Throws:

AuthorizationException

KeyNotFoundException

IOException

filterAndListKeys

public <R> Set<R> filterAndListKeys(KeyFilter<R> filter)

Filters keys based on the KeyFilter passed as the argument.

Parameters:

filter - KeyFilter

R - Type

Returns:

Set of filtered keys

validateKey

public static final void validateKey(String key)
                              throws AuthorizationException

Validates key checking for potentially harmful patterns

Parameters:

key - Key for the blob.

Throws:

AuthorizationException

createBlob

public void createBlob(String key,
                       byte[] data,
                       SettableBlobMeta meta,
                       Subject who)
                throws AuthorizationException,
                       KeyAlreadyExistsException,
                       IOException

Wrapper called to create the blob which contains the byte data

Parameters:

key - Key for the blob.

data - Byte data that needs to be uploaded.

meta - Metadata which contains the acls information

who - Is the subject creating the blob.

Throws:

AuthorizationException

KeyAlreadyExistsException

IOException

createBlob

public void createBlob(String key,
                       InputStream in,
                       SettableBlobMeta meta,
                       Subject who)
                throws AuthorizationException,
                       KeyAlreadyExistsException,
                       IOException

Wrapper called to create the blob which contains the byte data

Parameters:

key - Key for the blob.

in - InputStream from which the data is read to be written as a part of the blob.

meta - Metadata which contains the acls information

who - Is the subject creating the blob.

Throws:

AuthorizationException

KeyAlreadyExistsException

IOException

readBlobTo

public void readBlobTo(String key,
                       OutputStream out,
                       Subject who)
                throws IOException,
                       KeyNotFoundException,
                       AuthorizationException

Reads the blob from the blob store and writes it into the output stream.

Parameters:

key - Key for the blob.

out - Output stream

who - Is the subject having read privilege for the blob.

Throws:

IOException

KeyNotFoundException

AuthorizationException

readBlob

public byte[] readBlob(String key,
                       Subject who)
                throws IOException,
                       KeyNotFoundException,
                       AuthorizationException

Wrapper around readBlobTo which returns a ByteArray output stream.

Parameters:

key - Key for the blob.

who - Is the subject having the read privilege for the blob.

Returns:

ByteArrayOutputStream

Throws:

IOException

KeyNotFoundException

AuthorizationException