org.apache.storm.hdfs.blobstore

Class HdfsBlobStore

java.lang.Object

org.apache.storm.blobstore.BlobStore

org.apache.storm.hdfs.blobstore.HdfsBlobStore

All Implemented Interfaces:

AutoCloseable, Shutdownable

public class HdfsBlobStore
extends BlobStore

Provides a HDFS file system backed blob store implementation. Note that this provides an api for having HDFS be the backing store for the blobstore, it is not a service/daemon.

We currently have NIMBUS_ADMINS and SUPERVISOR_ADMINS configuration. NIMBUS_ADMINS are given READ, WRITE and ADMIN access whereas the SUPERVISOR_ADMINS are given READ access in order to read and download the blobs form the nimbus.

The ACLs for the blob store are validated against whether the subject is a NIMBUS_ADMIN, SUPERVISOR_ADMIN or USER who has read, write or admin privileges in order to perform respective operations on the blob.

For hdfs blob store 1. The USER interacts with nimbus to upload and access blobs through NimbusBlobStore Client API. Here, unlike local blob store which stores the blobs locally, the nimbus talks to HDFS to upload the blobs. 2. The USER sets the ACLs, and the blob access is validated against these ACLs. 3. The SUPERVISOR interacts with nimbus through HdfsClientBlobStore to download the blobs. Here, unlike local blob store the supervisor interacts with HDFS directly to download the blobs. The call to HdfsBlobStore is made as a “null” subject. The blobstore gets the hadoop user and validates permissions for the supervisor.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.storm.blobstore.BlobStore

BlobStore.BlobStoreFileInputStream, BlobStore.BlobStoreFileOutputStream, BlobStore.KeyTranslationIterator

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.storm.blobstore.BlobStore

BASE_BLOBS_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
HdfsBlobStore()

Method Summary

Modifier and Type	Method and Description
boolean	blobExists(String key, Subject who) Checks if a blob exists.
AtomicOutputStream	createBlob(String key, SettableBlobMeta meta, Subject who) Creates the blob.
void	deleteBlob(String key, Subject who) Deletes the blob data and metadata.
void	fullCleanup(long age)
InputStreamWithMeta	getBlob(String key, Subject who) Gets the InputStream to read the blob details.
ReadableBlobMeta	getBlobMeta(String key, Subject who) Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.
int	getBlobReplication(String key, Subject who) Gets the replication factor of the blob.
Iterator<String>	listKeys() Returns an iterator with all the list of keys currently available on the blob store.
void	prepare(Map<String,Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Allows us to initialize the blob store.
protected void	prepareInternal(Map<String,Object> conf, String overrideBase, org.apache.hadoop.conf.Configuration hadoopConf) Allow a Hadoop Configuration to be passed for testing.
void	setBlobMeta(String key, SettableBlobMeta meta, Subject who) Sets the metadata with renewed acls for the blob.
void	setLeaderElector(ILeaderElector leaderElector) Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.
void	shutdown()
AtomicOutputStream	updateBlob(String key, Subject who) Updates the blob data.
int	updateBlobReplication(String key, int replication, Subject who) Modifies the replication factor of the blob.
void	writeMetadata(String key, SettableBlobMeta meta)

Methods inherited from class org.apache.storm.blobstore.BlobStore

close, createBlob, createBlob, filterAndListKeys, readBlob, readBlobTo, startSyncBlobs, storedTopoIds, updateBlob, validateKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

HdfsBlobStore

public HdfsBlobStore()

Method Detail

prepare

public void prepare(Map<String,Object> conf,
                    String overrideBase,
                    NimbusInfo nimbusInfo,
                    ILeaderElector leaderElector)

Description copied from class: BlobStore

Allows us to initialize the blob store.

Specified by:

prepare in class BlobStore

Parameters:

conf - The storm configuration

overrideBase - The directory path to store the blobs

nimbusInfo - Contains the nimbus host, port and leadership information

prepareInternal

protected void prepareInternal(Map<String,Object> conf,
                               String overrideBase,
                               org.apache.hadoop.conf.Configuration hadoopConf)

Allow a Hadoop Configuration to be passed for testing. If it’s null then the hadoop configs must be in your classpath.

createBlob

public AtomicOutputStream createBlob(String key,
                                     SettableBlobMeta meta,
                                     Subject who)
                              throws AuthorizationException,
                                     KeyAlreadyExistsException

Description copied from class: BlobStore

Creates the blob.

Specified by:

createBlob in class BlobStore

Parameters:

key - Key for the blob

meta - Metadata which contains the acls information

who - Is the subject creating the blob

Returns:

AtomicOutputStream returns a stream into which the data can be written

Throws:

AuthorizationException

KeyAlreadyExistsException

updateBlob

public AtomicOutputStream updateBlob(String key,
                                     Subject who)
                              throws AuthorizationException,
                                     KeyNotFoundException

Description copied from class: BlobStore

Updates the blob data.

Specified by:

updateBlob in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the write privilege for the blob

Returns:

AtomicOutputStream returns a stream into which the data can be written

Throws:

AuthorizationException

KeyNotFoundException

getBlobMeta

public ReadableBlobMeta getBlobMeta(String key,
                                    Subject who)
                             throws AuthorizationException,
                                    KeyNotFoundException

Description copied from class: BlobStore

Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.

Specified by:

getBlobMeta in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the read privilege for the blob

Returns:

AtomicOutputStream returns a stream into which the data can be written

Throws:

AuthorizationException

KeyNotFoundException

setLeaderElector

public void setLeaderElector(ILeaderElector leaderElector)

Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.

Specified by:

setLeaderElector in class BlobStore

Parameters:

leaderElector - the leader elector

setBlobMeta

public void setBlobMeta(String key,
                        SettableBlobMeta meta,
                        Subject who)
                 throws AuthorizationException,
                        KeyNotFoundException

Description copied from class: BlobStore

Sets the metadata with renewed acls for the blob.

Specified by:

setBlobMeta in class BlobStore

Parameters:

key - Key for the blob

meta - Metadata which contains the updated acls information

who - Is the subject having the write privilege for the blob

Throws:

AuthorizationException

KeyNotFoundException

deleteBlob

public void deleteBlob(String key,
                       Subject who)
                throws AuthorizationException,
                       KeyNotFoundException

Description copied from class: BlobStore

Deletes the blob data and metadata.

Specified by:

deleteBlob in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having write privilege for the blob

Throws:

AuthorizationException

KeyNotFoundException

getBlob

public InputStreamWithMeta getBlob(String key,
                                   Subject who)
                            throws AuthorizationException,
                                   KeyNotFoundException

Description copied from class: BlobStore

Gets the InputStream to read the blob details.

Specified by:

getBlob in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the read privilege for the blob

Returns:

InputStreamWithMeta has the additional file length and version information

Throws:

AuthorizationException

KeyNotFoundException

blobExists

public boolean blobExists(String key,
                          Subject who)
                   throws AuthorizationException

Checks if a blob exists.

Parameters:

key - blobstore key

who - subject

Throws:

AuthorizationException - if authorization is failed

listKeys

public Iterator<String> listKeys()

Description copied from class: BlobStore

Returns an iterator with all the list of keys currently available on the blob store.

Specified by:

listKeys in class BlobStore

Returns:

Iterator<String>

shutdown

public void shutdown()

getBlobReplication

public int getBlobReplication(String key,
                              Subject who)
                       throws AuthorizationException,
                              KeyNotFoundException

Description copied from class: BlobStore

Gets the replication factor of the blob.

Specified by:

getBlobReplication in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the read privilege for the blob

Returns:

BlobReplication object containing the replication factor for the blob

Throws:

AuthorizationException

KeyNotFoundException

updateBlobReplication

public int updateBlobReplication(String key,
                                 int replication,
                                 Subject who)
                          throws AuthorizationException,
                                 KeyNotFoundException

Description copied from class: BlobStore

Modifies the replication factor of the blob.

Specified by:

updateBlobReplication in class BlobStore

Parameters:

key - Key for the blob

replication - The replication factor the blob has to be set

who - Is the subject having the update privilege for the blob

Returns:

BlobReplication object containing the updated replication factor for the blob

Throws:

AuthorizationException

KeyNotFoundException

writeMetadata

public void writeMetadata(String key,
                          SettableBlobMeta meta)
                   throws AuthorizationException,
                          KeyNotFoundException

Throws:

AuthorizationException

KeyNotFoundException

fullCleanup

public void fullCleanup(long age)
                 throws IOException

Throws:

IOException