public class HdfsBlobStore extends BlobStore
We currently have NIMBUS_ADMINS and SUPERVISOR_ADMINS configuration. NIMBUS_ADMINS are given READ, WRITE and ADMIN access whereas the SUPERVISOR_ADMINS are given READ access in order to read and download the blobs form the nimbus.
The ACLs for the blob store are validated against whether the subject is a NIMBUS_ADMIN, SUPERVISOR_ADMIN or USER who has read, write or admin privileges in order to perform respective operations on the blob.
For hdfs blob store 1. The USER interacts with nimbus to upload and access blobs through NimbusBlobStore Client API. Here, unlike local blob store which stores the blobs locally, the nimbus talks to HDFS to upload the blobs. 2. The USER sets the ACLs, and the blob access is validated against these ACLs. 3. The SUPERVISOR interacts with nimbus through HdfsClientBlobStore to download the blobs. Here, unlike local blob store the supervisor interacts with HDFS directly to download the blobs. The call to HdfsBlobStore is made as a "null" subject. The blobstore gets the hadoop user and validates permissions for the supervisor.
BlobStore.BlobStoreFileInputStream, BlobStore.BlobStoreFileOutputStream, BlobStore.KeyTranslationIterator
BASE_BLOBS_DIR_NAME
Constructor and Description |
---|
HdfsBlobStore() |
Modifier and Type | Method and Description |
---|---|
boolean |
blobExists(String key,
Subject who)
Checks if a blob exists.
|
AtomicOutputStream |
createBlob(String key,
SettableBlobMeta meta,
Subject who)
Creates the blob.
|
void |
deleteBlob(String key,
Subject who)
Deletes the blob data and metadata.
|
void |
fullCleanup(long age) |
InputStreamWithMeta |
getBlob(String key,
Subject who)
Gets the InputStream to read the blob details.
|
ReadableBlobMeta |
getBlobMeta(String key,
Subject who)
Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.
|
int |
getBlobReplication(String key,
Subject who)
Gets the replication factor of the blob.
|
long |
getLastBlobUpdateTime() |
Iterator<String> |
listKeys()
Returns an iterator with all the list of keys currently available on the blob store.
|
void |
prepare(Map<String,Object> conf,
String overrideBase,
NimbusInfo nimbusInfo,
ILeaderElector leaderElector)
Allows us to initialize the blob store.
|
protected void |
prepareInternal(Map<String,Object> conf,
String overrideBase,
org.apache.hadoop.conf.Configuration hadoopConf)
Allow a Hadoop Configuration to be passed for testing.
|
void |
setBlobMeta(String key,
SettableBlobMeta meta,
Subject who)
Sets the metadata with renewed acls for the blob.
|
void |
setLeaderElector(ILeaderElector leaderElector)
Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.
|
void |
shutdown() |
AtomicOutputStream |
updateBlob(String key,
Subject who)
Updates the blob data.
|
int |
updateBlobReplication(String key,
int replication,
Subject who)
Modifies the replication factor of the blob.
|
void |
updateLastBlobUpdateTime()
Updates the last update time of existing blobs in the blobstore to the current time.
|
void |
validateBlobUpdateTime()
Validates that the blob update time of the blobstore is up to date with the current existing blobs.
|
void |
writeMetadata(String key,
SettableBlobMeta meta) |
close, createBlob, createBlob, filterAndListKeys, readBlob, readBlobTo, startSyncBlobs, storedTopoIds, updateBlob, validateKey
public void prepare(Map<String,Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector)
BlobStore
protected void prepareInternal(Map<String,Object> conf, String overrideBase, org.apache.hadoop.conf.Configuration hadoopConf)
public AtomicOutputStream createBlob(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyAlreadyExistsException
BlobStore
createBlob
in class BlobStore
key
- Key for the blobmeta
- Metadata which contains the acls informationwho
- Is the subject creating the blobAuthorizationException
KeyAlreadyExistsException
public AtomicOutputStream updateBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
updateBlob
in class BlobStore
key
- Key for the blobwho
- Is the subject having the write privilege for the blobAuthorizationException
KeyNotFoundException
public ReadableBlobMeta getBlobMeta(String key, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
getBlobMeta
in class BlobStore
key
- Key for the blobwho
- Is the subject having the read privilege for the blobAuthorizationException
KeyNotFoundException
public void setLeaderElector(ILeaderElector leaderElector)
setLeaderElector
in class BlobStore
leaderElector
- the leader electorpublic void setBlobMeta(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
setBlobMeta
in class BlobStore
key
- Key for the blobmeta
- Metadata which contains the updated acls informationwho
- Is the subject having the write privilege for the blobAuthorizationException
KeyNotFoundException
public void deleteBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
deleteBlob
in class BlobStore
key
- Key for the blobwho
- Is the subject having write privilege for the blobAuthorizationException
KeyNotFoundException
public InputStreamWithMeta getBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
getBlob
in class BlobStore
key
- Key for the blobwho
- Is the subject having the read privilege for the blobAuthorizationException
KeyNotFoundException
public boolean blobExists(String key, Subject who) throws AuthorizationException
key
- blobstore keywho
- subjectAuthorizationException
- if authorization is failedpublic Iterator<String> listKeys()
BlobStore
public void shutdown()
public int getBlobReplication(String key, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
getBlobReplication
in class BlobStore
key
- Key for the blobwho
- Is the subject having the read privilege for the blobAuthorizationException
KeyNotFoundException
public int updateBlobReplication(String key, int replication, Subject who) throws AuthorizationException, KeyNotFoundException
BlobStore
updateBlobReplication
in class BlobStore
key
- Key for the blobreplication
- The replication factor the blob has to be setwho
- Is the subject having the update privilege for the blobAuthorizationException
KeyNotFoundException
public void writeMetadata(String key, SettableBlobMeta meta) throws AuthorizationException, KeyNotFoundException
public void fullCleanup(long age) throws IOException
IOException
public long getLastBlobUpdateTime() throws IOException
IOException
public void updateLastBlobUpdateTime() throws IOException
BlobStore
updateLastBlobUpdateTime
in class BlobStore
IOException
- on any errorpublic void validateBlobUpdateTime() throws IOException
BlobStore
validateBlobUpdateTime
in class BlobStore
IOException
- on any errorCopyright © 2023 The Apache Software Foundation. All rights reserved.