This is Part 18 of a series on Designing, Building & Packaging A Scalable, Testable .NET Open Source Component.

UPDATED to add Initialization method

In our last post, we looked at implementing large object storage to give our PostgreSQLStorageEngine parity with the SQLServerStorageEngine.

In this post, we will implement a storage engine for Azure Blob Storage.

The first step is understanding how blob storage works. There are three main concepts.

  1. Account
  2. Container
  3. Blob

AzureConcepts

Account

At the top level is the account that hosts all the services. This is accessed and manipulated using the BlobServiceClient.

Container

This level is the equivalent of a folder containing all the files. It is accessed and manipulated using the BlobContainerClient.

Blob

This level is where the actual files are stored. This is accessed and manipulated using the BlobClient.

In our design, we will have two containers - one will store the FileMetadata, and the other will contain the actual binary data.

Next, we shall begin the implementation.

To begin with, we need a way to pass some settings to Azure. We will create a class to store these settings:

public class AzureSettings
{
    [Required] public string AccountName { get; set; } = null!;
    [Required] public string DataContainerName { get; set; } = null!;
    [Required] public string MetadataContainerName { get; set; } = null!;
}

Here we have indicated that all the settings are required.

Next, we will implement the AzureBlobStorageEngine.

public class AzureBlobStorageEngine : IStorageEngine
{
  private readonly BlobContainerClient _dataContainerClient;
  private readonly BlobContainerClient _metadataContainerClient;

  /// <inheritdoc />
  public int TimeoutInMinutes { get; }

  public AzureBlobStorageEngine(int timeoutInMinutes, string account,
      string dataContainerName, string metadataContainerName)
  {
      TimeoutInMinutes = timeoutInMinutes;

      // Create a service client
      var blobServiceClient = new BlobServiceClient(
          new Uri($"https://{account}.blob.core.windows.net"),
          new DefaultAzureCredential());
}

We next implement an Initialization method that will create our buckets if they don’t already exist. This method will be run once, perhaps at startup.

/// <summary>
/// Initialize the engine
/// </summary>
/// <param name="accountName"></param>
/// <param name="accountKey"></param>
/// <param name="azureLocation"></param>
/// <param name="dataContainerName"></param>
/// <param name="metadataContainerName"></param>
/// <param name="cancellationToken"></param>
public async Task InitializeAsync(string accountName, string accountKey, string azureLocation,
    string dataContainerName, string metadataContainerName, CancellationToken cancellationToken = default)
{
    // Create a service client
    var blobServiceClient = new BlobServiceClient(
        new Uri($"{azureLocation}/{accountName}/"),
        new StorageSharedKeyCredential(accountName, accountKey));

    // Get our container clients
    var dataContainerClient = blobServiceClient.GetBlobContainerClient(dataContainerName);
    var metadataContainerClient = blobServiceClient.GetBlobContainerClient(metadataContainerName);

    // Ensure they exist
    if (!await dataContainerClient.ExistsAsync(cancellationToken))
        await dataContainerClient.CreateIfNotExistsAsync(cancellationToken: cancellationToken);
    if (!await metadataContainerClient.ExistsAsync(cancellationToken))
        await metadataContainerClient.CreateIfNotExistsAsync(cancellationToken: cancellationToken);
}

We then go on to implement the various methods of the IStorageEngine interface.

First, StoreFileAsync:

/// <inheritdoc />
public async Task<FileMetadata> StoreFileAsync(FileMetadata metaData, Stream data,
    CancellationToken cancellationToken = default)
{
    // Get the clients
    var dataClient = _dataContainerClient.GetBlobClient(metaData.FileId.ToString());
    var metadataClient = _metadataContainerClient.GetBlobClient(metaData.FileId.ToString());

    // Upload data in parallel
    await Task.WhenAll(
        metadataClient.UploadAsync(JsonSerializer.Serialize(metaData), cancellationToken),
        dataClient.UploadAsync(data, cancellationToken));

    return metaData;
}

Next, GetMetadataAsync:

/// <inheritdoc />
public async Task<FileMetadata> GetMetadataAsync(Guid fileId, CancellationToken cancellationToken = default)
{
    // Get the client
    var metadataClient = _metadataContainerClient.GetBlobClient(fileId.ToString());

    // Retrieve the metadata
    var result = await metadataClient.DownloadContentAsync(cancellationToken: cancellationToken);
    if (result != null && result.HasValue)
    {
        return JsonSerializer.Deserialize<FileMetadata>(result.Value!.Content!.ToString())!;
    }

    throw new FileNotFoundException($"File {fileId} not found");
}

Next, GetFileAsync:

/// <inheritdoc />
public async Task<Stream> GetFileAsync(Guid fileId, CancellationToken cancellationToken = default)
{
  // Get the client
  var dataClient = _dataContainerClient.GetBlobClient(fileId.ToString());

  // Download the blob as a stream
  var response = await dataClient.DownloadStreamingAsync(cancellationToken: cancellationToken);

  // Download into a memory stream
  await using (var stream = response.Value.Content)
  {
      var memoryStream = new MemoryStream();
      await stream.CopyToAsync(memoryStream, cancellationToken);
      return memoryStream;
  }
}

Next, DeleteFileAsync:

/// <inheritdoc />
public async Task DeleteFileAsync(Guid fileId, CancellationToken cancellationToken = default)
{
  // Get the clients
  var dataClient = _dataContainerClient.GetBlobClient(fileId.ToString());
  var metadataClient = _metadataContainerClient.GetBlobClient(fileId.ToString());

  // Delete in parallel
  await Task.WhenAll(
      metadataClient.DeleteAsync(cancellationToken: cancellationToken),
      dataClient.DeleteAsync(cancellationToken: cancellationToken));
}

Finally, FileExistsAsync:

  /// <inheritdoc />
    public async Task<bool> FileExistsAsync(Guid fileId, CancellationToken cancellationToken = default)
    {
        // Get the client
        var dataClient = _dataContainerClient.GetBlobClient(fileId.ToString());
        // Check for existence
        return await dataClient.ExistsAsync(cancellationToken);
    }

Our next post will look at how to test this locally.

TLDR

In this post, we implemented a storage engine for Azure Blob Storage.

The code is in my GitHub.

Happy hacking!