This is Part 18 of a series on Designing, Building & Packaging A Scalable, Testable .NET Open Source Component.

In our last post, we looked at implementing large object storage to give our PostgreSQLStorageEngine parity with the SQLServerStorageEngine.

In this post, we will implement a storage engine for Azure Blob Storage.

The first step is understanding how blob storage works. There are three main concepts.

  1. Account
  2. Container
  3. Blob

AzureConcepts

Account

At the top level is the account that hosts all the services. This is accessed and manipulated using the BlobServiceClient.

Container

This level is the equivalent of a folder containing all the files. It is accessed and manipulated using the BlobContainerClient.

Blob

This level is where the actual files are stored. This is accessed and manipulated using the BlobClient.

In our design, we will have two containers - one will store the FileMetadata, and the other will contain the actual binary data.

Next, we shall begin the implementation.

To begin with, we need a way to pass some settings to Azure. We will create a class to store these settings:

public class AzureSettings
{
    [Required] public string AccountName { get; set; } = null!;
    [Required] public string DataContainerName { get; set; } = null!;
    [Required] public string MetadataContainerName { get; set; } = null!;
}

Here we have indicated that all the settings are required.

Next, we will implement the AzureBlobStorageEngine.

public class AzureBlobStorageEngine : IStorageEngine
{
  private readonly BlobContainerClient _dataContainerClient;
  private readonly BlobContainerClient _metadataContainerClient;

  /// <inheritdoc />
  public int TimeoutInMinutes { get; }

  public AzureBlobStorageEngine(int timeoutInMinutes, string account,
      string dataContainerName, string metadataContainerName)
  {
      TimeoutInMinutes = timeoutInMinutes;

      // Create a service client
      var blobServiceClient = new BlobServiceClient(
          new Uri($"https://{account}.blob.core.windows.net"),
          new DefaultAzureCredential());

      // Create container clients
      _dataContainerClient = blobServiceClient.CreateBlobContainer(dataContainerName);
      _metadataContainerClient = blobServiceClient.CreateBlobContainer(metadataContainerName);
      // Ensure they exist
      _dataContainerClient.CreateIfNotExists();
      _metadataContainerClient.CreateIfNotExists();
}

We then go on to implement the various methods of the IStorageEngine interface.

First, StoreFileAsync:

/// <inheritdoc />
public async Task<FileMetadata> StoreFileAsync(FileMetadata metaData, Stream data,
    CancellationToken cancellationToken = default)
{
    // Get the clients
    var dataClient = _dataContainerClient.GetBlobClient(metaData.FileId.ToString());
    var metadataClient = _metadataContainerClient.GetBlobClient(metaData.FileId.ToString());

    // Upload data in parallel
    await Task.WhenAll(
        metadataClient.UploadAsync(JsonSerializer.Serialize(metaData), cancellationToken),
        dataClient.UploadAsync(data, cancellationToken));

    return metaData;
}

Next, GetMetadataAsync:

/// <inheritdoc />
public async Task<FileMetadata> GetMetadataAsync(Guid fileId, CancellationToken cancellationToken = default)
{
    // Get the client
    var metadataClient = _metadataContainerClient.GetBlobClient(fileId.ToString());

    // Retrieve the metadata
    var result = await metadataClient.DownloadContentAsync(cancellationToken: cancellationToken);
    if (result != null && result.HasValue)
    {
        return JsonSerializer.Deserialize<FileMetadata>(result.Value!.Content!.ToString())!;
    }

    throw new FileNotFoundException($"File {fileId} not found");
}

Next, GetFileAsync:

/// <inheritdoc />
public async Task<Stream> GetFileAsync(Guid fileId, CancellationToken cancellationToken = default)
{
  // Get the client
  var dataClient = _dataContainerClient.GetBlobClient(fileId.ToString());

  // Download the blob as a stream
  var response = await dataClient.DownloadStreamingAsync(cancellationToken: cancellationToken);

  // Download into a memory stream
  await using (var stream = response.Value.Content)
  {
      var memoryStream = new MemoryStream();
      await stream.CopyToAsync(memoryStream, cancellationToken);
      return memoryStream;
  }
}

Next, DeleteFileAsync:

/// <inheritdoc />
public async Task DeleteFileAsync(Guid fileId, CancellationToken cancellationToken = default)
{
  // Get the clients
  var dataClient = _dataContainerClient.GetBlobClient(fileId.ToString());
  var metadataClient = _metadataContainerClient.GetBlobClient(fileId.ToString());

  // Delete in parallel
  await Task.WhenAll(
      metadataClient.DeleteAsync(cancellationToken: cancellationToken),
      dataClient.DeleteAsync(cancellationToken: cancellationToken));
}

Finally, FileExistsAsync:

  /// <inheritdoc />
    public async Task<bool> FileExistsAsync(Guid fileId, CancellationToken cancellationToken = default)
    {
        // Get the client
        var dataClient = _dataContainerClient.GetBlobClient(fileId.ToString());
        // Check for existence
        return await dataClient.ExistsAsync(cancellationToken);
    }

Our next post will look at how to test this locally.

TLDR

In this post, we implemented a storage engine for Azure Blob Storage.

The code is in my GitHub.

Happy hacking!