[BUG] Memory leak in Azure.Storage.DataMovement.TransferManager #51327
Description
Library name and version
Azure.Storage.DataMovement 12.2.0-beta.1
Describe the bug
When migrating from Microsoft.Azure.Storage.DataMovement to Azure.Storage.DataMovement I have noticed a memory leak, where the only change was the migration to the new library.
I may be wrong, but looking at the code the TransferManager holds a dictionary of all transfers ( _transfers) which are only removed upon resume (line 303) or exception (line 408).
That means that for a normal flow, the transfers are held forever in the dictionary.
Assuming the correct usage of TransferManager is as a singleton this doesn't scale.
Expected behavior
Scan operations should be cleaned upon completion
Actual behavior
Upon investigation I see that there are many cancellation objects and these are held by the TransferManager.
Reproduction Steps
const int transferCount = 10;
// Create transferCount transfers using TransferManager.
var blobUri = new Uri("https://<mystorage>.blob.core.windows.net/mycontainer/myblob");
var transferManager = new TransferManager();
var credential = new DefaultAzureCredential();
var blobClient = new BlockBlobClient(blobUri, credential);
var src = BlobsStorageResourceProvider.FromClient(blobClient);
for (int i = 0; i < ; i++)
{
var dst = LocalFilesStorageResourceProvider.FromFile($"C:\\tmp\file{i}");
TransferOperation transferOperation = await transferManager.StartTransferAsync(src, dst);
await transferOperation.WaitForCompletionAsync(CancellationToken.None);
}
// Get TransferManager's transfer dictionary using reflection
Type typ = typeof(TransferManager);
FieldInfo field = typ.GetField("_transfers", BindingFlags.NonPublic | BindingFlags.Instance);
var transfers = field.GetValue(transferManager) as ConcurrentDictionary<string, TransferOperation>;
// Assert that all transfers are still held within the dictionary.
Debug.Assert(transfers.Count == transferCount);
Console.WriteLine(transfers.Count);
Environment
Hosting platform: Service Fabric
OS: Microsoft Windows Server 2022 Datacenter Azure Edition
Runtime: .NET 4.6.2