Skip to content

[QUERY] Azure.Storage.DataMovement performance copying containers #48329

@JonSmith-Work

Description

@JonSmith-Work

Library name and version

Azure.Storage.DataMovement.Blobs 12.0.0

Query/Question

Trying to use Azure.Storage.DataMovement to copy whole containers either within or across storage accounts from a console app. It works, but is a lot slower than using AzCopy. Should/could it be comparable in speed?

Test example with 7.5GB container containing 80k files

  • AzCopy (default options) took 4 minutes
  • TransferManager.StartTransferAsync took 30 minutes

Changing MaximumConcurrency to 64 (value we used for ParallelOperations with the old library) had no impact (its a nullable int so not sure what the default is).

Previously, for the old library, we also set System.Net.ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8 - which also had no impact to the library

Environment

No response

Metadata

Metadata

Assignees

Labels

ClientThis issue is related to a non-management packageService AttentionWorkflow: This issue is responsible by Azure service team.StorageStorage Service (Queues, Blobs, Files)customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions