Blob Code Download Much Slower Than MS Azure Storage Explorer


Answer :

You should specify which version of MS Azure Storage explorer your're using.

If you're using some newer versions of 1.9.0 / 1.8.1 / 1.8.0 etc.(please find more details in this link), then Azure Storage Explorer is integrated with azcopy which is using simple commands designed for optimal performance. So you can have a good-performance for downloading / uploading etc.

When using code for downloading / uploading blobs, you can take use of this Microsoft Azure Storage Data Movement Library. This library is based on the core data movement framework that powers AzCopy, which also provides you high-performance uploading, downloading.


I eventually tried 2 solutions proposed by @Ivan and @mjwills:

  • DownloadToFileParallelAsync resulted in 10min 12secs
  • Microsoft Azure Storage Data Movement Library resulted in 9min 35secs

Both solutions much faster than the original DownloadToFileAsync. DownloadToFileParallelAsync is only available in later versions of the library and hence was not available in the one I had installed.


I'd suggest using DownloadToFileParallelAsync.

As per the docs:

Initiates an asynchronous operation to download the contents of a blob to a file by making parallel requests.

and:

The parallelIOCount and rangeSizeInBytes should be adjusted depending on the CPU, memory, and bandwidth.

This API should only be used for larger downloads as a HEAD request is made prior to downloading the data.

For smaller blobs, please use DownloadToFileAsync().

To get the best performance, it is recommended to try several values, and measure throughput.

One place to start would be to set the parallelIOCount to the number of CPUs.

Then adjust the rangeSizeInBytes so that parallelIOCount times rangeSizeInBytes equals the amount of memory you want the process to consume.

The benefit of this method vs DownloadToFileAsync is that multiple 'slices' of the file are downloaded in parallel (at the same time). This can be beneficial for large files over fast internet connections (in most cases, I'd expect it to be 4-8 times faster).


Comments

Post a Comment

Popular posts from this blog

Are Regular VACUUM ANALYZE Still Recommended Under 9.1?

Answer : VACUUM is only needed on updated or deleted rows in non-temporary tables. Obviously you're doing lots of INSERTs but it's not obvious from the description that you're also doing lots of UPDATEs or DELETEs. These operations can be tracked with the pg_stat_all_tables view, specifically the n_tup_upd and n_tup_del columns. Also, even more to the point, there is a n_dead_tup column that tells, per table, how much rows need to be vacuumed. (see Monitoring statistics in the doc for functions and views related to statistics gathering). A possible strategy in your case would be to suppress the scheduled VACUUM, keeping an eye on this view and checking on which tables the n_dead_tup is going up significantly. Then apply the aggressive VACUUM to these tables only. This will be a win if there are large tables whose rows never get deleted nor updated and the aggressive VACUUM is really necessary only on smaller tables. But keep running the ANALYZE for the optimiz
Read more

Can Feynman Diagrams Be Used To Represent Any Perturbation Theory?

Answer : Diagram machinery works also for perturbation theory in classical statistical mechanics and classical field theories. Generally, various kinds of diagrams constitute a pictorial way of talking about tensor products and their contractions while hiding the multi-linear algebra from the layman. In the simplest case, vertices (or blobs) represent vectors, matrices, tensors; vertices have ingoing and/or outgoing ports; ingoing ports denote (contravariant) ro indices; outgoing ports denote (covariant) co indices; directed arcs between blobs give a pair of equal indices summed over; different base spaces ⇔ \Leftrightarrow ⇔ different arc types; symmetric tensors ⇔ \Leftrightarrow ⇔ undirected diagrams; no labels are needed for internal lines. In many cases, the directed arcs are decorated (as thin, thick, broken,wavy, curly lines), each decoration indicating the presence of a so-called propagator, a function of its label to use as a weight in the sum, which may beco
Read more