Windows Azure Blob Storage

Windows Azure Storage is one of the core components in Windows Azure that offers a scalable, highly available, and competitively priced storage option. Amongst others abstractions in Azure Storage (Table Storage and Queue Storage), Blob Storage is perhaps the most widely-used service. Blob Storage allows us to store any unstructured text and binary data such as video, audio, images, and so many more.

Blob Storage can either be accessed through the API programmatically or explorer tools. This article discusses and reviews several popular explorer tools for Blob Storage.

Reviews and Ratings

Disclaimer

The reviews and ratings are entirely my individual opinion and preference. The reviews and ratings are based on my personal experience when using each product, and what I consider important.

Measurement Criteria

This review will examine these products using the following four dimensions:

• User interface and experience
  I’ll look at how usable the product is. Have the user interface and experience been designed to be comfortable and user friendly?
• Basic features
  This category covers the standard and basic functionality when dealing with Blob Storage. This includes operations such copying / moving files, managing security, and access.
• Advanced settings
  This dimension measures how flexible and configurable the product is. This includes the ability to adjust settings or preferences such as defining block size, retry policy, bandwidth settings, and so on.
• Others notable features
  This metric is about supplementary features that enrich the product, making the product more powerful and beneficial for users. This might include innovative features such as multi-language support, directory comparison, graphical user interface for logging, etc.

For each measurement, I’ll provide a brief description and rating ranging from 1 to 5. 1 means the product provides a poor experience or lacks capability, 5 means the product provides awesome proficiency. Additionally, I would be also giving a N/A (not applicable) for the product that doesn’t have any applicable features.

1. Cloud Storage Studio 2 by Cerebrata

We start the review with Cloud Storage Studio 2 (CSS2) from Cerebrata, a company acquired by Red Gate last Oct 2011. CSS 2 is an exploration tool not only for Blob Storage, but also for Tables and Queue Storage.

The product costs $195 for a Professional License (volume discounts apply). Customers are encouraged to try it out with a 30-day free trial.

a) User interface and experience

CSS2 provides a powerful UI grouping concept and navigation, enabling users to group related storage accounts and subscriptions together – as can be seen in the Figure 1.

Figure 1 – Cloud Storage Studio UI

The Navigation (in red) and Tabs (in yellow) area look good to me. However, I find the Explorer Area (in blue) is tedious. Copying files and directories will prompt a dialog box that only allows us to copy the blobs within the container only as can be seen in Figure 2. I believe there should be more intuitive way to implement this.

Figure 2 – Cloud Storage Studio Copying Blobs

Rating: 3.5

b) Basic features

I would say it satisfies most of the basic needs when dealing with Blob Storage. Starting from managing containers, displaying directories, all the way down to individual blob level are all properly supported.

Rating: 5.0

c) Advanced settings

CSS2 provides powerful settings that enable users to easily define the configuration settings.

Figure 3 – Cloud Storage Studio Configuration Settings

Rating: 4.5

d) Other notable features

One of the features that I like most in CSS2 is the graphical UI for Storage Analytic Logging and Metric. It provides a really expressive experience and has a good look and feel.

Figure 4 – View Storage Analytics Data

Rating: 4.0

2. CloudXplorer by ClumsyLeaf

CloudXplorer is a lightweight yet handy explorer tool from ClumsyLeaf Software. It has been very popular and has been used by many people including Microsoft folks in various events.

CloudXplorer is entirely free-of-charge, downloadable from here.

a) User interface and experience

CloudXplorer comes with Windows Explorer-like user interface, providing a friendly experience, especially for Windows users. Uploading and downloading Blobs are implemented with “Copy / Cut and Paste” experience, and the same when dealing with our local files.

Figure 5 – CloudXplorer User Interface

Rating: 5.0

b) Basic features

I would say it satisfies most of the basic needs.

Rating: 5.0

c) Advanced settings

I don’t find any options for user to define advanced configuration and settings.

Rating: N/A

d) Other notable features

Unfortunately, I also didn’t find any fancy features in CloudXplorer.

Rating: N/A

3. CloudBerry Explorer for Azure Blob Storage by CloudBerry Lab

The last product I’m reviewing is the CloudBerry Explorer. CloudBerry Labs offers many great products focusing for explorer tools and online backup for various cloud providers such asAmazon AWS, Windows Azure, and RackSpace.

Furthermore, CloudBerry Explorer supports multi-languages: English, Chinese, and Japanese. The product comes in two versions:

• Free version
• And PRO version, purchasable at $ 39.99

Check out the following for the comparison between the two.

Figure 6 – CloudBerry Explorer User Interface

Rating: 4.5

b) Basic features

Like the other two tools, I would say it satisfies most needs.

Rating: 5.0

c) Advanced settings

CloudBerry Explorer also provides a powerful and flexible option for user to configure settings such as setting bandwidth, chunk size, encryption, etc. However, I notice that a few of the features such as encryption and compression are only available in PRO version.

Figure 7 – CloudBerry Explorer Options

Rating: 5.0

d) Other notable features

My favorite feature of CloudBerry Explorer is Compare and Sync Folders. This is an extremely useful feature enabling us to compare and sync either cloud or local folders. As seen in the screenshot below, the tool shows the comparison result between the two displays. Then we can finally define to either sync left to right, right to left, or in both directions.

Figure 5 – CloudBerry Explorer User Interface

Rating: 4.5

Conclusion

We have gone through three explorer tools for Windows Azure Blob Storage. I would say all three products are pretty awesome. There are always advantages from one to another. The following table summarizes reviews and ratings that we’ve come across.

In conclusion, if you need a simple and lightweight explorer, CloudXplorer is probably the way to go. However, if you need more flexible settings and innovative features, you should consider Cloud Storage Studio or CloudBerry Explorer.

One of the most common questions in developing ASP.NET applications on Windows Azure is how to manage session state. The intention of this article is to discuss several options to manage session state for ASP.NET applications in Windows Azure.

What is session state?

Session state is usually used to store and retrieve values for a user across ASP.NET pages in a web application. There are four available modes to store session values in ASP.NET:

1. In-Proc, which stores session state in the individual web server’s memory. This is the default option if a particular mode is not explicitly specified.
2. State Server, which stores session state in another process, called ASP.NET state service.
3. SQL Server, which stores session state in a SQL Server database
4. Custom, which lets you choose a custom storage provider.

You can get more information about ASP.NET session state here.

In-Proc session mode does not work in Windows Azure

The In-Proc option, which uses an individual web server’s memory, does not work well in Windows Azure. This may be applicable for those of you who host your application in a multi-instance web-farm environment; Windows Azure load balancer uses round-robin allocation across multi-instances.

For example: you have three instances (A, B, and C) of a Web Role. The first time a page is requested, the load balancer will allocate instance A to handle your request. However, there’s no guarantee that instance A will always handle subsequent requests. Similarly,the value that you set in instance A’s memory can’t be accessed by other instances.

The following picture illustrates how session state works in multi-instances behind the load balancer.

Figure 1 – WAPTK BuildingASP.NETApps.pptx Slide 10

The other options

1.     Table Storage

Table Storage Provider is a subset of the Windows Azure ASP.NET Providers written by the Windows Azure team. The Table Storage Session Provider is,in fact, a custom provider that is compiled into a class library (.dll file), enabling developers to store session state inside Windows Azure Table Storage.

The way it actually works is to store each session as a record in Table Storage. Each record will have an expired column that describe the expired time of each session if there’s no interaction from the user.

The advantage of Table Storage Session Provider is its relatively low cost: $0.14 per GB per month for storage capacity and $0.01 per 10,000 storage transactions. Nonetheless, according to my own experience, one of the notable disadvantages of Table Storage Session Provider is that it may not perform as fast as the other options discussed below.

The following code snippet should be applied in web.config when using Table Storage Session Provider.

<sessionState mode="Custom" customProvider="TableStorageSessionStateProvider">   <providers>     <clear/>    <add name="TableStorageSessionStateProvider"         type="Microsoft.Samples.ServiceHosting.AspProviders.TableStorageSessionStateProvider" />   </providers>
</sessionState>

You can get more detail on using Table Storage Session Provider step-by-step here.

2.     SQL Azure

As SQL Azure is essentially a subset of SQL Server, SQL Azure can also be used as storage for session state. With just a few modifications, SQL Azure Session Provider can be derived from SQL Server Session Provider.

You will need to apply the following code snippet in web.config when using SQL Azure Session Provider:

<sessionState mode="SQLServer"
sqlConnectionString="Server=tcp:[serverName].database.windows.net;Database=myDataBase;User ID=[LoginForDb]@[serverName];Password=[password];Trusted_Connection=False;Encrypt=True;"
cookieless="false" timeout="20" allowCustomSqlDatabase="true"
/>

For the detail on how to use SQL Azure Session Provider, you can either:

• Follow the walkthrough from this post.
• Or use ASP.NET Universal Providers via Nuget.

The advantage of using SQL Azure as session provider is that it’s cost effective, especially when you have an existing SQL Azure database. Although it performs better than Table Storage Session Provider in most cases, it requires you to clean the expired session manually by calling the DeleteExpiredSessions stored procedure. Another drawback of using SQL Azure as session provider is that Microsoft does not provide any official support for this.

3.     Windows Azure Caching

Windows Azure Caching is probably the most preferable option available today. It provides a high-performance, in-memory, distributed caching service. The Windows Azure session state provider is an out-of-process storage mechanism for ASP.NET applications. As we all know, accessing RAM is very much faster than accessing disk, so Windows Azure Caching obviously provides the highest performance access of all the available options.

Windows Azure Caching also comes with a .NET API that enables developers to easily interact with the Caching Service. You should apply the following code snippet in web.config when using Cache Session Provider:

<sessionState mode="Custom" customProvider="AzureCacheSessionStoreProvider">   <providers>     <add name="AzureCacheSessionStoreProvider"           type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache"           cacheName="default" useBlobMode="true" dataCacheClientName="default" />   </providers>
</sessionState>

A step-by-step tutorial for using Caching Service as session provider can be found here.

Other than providing high performance access, another advantage about Windows Azure Caching is that it’s officially supported by Microsoft. Despite its advantages, the charge of Windows Azure Caching is relatively high, starting from $45 per month for 128 MB, all the way up to $325 per month for 4 GB.

Conclusion

I haven’t discussed all the available options for managing session state in Windows Azure, but the three I have discussed are the most popular options out there, and the ones that most people are considering using.

Windows Azure Caching remains the recommended option, despite its cons but developers and architects shouldn’t be afraid to decide on a different option, if it’s more suitable for them in a given scenario.

This post was also published at A Cloud Place blog.

Windows Azure Blob Storage could be analogized as file-system on the cloud. It enables us to store any unstructured data file such as text, images, video, etc. In this post, I will show how to upload big file into Windows Azure Storage. Please be inform that we will be using Block Blob for this case. For more information about Block Blob and Page Block, please visit here.

I am assume that you know how to upload a file to Windows Azure Storage. If you don’t know, I would recommend you to check out this lab from Windows Azure Training Kit.

Uploading a blob (commonly-used technique)

The following snippet show you how to upload a blob using a commonly-used technique, blob.UploadFromStream() which eventually invoking PutBlob REST-API.

protected void btnUpload_Click(object sender, EventArgs e)
{
    var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
    blobClient = storageAccount.CreateCloudBlobClient();

    CloudBlobContainer container = blobClient.GetContainerReference("image2");
    container.CreateIfNotExist();

    var permission = container.GetPermissions();
    permission.PublicAccess = BlobContainerPublicAccessType.Container;
    container.SetPermissions(permission);

    string name = fu.FileName;
    CloudBlob blob = container.GetBlobReference(name);
    blob.UploadFromStream(fu.FileContent);
}

The above code snippet works well in most case. Although you could upload at maximum 64 MB per file (for block blob), it’s more recommended to upload using another technique which I am going to describe more detail.

Uploading a blob by splitting it into chunks and calling PutBlockList

The idea of this technique is to split a block blob into smaller chunk of blocks, uploading them one-by-one or in-parallel and eventually join them all by calling PutBlockList().

protected void btnUpload_Click(object sender, EventArgs e)
{
    CloudBlobClient blobClient;
    var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
    blobClient = storageAccount.CreateCloudBlobClient();

    CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
    container.CreateIfNotExist();

    var permission = container.GetPermissions();
    permission.PublicAccess = BlobContainerPublicAccessType.Container;
    container.SetPermissions(permission);

    string name = fu.FileName;
    CloudBlockBlob blob = container.GetBlockBlobReference(name);

    blob.UploadFromStream(fu.FileContent);

    int maxSize = 1 * 1024 * 1024; // 4 MB

    if (fu.PostedFile.ContentLength > maxSize)
    {
        byte[] data = fu.FileBytes; 
        int id = 0;
        int byteslength = data.Length;
        int bytesread = 0;
        int index = 0;
        List<string> blocklist = new List<string>();
        int numBytesPerChunk = 250 * 1024; //250KB per block
    
        do
        {
            byte[] buffer = new byte[numBytesPerChunk];
            int limit = index + numBytesPerChunk;
            for (int loops = 0; index < limit; index++)
            {
                buffer[loops] = data[index];
                loops++;
            }
            bytesread = index;
            string blockIdBase64 = Convert.ToBase64String(System.BitConverter.GetBytes(id));

            blob.PutBlock(blockIdBase64, new MemoryStream(buffer, true), null); 
            blocklist.Add(blockIdBase64);
            id++;
        } while (byteslength - bytesread > numBytesPerChunk);

        int final = byteslength - bytesread;
        byte[] finalbuffer = new byte[final];
        for (int loops = 0; index < byteslength; index++)
        {
            finalbuffer[loops] = data[index];
            loops++;
        }
        string blockId = Convert.ToBase64String(System.BitConverter.GetBytes(id));
        blob.PutBlock(blockId, new MemoryStream(finalbuffer, true), null);
        blocklist.Add(blockId);

        blob.PutBlockList(blocklist); 
    }
    else
        blob.UploadFromStream(fu.FileContent);            
}

Explanation about the code snippet

Since the idea is to split the big file into chunks. We would need to define size of each chunk, in this case 250KB. By dividing actual size with size of each chunk, we should be able to know number of chunk we need to split.

We also need to have a list of string (in this case: blocklist variable) to determine the blocks are in one group. Then we will loop to through each chunk and perform and upload by calling blob.PutBlock() and add it (as form of Base64 String) into the blocklist.

Note that there’s actually a left-over block that didn’t uploaded inside the loop. We will need to upload it again. When all blocks are successfully uploaded, finally we call blob.PutBlockList(). Calling PutListBlock() will commit all the blocks that we’ve uploaded previously.

Pros and Cons

The benefits (pros) of the technique

There’re a few benefit of using this technique:

• In the event where uploading one of the block fail due to whatever condition like connection time-out, connection lost, etc. We’ll just need to upload that particular block only, not the entire big file / blob.
• It’s also possible to upload each block in-parallel which might result shorter upload time.
• The first technique will only allow you to upload a block blob at maximum 64MB. With this technique, you can do more almost unlimited.

The drawbacks (cons) of the technique

Despite of the benefits, there’re also a few drawbacks:

• You have more code to write. As you can see from the sample, you can simply call the one line blob.UploadFromStream() in the first technique. But you will need to write 20+ lines of code for the second technique.
• It incurs more storage transaction as may lead to higher cost in some case. Referring to a post by Azure Storage team. The more chuck you have, the more storage transaction is incurred.

Large blob upload that results in 100 requests via PutBlock, and then 1 PutBlockList for commit = 101 transactions

Summary

I’ve shown you how to upload file with simple technique at beginning. Although, it’s easy to use, it has a few limitation. The second technique (using PutListBlock) is more powerful as it could do more than the first one. However, it certainly has some pros and cons as well.

I hope you could be able to use either one of them appropriately in your scenario. Hope this helps!

In many scenario, you would need to give somebody an access (regardless write, read, etc.) but you don’t want to give him / her full permission. Wouldn’t also be great if you could control the access on certain time frame. The “somebody” could be person or system that use various different platform other than .NET. This post is about to show you how to upload a file to Windows Azure Storage with REST-based API without having to expose your Storage Account credential.

Shared Access Signature

A cool feature Shared Access Signature (SAS) is built-in on Windows Azure Storage. In a nutshell, SAS is a mechanism to give permission while retaining security by producing a set of attributes and signature in the URL.

For the fundamental of SAS, I recommend you to read the following post:

• http://blog.smarx.com/posts/new-storage-feature-signed-access-signatures
• http://blog.smarx.com/posts/shared-access-signatures-are-easy-these-days
• http://msdn.microsoft.com/en-us/library/windowsazure/ee395415.aspx

Here’re the walkthrough how you can do that:

I assume that you’ve the Windows Azure Storage Account and Key with you.

Preparing SAS and Signature

1. Giving access to your container. You can either use tools or library to set SAS permission access on container or blobs. In this example, I use Cerebrata’s Cloud Storage Studio.

As could be seen, I’ve created a policy with the following attributes:

• Start Date Time: Jan 8, 2012 / 00:00:00
• Expiry Date Time: Jan 31, 2012 / 00:00:00
• Permission: Write only
• Signed Identifier: Policy1

By applying this policy to some particular container, somebody who possess a signature will only be able to write something inside this container on the given timeframe. I mentioned “a signature”, what’s the signature then?

2. You can click on “Generate Signed URL” button if you’re using Cloud Storage Studio. But I believe you can do similarly feature although using different tool.

In the textbox, you’ll see something like this:

https://[your-account].blob.core.windows.net/samplecontainer1?&sr=c&si=Policy1&sig=pjJhE%2FIgsGQN9Z1231312312313123123A%3D

Basically, starting the ? symbol to the end, that’s the signature: ?&sr=c&si=Policy1&sig=pjJhE%2FIgsGQN9Z1231312312313123123A%3D

*Copy that value first, you will need this later.

The signature will be signed securely according to your storage credentials and also the properties you’ve specified.

Let’s jump to Visual Studio to start our coding!!!

I use the simplest C# Console Application to get started. Prepare the file to be uploaded. In my case, I am using Penguin.jpg which you can find in Windows sample photo.

3. Since I am about to upload a picture, I will need to get byte[] of data from the actual photo. To do that, I use the following method.

        public static byte[] GetBytesFromFile(string fullFilePath)
        {
            FileStream fs = File.OpenRead(fullFilePath);
            try
            {
                byte[] bytes = new byte[fs.Length];
                fs.Read(bytes, 0, Convert.ToInt32(fs.Length));
                fs.Close();
                return bytes;
            }
            finally
            {
                fs.Close();
            } 
        } 

4. The next step is the most important one, which is to upload a file to Windows Azure Storage through REST with SAS.

        static WebResponse UploadFile(string storageAccount, string container, string filename, string signature, byte[] data)
        {
            var req = (HttpWebRequest)WebRequest.Create(string.Format("http://{0}.blob.core.windows.net/{1}/{2}{3}", storageAccount, container , filename, signature)); 
            req.Method = "PUT"; 
            req.ContentType = "text/plain";

            using (Stream stream = req.GetRequestStream())
            { 
                stream.Write(data, 0, data.Length); 
            }

            return req.GetResponse();
        }

5. To call it, you will need to do the following:

        static void Main(string[] args)
        {
            string storageAccount = "your-storage-account";
            string file = "Penguins.jpg";
            string signature = "?&sr=c&si=Policy1&sig=pjJhE%2FIgsGQN9Z1231312312313123123A%3D";
            string container = "samplecontainer1";

            byte[] data = GetBytesFromFile(file);
            WebResponse resp = UploadFile(storageAccount, container, file, signature, data);
            
            Console.ReadLine();
        }

The signature variable should be filled with the signature that you’ve copied in step 2 just now.

6. Let’s try to see if it works!

And yes, it works .

Hope this helps!

Background

I was quite surprise when seeing the Storage Transaction bills 2000% more than Storage Capacity, and it’s about 40% of my total bill. Wow… How can that be?

Isn’t that the storage transaction just costs $ 0.01 per 10,000 transactions, but why it’s become so expensive? In fact, this is the component that many people ignore when doing the running cost estimation for Windows Azure project.

This leads me to explore and understand Window Azure Storage Transaction more deeply. This article will unveil this unforeseen cost, explain thoroughly how the Storage Transaction costs charge, followed by scenarios that potentially cause the high cost of Storage Transaction. Eventually, I’ll provide some tips to avoid the costly charge of Storage Transaction.

Before getting into the detail, let’s refresh our mind to understanding how Windows Azure Storage costs in overview.

Understanding Windows Azure Storage Billing

Brad Calder from Windows Azure Storage Team did a great post on explaining how the billing looks like for Windows Azure Storage including the Capacity, Bandwidth, and Transaction.

In summary, here’re how it costs (as per Nov 2011). Keep in mind that the cost may change (although not very frequent, but who knows)

1. Storage Capacity = $0.14 per GB stored per month, based on the daily average

2. Storage Transactions = $0.01 per 10,000 transactions

3. Data Transfer (Bandwidth)

• Free Ingress (inbound)
• Outbound:
  • North America and Europe region = $ 0.15 per GB
  • Asia Pacific region = $ 0.20 per GB

Please always refer to the following for latest pricing:

• http://www.microsoft.com/windowsazure/pricing/
• http://www.microsoft.com/windowsazure/pricing-calculator/

Many people argue that Windows Azure Storage is much more cost-effective than SQL Azure.

Well, that’s true in “most of the time”, but not “all the time”. 

Understanding How Storage Transaction Charge in More Detail

Now, let’s forget the Storage Capacity and Bandwidth first, let’s talk about Storage Transaction now. It’s considered 1 transaction whenever you “touch” any component of Windows Azure Storage.

• “Touch” means any REST calls or operation including read, write, delete, update.
• “Any Component” means any entity inside Blobs, Tables, or Queues.

Here’re some examples of transactions that extracted from “Understanding Windows Azure Storage Billings” post.

• A single GetBlob request to the blob service = 1 transaction
• PutBlob with 1 request to the blob service = 1 transaction
• Large blob upload that results in 100 requests via PutBlock, and then 1 PutBlockList for commit = 101 transactions
• Listing through a lot of blobs using 5 requests total (due to 4 continuation markers) = 5 transactions
• Table single entity AddObject request = 1 transaction
• Table Save Changes (without SaveChangesOptions.Batch) with 100 entities = 100 transactions
• Table Save Changes (with SaveChangesOptions.Batch) with 100 entities = 1 transaction
• Table Query specifying an exact PartitionKey and RowKey match (getting a single entity) = 1 transaction
• Table query doing a single storage request to return 500 entities (with no continuation tokens encountered) = 1 transaction
• Table query resulting in 5 requests to table storage (due to 4 continuation tokens) = 5 transactions
• Queue put message = 1 transaction
• Queue get single message = 1 transaction
• Queue get message on empty queue = 1 transaction
• Queue batch get of 32 messages = 1 transaction
• Queue delete message = 1 transaction

Scenarios

Having done understanding how the storage transaction charge, considering the following scenarios:

Scenario 1 – Iterating files inside Blob container

An application will organize the blobs in different container per each users. It also allows the users to check size of each container. For that, a function is created to loop through entire files inside the container and return the size in decimal. Now, this functionality is exposed at UI screen. An admin can typically call this function a few times a day.

*Update: Actually, we can use ListBlobs method to get the length / size of files inside the container. But anyway, just forget it at the moment. (Thanks to Jai Haridas for this comment)

Some Figures for Illustration

Assuming the following figures are used for illustration:

• I have 1,000 users.
• I have 10,000 of files in average for each container.
• Admin call this function 5 times a day in average.

How much it costs for Storage Transaction per month?

Remember: a single GetBlob request is considered 1 transaction!

1,000 users X 10,000 files X 5 times query X 30 days = 1,500,000,000 transaction

$ 0.01 per 10,000 transactions X 1,500,000,000 transactions = $ 1,500 per month

Well, that’s not cheap at all.

Tips to Bring it Down

• Verify with the admin if they really need to use the function for 5 times a day? Educate them, tell them that each time this function is being called, it roughly costs $ 10 since it involves 10 million transaction (10,000 files X 1,000 users). I bet the admin will also avoid that if he/she knows the cost.
• Do not expose this functionality as real time query to admin. Considering to automatically run this function once in a day, save the size in somewhere. Just let admin to view the daily result (day by day).

With limiting the admin to just only view once a day, what will be the monthly cost looks like:

1,000 users X 10,000 files X 1 times query X 30 days = 300,000,000 transaction

$ 0.01 per 10,000 transactions X 300,000,000  transactions = $ 300 per month

Well, I think that’s fair enough!

Scenario 2 – Worker Role Constantly Pinging Queue

An application enables user to upload some document for processing. The uploaded document will be processed asynchronously at the backend. When processing is done, the user will get notified by email.

Technically, it uses Queue to store and centralize all tasks. Two instances of web roles to take the input and store task as message inside the Queue. On the other hand, 5 instances of Worker Role are provisioned, they will constantly pinging Queue Storage to check if there’s new message to be processed.

The following diagrams illustrates how the architecture may look like.

*icons by http://azuredesignpatterns.com/, David Pallman

Some Figures for Illustration

Assuming the following figures:

• It has 5 instances of Worker Role
• Those Worker Role will constantly get message from Queue (regardless it’s empty or filled)

public override void Run()
{ 
    while (true)
    { 
        CloudQueueMessage msg = queue.GetMessage();
        if (msg != null)
        {  
            // process the message
               }  
    }
}

• Those Worker Role will run 24 hours per day, 30 days per month
• It’s stated here that a single queue is able to process up to 500 messages per second. Let’s assume in average, it will process 200 messages per second (considering some tiny latency between Worker Role and Storage)

How much it costs for Storage Transaction per month?

Remember: a GetMessage on Queue function (regardless empty or filled) is considered 1 transaction

200 req X 60 sec X 60 min X 24 hours X 30 days X 5 instances = 2,592,000,000 transactions

$ 0.01 per 10,000 transactions X 2,592,000,000 transactions = $ 2,592 per month

Tips to Bring it Down  #1

Unless there’s requirement to meet certain number of target, otherwise consider to put some Sleep to especially when you’ve got empty message result for several times.

Assuming we put Thread.Sleep(100) = 0.1 second, which means for every second there will be 10 time polling to the queue to check if there’s message.

public override void Run()
{ 
    while (true)
    {
        CloudQueueMessage msg = queue.GetMessage();
        if (msg != null)
        { 
            // process the message

               }
               else Thread.Sleep(100);
    }
}

With that, how much do you think it will cost for a month?

10 req X 60 sec X 60 min X 24 hours X 30 days X 5 instances = 129,600,000  transactions

$ 0.01 per 10,000 transactions X 129,600,000 transactions = $ 129.6 per month

Well, that’s fair enough.

Tips to Bring it Down #2

When your 5 instances of Worker Role have fetched so many times of empty message, then you should start asking yourself if you really need those 5 instances of Worker Roles?

Scaling them in will not only can bring down the Storage Transaction costs, but also will save you some money on Windows Azure Compute Instances.

*Thanks to Brad Calder for this thought.

Scenario 3 – Be Aware with Turning on Windows Azure Diagnostic

Another hidden scenario that may burst your bill on Storage Transaction is turning on Windows Azure Diagnostic if you do not control it properly.

How Windows Azure Diagnostic Work

Windows Azure Diagnostic collects diagnostic data from your instances and copies it to a Window Azure Storage account (either on blob and table storage). Those diagnostic data (such as log) can indeed help developer for the purpose of monitoring performance and tracing source of failure if exception occurs.

We’ll need to define what kind of log (IIS Logs, Crash Dumps, FREB Logs, Arbitrary log files, Performance Counters, Event Logs, etc.) to be collected and send to Windows Azure Storage either on-schedule-basis or on-demand.

However, if you are not carefully define what you are really need for the diagnostic info, you might end up paying the unexpected bill.

Some Figures for Illustration

Assuming the following figures:

• You a few application that require high processing power of 100 instances
• You apply 5 performance counter logs (Processor% Processor Time, MemoryAvailable Bytes, PhysicalDisk% Disk Time, Network Interface Connection: Bytes Total/sec, Processor Interrupts/sec)
• Performing a schedule transfer for every 5 seconds
• The instance will run 24 hours per day, 30 days per month

How much it costs for Storage Transaction per month?

5 counters X 12 times X 60 min X 24 hours X 30 days X 100 instances = 259,200,000 transactions

$ 0.01 per 10,000 transactions X 129,600,000 transactions = $ 259.2 per month

Tips to Bring it Down #2

Ask yourself again if you really need to monitor all 5 performance counters on every 5 seconds? What if you reduce them to 3 counters and monitor it every 20 seconds?

3 counters X 3 times X 60 min X 24 hours X 30 days X 100 instances = 3,8880,000 transactions

$ 0.01 per 10,000 transactions X 129,600,000 transactions = $ 38.8 per month

You can see how much you save for this numbers. Windows Azure Diagnostic is really needed but use it improperly may cause you paying unnecessary money. It’s double-edge sword, be careful.

Conclusion

To conclude, this article gives you a view of how Transaction Cost of Windows Azure Transaction may lead to costly charge if it’s not properly used. Different component in Windows Azure Platform charges differently, cloud architect should have deep understanding in order to design scalable, reliable, yet cost-effective solution to customer.

In some case where constantly request is requirement, you may also would like to evaluate using SQL Azure instead of Windows Azure Storage because there will no any storage transaction cost in SQL Azure.

Do not worry of using any component if you really need. As long as you architect and design the solution properly, the cost should be reasonable enough.

Hopefully by reading this article, you’ll save some money for storage transaction .