I am glad to share that “a cloudy place” blog is finally up here: http://acloudyplace.com

What is “a cloudy place”?

A centralized blog focused on cloud technology exclusively for developers. If you’ve heard about SQL Server Central, it’s somewhat similar but focus on cloud computing. You can find topics such as general cloud info, Amazon Web Services, Windows Azure, and so many more.

Who own “a cloudy place”?

“a cloudy place” is owned and managed by Red Gate, a software company specializing in SQL, DBA, .NET, and Oracle development tools based in Cambridge, UK.

What to do with me?

Aha! I am invited to become a contributor on “a cloudy place”. In a first few post, you will see me write about some generic cloud concept. Of course, I’ll discuss more about Windows Azure later on.

You can find my articles here.

Happy “cloudy” reading on “a cloudy place”!

What’s new?

There’re several announcements that are made by Windows Azure team yesterday. Here’re the summary:

• Metro-style Windows Azure Home Page http://www.windowsazure.com/en-us/

The site is more simplified now, the look-and-feel looks clean and comfortable.

• Windows Azure SDK for Node.js

https://github.com/WindowsAzure/azure-sdk-for-node

• Maximum database size has been increased from 50GB to 150GB.

http://www.windowsazure.com/en-us/home/tour/database/

*Price will be capped to $ 499.95 per DB per month.

• SQL Azure Federation

http://social.technet.microsoft.com/wiki/contents/articles/2281.aspx

• Metro-styled SQL Azure Management Portal



• Some adjustment on pricing

http://www.windowsazure.com/en-us/pricing/details/

More Info?

For more detail, please refer to the following links:

• http://blogs.msdn.com/b/windowsazure/archive/2011/12/12/improved-developer-experience-interoperability-and-scalability-on-windows-azure.aspx
• http://blogs.msdn.com/b/windowsazure/archive/2011/12/12/now-available-sql-azure-q4-2011-service-release.aspx

Background

I was quite surprise when seeing the Storage Transaction bills 2000% more than Storage Capacity, and it’s about 40% of my total bill. Wow… How can that be?

Isn’t that the storage transaction just costs $ 0.01 per 10,000 transactions, but why it’s become so expensive? In fact, this is the component that many people ignore when doing the running cost estimation for Windows Azure project.

This leads me to explore and understand Window Azure Storage Transaction more deeply. This article will unveil this unforeseen cost, explain thoroughly how the Storage Transaction costs charge, followed by scenarios that potentially cause the high cost of Storage Transaction. Eventually, I’ll provide some tips to avoid the costly charge of Storage Transaction.

Before getting into the detail, let’s refresh our mind to understanding how Windows Azure Storage costs in overview.

Understanding Windows Azure Storage Billing

Brad Calder from Windows Azure Storage Team did a great post on explaining how the billing looks like for Windows Azure Storage including the Capacity, Bandwidth, and Transaction.

In summary, here’re how it costs (as per Nov 2011). Keep in mind that the cost may change (although not very frequent, but who knows)

1. Storage Capacity = $0.14 per GB stored per month, based on the daily average

2. Storage Transactions = $0.01 per 10,000 transactions

3. Data Transfer (Bandwidth)

• Free Ingress (inbound)
• Outbound:
  • North America and Europe region = $ 0.15 per GB
  • Asia Pacific region = $ 0.20 per GB

Please always refer to the following for latest pricing:

• http://www.microsoft.com/windowsazure/pricing/
• http://www.microsoft.com/windowsazure/pricing-calculator/

Many people argue that Windows Azure Storage is much more cost-effective than SQL Azure.

Well, that’s true in “most of the time”, but not “all the time”. 

Understanding How Storage Transaction Charge in More Detail

Now, let’s forget the Storage Capacity and Bandwidth first, let’s talk about Storage Transaction now. It’s considered 1 transaction whenever you “touch” any component of Windows Azure Storage.

• “Touch” means any REST calls or operation including read, write, delete, update.
• “Any Component” means any entity inside Blobs, Tables, or Queues.

Here’re some examples of transactions that extracted from “Understanding Windows Azure Storage Billings” post.

• A single GetBlob request to the blob service = 1 transaction
• PutBlob with 1 request to the blob service = 1 transaction
• Large blob upload that results in 100 requests via PutBlock, and then 1 PutBlockList for commit = 101 transactions
• Listing through a lot of blobs using 5 requests total (due to 4 continuation markers) = 5 transactions
• Table single entity AddObject request = 1 transaction
• Table Save Changes (without SaveChangesOptions.Batch) with 100 entities = 100 transactions
• Table Save Changes (with SaveChangesOptions.Batch) with 100 entities = 1 transaction
• Table Query specifying an exact PartitionKey and RowKey match (getting a single entity) = 1 transaction
• Table query doing a single storage request to return 500 entities (with no continuation tokens encountered) = 1 transaction
• Table query resulting in 5 requests to table storage (due to 4 continuation tokens) = 5 transactions
• Queue put message = 1 transaction
• Queue get single message = 1 transaction
• Queue get message on empty queue = 1 transaction
• Queue batch get of 32 messages = 1 transaction
• Queue delete message = 1 transaction

Scenarios

Having done understanding how the storage transaction charge, considering the following scenarios:

Scenario 1 – Iterating files inside Blob container

An application will organize the blobs in different container per each users. It also allows the users to check size of each container. For that, a function is created to loop through entire files inside the container and return the size in decimal. Now, this functionality is exposed at UI screen. An admin can typically call this function a few times a day.

*Update: Actually, we can use ListBlobs method to get the length / size of files inside the container. But anyway, just forget it at the moment. (Thanks to Jai Haridas for this comment)

Some Figures for Illustration

Assuming the following figures are used for illustration:

• I have 1,000 users.
• I have 10,000 of files in average for each container.
• Admin call this function 5 times a day in average.

How much it costs for Storage Transaction per month?

Remember: a single GetBlob request is considered 1 transaction!

1,000 users X 10,000 files X 5 times query X 30 days = 1,500,000,000 transaction

$ 0.01 per 10,000 transactions X 1,500,000,000 transactions = $ 1,500 per month

Well, that’s not cheap at all.

Tips to Bring it Down

• Verify with the admin if they really need to use the function for 5 times a day? Educate them, tell them that each time this function is being called, it roughly costs $ 10 since it involves 10 million transaction (10,000 files X 1,000 users). I bet the admin will also avoid that if he/she knows the cost.
• Do not expose this functionality as real time query to admin. Considering to automatically run this function once in a day, save the size in somewhere. Just let admin to view the daily result (day by day).

With limiting the admin to just only view once a day, what will be the monthly cost looks like:

1,000 users X 10,000 files X 1 times query X 30 days = 300,000,000 transaction

$ 0.01 per 10,000 transactions X 300,000,000  transactions = $ 300 per month

Well, I think that’s fair enough!

Scenario 2 – Worker Role Constantly Pinging Queue

An application enables user to upload some document for processing. The uploaded document will be processed asynchronously at the backend. When processing is done, the user will get notified by email.

Technically, it uses Queue to store and centralize all tasks. Two instances of web roles to take the input and store task as message inside the Queue. On the other hand, 5 instances of Worker Role are provisioned, they will constantly pinging Queue Storage to check if there’s new message to be processed.

The following diagrams illustrates how the architecture may look like.

*icons by http://azuredesignpatterns.com/, David Pallman

Some Figures for Illustration

Assuming the following figures:

• It has 5 instances of Worker Role
• Those Worker Role will constantly get message from Queue (regardless it’s empty or filled)

public override void Run()
{ 
    while (true)
    { 
        CloudQueueMessage msg = queue.GetMessage();
        if (msg != null)
        {  
            // process the message
               }  
    }
}

• Those Worker Role will run 24 hours per day, 30 days per month
• It’s stated here that a single queue is able to process up to 500 messages per second. Let’s assume in average, it will process 200 messages per second (considering some tiny latency between Worker Role and Storage)

How much it costs for Storage Transaction per month?

Remember: a GetMessage on Queue function (regardless empty or filled) is considered 1 transaction

200 req X 60 sec X 60 min X 24 hours X 30 days X 5 instances = 2,592,000,000 transactions

$ 0.01 per 10,000 transactions X 2,592,000,000 transactions = $ 2,592 per month

Tips to Bring it Down  #1

Unless there’s requirement to meet certain number of target, otherwise consider to put some Sleep to especially when you’ve got empty message result for several times.

Assuming we put Thread.Sleep(100) = 0.1 second, which means for every second there will be 10 time polling to the queue to check if there’s message.

public override void Run()
{ 
    while (true)
    {
        CloudQueueMessage msg = queue.GetMessage();
        if (msg != null)
        { 
            // process the message

               }
               else Thread.Sleep(100);
    }
}

With that, how much do you think it will cost for a month?

10 req X 60 sec X 60 min X 24 hours X 30 days X 5 instances = 129,600,000  transactions

$ 0.01 per 10,000 transactions X 129,600,000 transactions = $ 129.6 per month

Well, that’s fair enough.

Tips to Bring it Down #2

When your 5 instances of Worker Role have fetched so many times of empty message, then you should start asking yourself if you really need those 5 instances of Worker Roles?

Scaling them in will not only can bring down the Storage Transaction costs, but also will save you some money on Windows Azure Compute Instances.

*Thanks to Brad Calder for this thought.

Scenario 3 – Be Aware with Turning on Windows Azure Diagnostic

Another hidden scenario that may burst your bill on Storage Transaction is turning on Windows Azure Diagnostic if you do not control it properly.

How Windows Azure Diagnostic Work

Windows Azure Diagnostic collects diagnostic data from your instances and copies it to a Window Azure Storage account (either on blob and table storage). Those diagnostic data (such as log) can indeed help developer for the purpose of monitoring performance and tracing source of failure if exception occurs.

We’ll need to define what kind of log (IIS Logs, Crash Dumps, FREB Logs, Arbitrary log files, Performance Counters, Event Logs, etc.) to be collected and send to Windows Azure Storage either on-schedule-basis or on-demand.

However, if you are not carefully define what you are really need for the diagnostic info, you might end up paying the unexpected bill.

Some Figures for Illustration

Assuming the following figures:

• You a few application that require high processing power of 100 instances
• You apply 5 performance counter logs (Processor% Processor Time, MemoryAvailable Bytes, PhysicalDisk% Disk Time, Network Interface Connection: Bytes Total/sec, Processor Interrupts/sec)
• Performing a schedule transfer for every 5 seconds
• The instance will run 24 hours per day, 30 days per month

How much it costs for Storage Transaction per month?

5 counters X 12 times X 60 min X 24 hours X 30 days X 100 instances = 259,200,000 transactions

$ 0.01 per 10,000 transactions X 129,600,000 transactions = $ 259.2 per month

Tips to Bring it Down #2

Ask yourself again if you really need to monitor all 5 performance counters on every 5 seconds? What if you reduce them to 3 counters and monitor it every 20 seconds?

3 counters X 3 times X 60 min X 24 hours X 30 days X 100 instances = 3,8880,000 transactions

$ 0.01 per 10,000 transactions X 129,600,000 transactions = $ 38.8 per month

You can see how much you save for this numbers. Windows Azure Diagnostic is really needed but use it improperly may cause you paying unnecessary money. It’s double-edge sword, be careful.

Conclusion

To conclude, this article gives you a view of how Transaction Cost of Windows Azure Transaction may lead to costly charge if it’s not properly used. Different component in Windows Azure Platform charges differently, cloud architect should have deep understanding in order to design scalable, reliable, yet cost-effective solution to customer.

In some case where constantly request is requirement, you may also would like to evaluate using SQL Azure instead of Windows Azure Storage because there will no any storage transaction cost in SQL Azure.

Do not worry of using any component if you really need. As long as you architect and design the solution properly, the cost should be reasonable enough.

Hopefully by reading this article, you’ll save some money for storage transaction .