Should you upgrade from App Service Standard Plan to Premium v2?

Updated (17 August 2020)!

Things change fast the cloud!
I mentioned in the original post that redeployment was required to move to Premium v2. Now, glad to see that the App Service Engineering team has simplified the experience so that you now can move to Premium v2 just like other Tiers without redeployment.
However, there’s a catch to that: “The IP addresses (both inbound and outbound) will change, as can be seen in the diagram below.

IP addresses will change when upgrade to Premium V2

IP addresses will change when upgrade to Premium v2

As such, you may need to make necessary arrangement such as update the ip address whitelist should there be any dependency over it.

Background

App Service Premium (v2) Plan has been made generally available since Oct 2017. This premium plan which uses Dv2-series VM promises faster processors, SSD storage, and double the memory-to-core ratio compared to the older plan which uses “the legacy” A-series VM. However, i can’t find any article or exercise showing the actual comparison, how much the improvement would bring. And for this reason, i decide to perform an experiment and write this blog.

At a glance by peeking at Azure App Service pricing page, comparing the core count seems like bringing the Pv2 to a higher price point. As such, it might be a disappointment for the existing customers which are looking to upgrade.

The big questions for the current App Service customers are:

  • “Should we upgrade to this new plan?”
  • “Is the upgrade worth the effort? Note that it will be a re-deploy as there is no One-Click-Upgrade for this.”

The objective of this article is to demystify that it’s actually more cost-effective in conjunction with the performance. It will be backed with my synthetically load testing result.

Here’s a specs and pricing screenshot from the App Service pricing page:

Load Test Result of Test #1

As comparison, shall should we compared S2 with P1v2 or P2v2? I’ll be comparing S2 with P1v2 in this article since it offers at the same price point.

Methodology: how the tests were being performed

  • I used cloud-based load testing of Azure DevOps (aka VSTS)
  • The app used for this exercise was NopCommerce (ASP.NET based) which is available from the marketplace.
  • The architecture is rather simple, WebApp accessing SQL Database.
  • The first app (let’s call it NopS2App run on S2 App Service Plan: 2 core CPU and 3.5 GB RAM); It accessed SQL database with GP_Gen5_2 spec (2 vCores; 32 GB RAM)
  • The second app (let’s call it NopP1v2App run on P1v2 App Service Plan: 1 core CPU, 3.5 GB RAM, and SSD storage); Same with the first app, the second app also accessed SQL database with GP_Gen5_2 spec (2 vCores; 32 GB RAM)
  • I will be capturing 2 sets of outputs: First, performance under load including: number of successful request and RPS. I would consider these as my primary measures. Second, the memory and CPU percentage of the app service plan during load testing period. These are more for informational measure as we’d like to observe how the utilization look like during the load tests.
  • For a fair comparison, both of the apps are freshly deployed own their own resource group with identical setup.

Test #1

This is a very simple test as i just load an URL endpoint https://app.azurewebsites.net/camera-photo (which showed a product category)

  • Duration of test: 5 minutes
  • Number of user: 250 users
  • Load was generated from Southeast Asia Azure region

Here’s the result:

Load Test Result of Test #1
Load Test Result of Test #1
Load Test Result of Test #1

You’ll notice that P1v2 would be able to handle about 27% more successful request than S2. Likewise, about 27% more RPS than S2.

Test #2

Test #2 is similar to #1 except with higher load and slightly longer duration.

  • Duration of test: 10 minutes
  • Number of user: 500 users
  • Load was generated from Southeast Asia Azure region

Result:

Load Test Result of Test #1
Load Test Result of Test #1
Load Test Result of Test #1

You’ll notice that P1v2 would be able to handle about 35% more successful request than S2. Likewise, about 34% more RPS than S2. It’s more significant improvement than Test #1.

Test #3

Test #3 and #4 are different from #1 and #2, in the sense that i used Visual Studio Web Test and Load Test to record a scenario, basically to simulate a user who is trying to browse a product and eventually check out and purchase.

  • Duration of test: 5 minutes
  • Number of user: start with 10 users; with the increment of additional 10 users per 10 seconds, till maximum of 200 users.
  • Load was generated from East US 2 Azure region
Load Test Result of Test #1
Load Test Result of Test #1
Load Test Result of Test #1

As can be seen, we see about 18% improvement.

Test #4

Test #4 used the same scenario as #3 except with a longer duration and higher load.

  • Duration of test: 10 minutes
  • Number of user: start with 50 users; with the increment of additional 20 users per 5 seconds, till maximum of 500 users.
  • Load was generated from East US 2 Azure region

Let’s review the result.

Load Test Result of Test #1
Load Test Result of Test #1
Load Test Result of Test #1

Test #4 shows about 4% improvement.

Conclusion

By doing 4 round of tests, we could conclude the following:

  1. We’ve seen the RPS and Successful request improvement ranging from 4%–35%.
  2. We observe that both Memory and CPU consumed slightly increased about 7%–18%, except the CPU for both Test #3 and #4. I think this is very reasonable.
  3. Obviously this is synthetic test and may very depending on other factors such as the application, etc.

I hope this exercise would give you more confidence to upgrade to Premium v2 App Service plan.

This entry was posted in App Services, Azure, Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.