Abstract: “We all saw the recent headline: “Obama Inauguration Brings BBC and CNN to a standstill”. In these hard economic times companies are increasingly relying on the Web as their main sales channel. Its use has moved from a convenient option to something endemic. Expectations for reliability (availability & speed) have become much higher. We want the latest updates and headlines, the best sources, and the most answers in the shortest time possible, and with the least amount effort and constraint. Amazon recently estimated that 100ms of latency cost them 1% of their book sales. As such, Web performance has become the new customer service.
Understanding what the user experience will be before you "go live" is more and more critical. Performance or stress testing is no longer an option, it’s an imperative. In this session we will discuss the value of Cloud Testing and real world case studies of customers using it to ensure Web reliability and performance.”
Tom Lounibos is the President and CEO of SOASTA, the company behind CloudTest.
Tom is talking about consequences of reliability issues. Interesting to note recent reports where latency is now being measured in lost sales:
- Amazon loses up to 1% of Sales with 100ms of latency on Site
- Google loses 20% Traffic with .5 second of Latency
Calls out recent crash headlines: 40 million users crash Tata Motors site, J. Crew Website Crashes due to people searching for Obama Items, 11-hour Netflix outage
Root Causes of Poor Performance: Application – Network – Environment:
- Sudden Increase in User Activity
- Sustained Increased in User Activity
- New Application Features
- Changes in your Deployment Environment
- Changes in your Application Software
- Changes in an External Component Used
Web Applications are Built Differently: internal & external components assembled into a highly scalable web service. Tom gives example of an application tying in Facebook, PayPal, external content distribution network and internal database.
Traditional Approach – Internal Testing
- Limited ability to test at web scale
- limited ability to simulate real world user activity
- limited ability to test external components
- limited ability to test firewalls and load balancers
- incredibly expensive: hardware-software-people
- incredibly slow process: 8 week advance notice
Cloud Testing – externally load testing your web application and networks with real world user traffic; your web application (being tested) can be behind your firewall, at a managed hosting provider, or deployed in the cloud. The “cloud testing” part is using the cloud to produce real world load (geographic and volume) against your web application. The testers are virtual users running actual scenarios, not just hits.
Type of Cloud Tests:
- Stress Test – where will application break, find stress point
- Load Test
- Performance
- Latency Test
- Browser Performance
Cloud Testing is delivered on pay-per-use model, pay for test hours used
Tom did an admitted mini-commercial for SOASTA, his company. Now, he’s giving customer case studies. First customer, TurboTax to understand how the tax filing site would perform on tax day. Spoke of issue a couple of years ago that resulted in the IRS extending tax filing deadline.
I snapped some screen shots of the TurboTax case study slides:
Next example, Hallmark.com for the e-card superbowl, Valentine’s Day. This test plan shows both ramp-up and ramp-down. Can cause application problems if you ramp down too quickly. Need to see how application responds. Another example, QTRAX, a new online music service. QTRAX test plan (and execution) went up to 500k concurrent users.
In closing, Tom asks and answers: Why Cloud Test? Achieve Web Scale. Real world test. Easy-fast-affordable.
[Disclosure: SOASTA has done business with my company, Elemental Links, but is not a current client.]
PDF 
