Hoping to see you on Feb 24th!

https://www1.gotomeeting.com/register/912658288

When discussing this with one of Nolio’s customers, a large online bank operation, they told me that in the past, in cases of a ’service unreachable’ alert with green servers, they didn’t really know what the source of the problem was. The first thing they would do when troubleshooting this type of problems is look into their monitoring system and check if there are problematic servers, but in most cases the issue is in the application layer, where some services or processes are stalled.

Next, they would start manually going into each server to try and figure out where the application failure was – a time-consuming task since they had over 350 servers to check! They would go into each application tier and simulate an application connectivity to the next relevant tier. They needed to know exactly which application tier on each server was connected to each of the other servers from the next tier. It was very complicated to go through dozens or hundreds of servers to find the problem, especially when the operations team was under heavy pressure to get things up and running.

These days, using Nolio, they have created an automated process that “travels” through their different application tiers and initiates connectivity to the next appropriate tier. For example: “go into all the web servers and open a web service (URL) located in the relevant application server that services that web server. Then go into all the application servers and query the relevant DB servers.” This way they test where the problems are so they can find them in minutes.

Nolio Automation Center enabled this customer to create an automated process that discovers where the problems are, but they still needed to go into each problematic server and reset the problematic service. They wanted to automate that as well, so they modified this process to fix this problem automatically.

The addition was very simple. All they needed to do was reset the next tier of service. For example: if the automation process has found that a web server cannot initiate a web service from its relevant application server, it resets the web service in the application server. So when the URL initiation action fails in the web server, the other action that resets the web service is triggered in the application server.

Time savings to this customer by using automation in their problem resolution process are dramatic. The overall time it took the customer to create such a process in Nolio was just 20 minutes. Now the time it takes them to solve this situation automatically is 5 minutes, over all 350 servers they have! Before they started using Nolio, the entire operations team usually spent 1.5 hours of hysteric work doing the same thing. Now this process is handled by just one person

What’s next? They are now integrating this process into their application monitoring system. They want it to be activated automatically when the ‘service unavailable alert’ is triggered. This is also very simple to do, since the Nolio system exposes a web service and command line APIs to enable that.

Once this is implemented, they will get an email from the Nolio system that the process was activated and finished successfully and then they can close the alert ticket within few minutes after it was triggered, without any manual work or team hysteria.

Post written by Alon Eizenman, CTO, Nolio

—
Nolio Application Service Automation is a software platform for designing and executing automated application service workflows across the data center, enabling reliable, effective processes for the management of application change.

One of the biggest appeals of the SaaS model is that upgrades are seamless and transparent. You log off in the evening with one version and the next morning when you log in, voila! you have an upgraded version with all the bug fixes and new features.

What happens in the background is another story. Scary sometimes. A Midnight Summer’s Nightmare.

If you, a SaaS company, have your SLAs in place and adhere to them, your full upgrade windows (i.e. planned downtime) are probably few and far between. Some SLAs allow for a periodic one hour downtime with a week’s notification, but anything longer than one hour will require an earlier notification. You must ensure that the upgrade is performed within that window, or any minute beyond that will start counting against your SLAs. You do not want to extend that window unnecessarily, as your customers are counting on your 24X7 service.

Upon receiving the upgrade instructions from Engineering, you will need to start building a plan. Often, the plan will include dozens, or even hundreds of short steps, and complex dependencies. In order to ensure that the upgrade will be performed correctly and within the allotted time frame you should be able to answer the following questions:

• How long will it take the team to run the upgrade?
• What happens if you need to rollback?
• What is the point of no return, when you must rollback without violating the change window?

A mature SaaS operations group will likely have a Staging environment, closely mimicking production. The best way to go about planning and executing the upgrade is to perform the process on the Staging environment. You will need to carefully design the procedure and run it multiple times until you perfect the process, taking note of how long it takes and practicing the rollback.

This is a tedious process, treasured by engineers as much as they love deciphering undocumented legacy code. But it is essential to ensure success.

So you need to make the process as short as possible and measurable. If you had the tools to automate the procedure you could reduce the time of execution (and of the dry runs on Staging). You could know exactly how long it will take and have the needed repeatability. Repeatability cannot be overrated when encroaching upon the Holy of Holies, under the pressure of timetables and the short temper of the anxious COO/VP Ops.

Since upgrades are an integral part of a SaaS operation and one of the main causes of the gray hair, ulcers, I wanna go home factor, you would do yourself and your customers a great service if you had the tools to automate the entire process.

To deliver real-time option pricing, SuperDerivatives delivers 15 applications, operating across 100 servers, distributed over 8 global data centers. Each application requires weekly updates: content changes, new features and bug fixes.

Prior to adopting Nolio’s solution, these weekly changes involved R&D, QA and Operations and were done by executing a large number of error-prone manual service tasks. This obviously consumed a large chunk of SuperDerivatives’ web operation team’s time.

SuperDerivatives desperately needed to streamline application changes and eliminate misconfiguration errors.

Nolio’s Application Service Automation enabled SuperDerivatives to eliminate the use of scripts, affording a highly streamlined, error-free process for deploying application changes.

Matty Rosen, SuperDerivatives’ Head of Data Center Operations says, “Our customers depend on our hosted financial tools. Managing application consistency across our 8 global data centers, and our ability to reliably execute application changes in a timely manner, is therefor essential for our business.”

Rosen adds, “In addition to Nolio’s measurable productivity benefits, translating into labor cost savings, the ability to establish and enforce best practices across operations and all departments involved in application production, including R&D and QA, is Nolio’s intangible value of heightened application quality and manageability.”

SuperDerivatives experienced dramatic productivity gains once it adopted the Nolio solution for application change automation:

• Eliminating scripting reduced labor overhead from 2 days to just 30 minutes.
• Labor overhead for weekly updates has been reduced from 3 days involving 3 employees to JUST 1 HOUR.
• Labor for troubleshooting tasks has been reduced from 10 hours to 15 minutes.
• Application service auditing, not possible prior to Nolio, is now provided on demand.

Visit us at www.noliosoft.com to see how Nolio can dramatically improve your data center application operations.