After years of hearing how the cloud would revolutionize everything, our primary systems were still on physical servers. Real world testing had shown that the physical servers provided better performance for our applications and cost less over a three year period. We had our testing environment running on AWS and were disappointed with the performance.
That was fine, until season hit and we experienced a large surge in traffic that caused our site to be brought to its knees. Response times averaged 30 seconds and dollars were going down the drain. Unable to quickly determine the cause, we turned to a seemingly simple solution: build a small server farm in the Rackspace Cloud. I say simple because it really was simple - it took us less than 6 hours to build the farm. Our application was designed to scale to a farm on demand, but we had never needed to do so. We used the opensource Nginx web server in front of our IIS servers to load balance the traffic. This was an amazing solution that cost relatively little to implement. In addition, it was the first real use of the cloud we'd seen value in – we could scale up or down in real time. With this solution, our downtime was limited to six hours. That's six more hours too many, but looking back, I'm quite happy that it was not any more than that It took us several weeks to determine the real cause that brought the site down, but by that time, season was over.
We encountered two major issues implementing a server farm: The cloud load balancer offering cannot talk to your physical network (unlike cloud servers) and the time on the Next Generation Windows cloud servers can go haywire.
Cloud Load Balancer Alternatives
Our company utilizes both physical and cloud services from Rackspace. Through their RackConnect program, these two networks can communicate directly with each other. They handle the setup and make it happen seamlessly. Their cloud load balancing product also claims RackConnect support, but it cannot communicate with physical network. In addition, the lead time and cost of a hardware load balancer were extreme. Frustrated, we decided to take a stab at Nginx.
Nginx is an amazing opensource web server that can act as a reverse proxy / load balancer. The design is different than Apache or IIS; it's designed to be asynchronous whereas IIS and Apache are both process based. I'm oversimplifying the significance of this, but you can find plenty of articles on the subject. Each has its purpose depending on your needs. Nginx can detect failed nodes, cache backend responses, and even be configured to act differently based on the request path. The Rackspace Cloud allows you to power this box for pennies and gives you more flexibility than their standard load balancer. Tip: focus on the network card throughput more than the cpu / ram offering. Nginx is very light on resources, but network throughput was very important for us.
The Clock is Wrong?
It took weeks for this issue to crop up, but one day two of our servers died without reason. The boxes came back up without any issue, but the clock was off by 12 hours. We could not find any reason for this to happen other than the hardware clock getting reset. The solution was to call “w32tm /resync” on boot and every day afterwards. This forces the server to synchronize the clock with the configured time server immediately. For some odd reason, the boxes do not come configured to use Rackspace's internal NTP servers, but you can configure them to do so. I'm not sure why Rackspace doesn't address this internally since time affects many services – in our case, authentication for WCF services.
I hope these tips help anyone who is starting their organization off with Rackspace cloud. Compared to AWS, the interface is much simpler to use. We also found the cpu cycle to be more generous with Rackspace's offerings. It's not scientific and your mileage may vary. Unfortunately, right now there are two interfaces and getting to the correct one is like navigating a maze. There are other issues such as the interface telling you to connect to your windows box via ssh. Hopefully the team will clean it up soon.
Overall, Rackspace has a solid cloud offering. I would recommend them to anyone. If you select their managed service option, you will be suprised!
Update - 02/19/2013
As usual, within minute of publishing this story, Rackspace responded. I'm sure they will polish it quickly!