A recent comment by Martyn on my cloud performance shoot-out post prompted me to do another round of testing. As the bootstrap process I described on the last post evolved, it’s always a good idea to test it anyway, so why not kill two birds with one stone? The comment suggested that the Amazon EC2 micro instance is CPU throttled, and that after a long period (in computer terms: about 20 seconds according to the comment), you could lose up to 99% of CPU power. Whereas on a small instance, this shouldn’t happen. So is EC2 small going to perform way-better than the micro instance? How is it going to perform against Linode or Rackspace equivalent VPS?
Bootstrap updates
The test was very similar to the last one, and essentially involved running a bootstrap process via fabric, which installs a bunch stuff. Since the last post, the bootstrap process accommodates a bigger software stack, and therefore includes much more of everything (more packages, more compiled code, more scripts). I therefore expected the entire process to take longer on ALL cloud providers. That said, the mix of components still involve IO, CPU and Networking as before, so I think it provides a good balance for comparison.
Tested platforms
Just as the last test, these tests were carried out using the default Debian 6 Squeeze. This time however, on slightly higher spec’d cloud instances on all three providers (that is, not the very entry-level VPS). To be on the safe side, I tested the micro instance again too:
- Rackspace 2048Mb – using the London data centre.
- Linode 2048 – using the London data centre.
- EC2 small instance (EBS volume) – using the Ireland data centre.
- EC2 micro instance (EBS volume) – using the Ireland data centre.
Results
- Rackspace 2048: 2209 seconds (~36 minutes)
- Linode 2048: 2447 seconds (~40 minutes)
- EC2 small: 3430 seconds (~57 minutes)
- EC2 micro: 8723 seconds (2 hours and 25 minutes!!)
Again, this is really far from scientific. Given the time it took for each process to run, I couldn’t repeat it many times to average out any temporary issues. Of course, I might have ended up on a particularly busy or not-so-busy hardware server on each of the providers, so the test won’t necessarily reflect these types of variance.
This time Rackspace came first, running about 10% faster than Linode, but 55% faster than Amazon’s EC2 small. On this run, the gap has certainly narrowed compared to the whopping 378% on the previous test, but Amazon still seems considerably slower. Compared to the EC2 micro, we see a similar difference to the previous results, with Rackspace 2048 running 394% faster than EC2 micro, and EC2 small running 254% faster than micro.
Conclusion
It seems like on the lowest-level cloud instances, Amazon micro really lags behind Linode and Rackspace. Once you raise the bar a little, the gaps seem to narrow, but Amazon EC2 small still seems noticably slower than Rackspace or Linode. Perhaps as Martyn suggested, you can build your systems in such a way that would avoid hitting the CPU throttles on the micro instance. If you want more predictable performance, then it seems like the micro isn’t really suitable, and you’d be much better off with one of the lower-end offers from providers like Rackspace and Linode. As with almost anything in life, YMMV.
