When we launched our services in EU in 2012, we doubled our deployment environments from two to four: Test and Production, US-East and EU-West. Similarly, our decision to deploy our North American services in Active-Active mode increased deployment environments to six — adding Test and Production in the US-West region. While for any individual deployment we utilize Asgard — an extremely powerful and flexible deployment and configuration console, we quickly realized that our developers should not have to go through a sequence of at least 6 (some applications have more “flavors” that they support) manual deployment steps in order to keep their services consistent through all the regions.

To make the multi-regional deployment process more automated, our Tools team developed a workflow tool called Mimir, based on our open source Glisten workflow language, that allows developers to define multi-regional deployment targets and specifies rules of how and when to deploy. This, combined with automated canary analysis and automated rollback procedures allows applications to be automatically deployed in several places as a staged sequence of operations. Typically we wait for many hours between regional updates, so we can catch any problems before we deploy them world-wide.