Multisite Application Initialization for Zero Downtime Azure Deployments

Obtaining Zero Downtime Deployments using Azure PaaS for Sitecore is achievable, even with a Multisite configuration, but requires some steps to avoid 500 errors and other aspects that may place your Azure Application Service Sitecore instances into rotation before they are ready. This article assumes you:

• Have a Target (aka, Production) and Source (aka, Standby) Deployment Slots in your Azure PaaS Application Service
• Have at least two instances in your Target (aka, Production) Deployment Slot in an Azure Application Service
• Are running Sitecore 10x in a XM configuration (not to be confused with Sitecore XM Cloud which is SaaS based)

How a Swap is Supposed to Work

When swapping two Deployment Slots, the essential/high level flow is as follows:

• Copy Target (Prod) Slot Settings to Source Slot >
• Restart Source Slot (Standby) Instances to Apply Settings from Target (Prod) >
• Warm Up Source Slot Instances >
• Swap to Target Slot (Prod ) After Warmup Completes >
• Copy Source Slot (Standby) Settings over Source Slot that Has the Code Swapped in From Prod >
• Restart Instances in the Source Slot to Apply Settings

During this process, the Environmental Variables are frozen on both the Target and Source Slots as they are being held/applied to each other in a sequence where Prod Settings come to Source as Source will become the new Prod… and Source Settings will be applied to what was in Prod before you Swapped.

If you are able to access the Configuration and Management section under Diagnose and Solve Problems of your Application Service, you can “Check Swap Operations”. Note the two sets of restarts. Both of the restarts happen in the Source Slot so the Target Slot (Prod) is never restarted:

• Target Slot (Prod) Settings are applied in the Source Slot triggering a restart before Swapping what is in the Source Slot into the Target so that the Target/Prod maintains its pre-Swap settings
• Source Slot (Standby) Settings are applied in the Source Slot over what was swapped in from Target/Prod triggering a restart so the Source/Standby slot maintains its pre-swap settings

We will discuss the potential issues that occur in this step later in this article: “Swap to Target Slot (Prod ) After Warmup Completes >”

Per Microsoft, the highly detailed view of this process is as follows: https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots

1. Apply the following settings from the target slot (for example, the production slot) to all instances of the source slot:
   • Slot-specific app settings and connection strings, if applicable.Continuous deployment settings, if enabled.App Service authentication settings, if enabled.
   Any of these cases trigger all instances in the source slot to restart. During swap with preview, this marks the end of the first phase. The swap operation is paused, and you can validate that the source slot works correctly with the target slot’s settings.
   
2. Wait for every instance in the source slot to complete its restart. If any instance fails to restart, the swap operation reverts all changes to the source slot and stops the operation.
   
3. If local cache is enabled, trigger local cache initialization by making an HTTP request to the application root (“/”) on each instance of the source slot. Wait until each instance returns any HTTP response. Local cache initialization causes another restart on each instance.
   
4. If auto swap is enabled with custom warm-up, trigger Application Initiation by making an HTTP request to the application root (“/”) on each instance of the source slot.If applicationInitialization isn’t specified, trigger an HTTP request to the application root of the source slot on each instance. If an instance returns any HTTP response, it’s considered to be warmed up.
   
5. If all instances on the source slot are warmed up successfully, swap the two slots by switching the routing rules for the two slots. After this step, the target slot (for example, the production slot) has the app that’s previously warmed up in the source slot.
   
6. Now that the source slot has the pre-swap app previously in the target slot, perform the same operation by applying all settings and restarting the instances.

https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots

Why is a Swap Not Working

Typical issues that prevent a Swap operation from being Zero Downtime is as follows:

• Azure puts instances into rotation even with a 500 status code as Sitecore takes a while to warmup and may generate a 500 error.
  • A bug request has been raised with Microsoft to address why it places instances into rotation that have an error code, understanding that Step 4 of above states “If an instance returns any HTTP response, it’s considered to be warmed up.” a 500 error indicates a non-warmed up app
• Lack of Application Initialization in your web.config
• You have a Rewrite rule in your web.config to redirect all pages from http to https, and since Application Initialization requires http, you need an https ignore in the rewrite rules
• Missing an Azure App Setting to only allow instances that are 200 in their status code

Multisite Application Initialization Including Sub Pages

In a Sitecore Multisite solution with Deployment Slots, it can be challenging to execute Application Initialization as the underlying URL changes on Swap. Example, the Target Slot (Prod) has mysites.com/site1, mysites.com/site2, etc. as URLs and the Source Slot (Standby) has standby.mysites.com/site1, standby.mysites.com/site2, etc.

NOTE: the use of trailing slashes to differentiate sites. If you use subdomains (ex. site1.mysites.com), the same logic below applies.

Because of these URL changes on Swap, I prefer to use the AzureWebsites.net address attached to the Slot for resolution instead of selecting a host in the Application Initialization. This is essentially “localhost”.

Here is a sample Application Initialization that uses the Site Definition names to ping the separate sites and pages. Note the following:

• Adding ?sc_site then your Site Definition name pulls up the desired sub/multi site
• For subsequent pages to hit, it follows the Content Tree, so if you have a URL pattern where it’s something like mysites.com/site1/page1, you don’t add site1 in the pathing for an initializationPage… you just add the path like /page1/?sc_site=mysite1

<!-- Application Initialization to Ensure the App Instance Warms Up on a CD before being swapped in or added via Azure Auto Scale
    -->
<system.webServer>
    <applicationInitialization doAppInitAfterRestart="true">       
        <add initializationPage="/?sc_site=website" /> 
        <add initializationPage="/?sc_site=mysite1" />               
        <add initializationPage="/page1/?sc_site=mysite1" />
        <add initializationPage="/page1/subpage1/?sc_site=mysite1" />
        <add initializationPage="/page1/subpage2/?sc_site=mysite1" />
        <add initializationPage="/page2/?sc_site=mysite1" />
        <add initializationPage="/?sc_site=mysite2" />               
        <add initializationPage="/page1/?sc_site=mysite2" />
        <add initializationPage="/page1/subpage1/?sc_site=mysite2" />
        <add initializationPage="/page1/subpage2/?sc_site=mysite2" />
        <add initializationPage="/page2/?sc_site=mysite2" />                     
    </applicationInitialization>
</system.webServer>

Rewrite Rules to Skip HTTPS for Application Initialization

As its very common to redirect pages from http to https, but Application Initialization runs over http, we need to add an exception when an http to https rule is being processed. The two conditions we add are as follows:

<add input="{WARMUP_REQUEST}" pattern="1" negate="true" />
<add input="{REMOTE_ADDR}" pattern="^100?\." negate="true" />

Detailed information on these and other settings can be found at: Azure App Service Deployment Slots Tips and Tricks – RuslanY Blog

Adding these as conditions to ensure Application Initialization runs on http ends up looking like:

        <rule name="Root Hit Force HTTPS Redirection" enabled="true" stopProcessing="true">
          <match url="^$" ignoreCase="false" />
          <conditions>
            <add input="{WARMUP_REQUEST}" pattern="1" negate="true" />
            <add input="{REMOTE_ADDR}" pattern="^100?\." negate="true" />
            <add input="{HTTPS}" pattern="^OFF$" />
            <add input="{HTTP_METHOD}" pattern="GET" />
          </conditions>
          <action type="Redirect" url="https://{HTTP_HOST}/" redirectType="Permanent" />
        </rule>

Azure App Setting: WEBSITE_SWAP_WARMUP_PING_STATUSES and WEBSITE_SWAP_WARMUP_PING_PATH

Since Azure will throw instances into rotation on Swap with any status code, we need a way to override this behavior (until hopefully Microsoft fixes this issue or provides more controls). To that end, we can use the following App Settings:

WEBSITE_SWAP_WARMUP_PING_PATH

Set this as the path for the warmup request, such as /healthz/live/

WEBSITE_SWAP_WARMUP_PING_STATUSES

You will want to set the value of this to 200 (or whatever status code you desire for your warmed up state).

These settings must be on the Target Slot as those settings are copied down to the Source Slot before a restart. If you don’t have this is in place, it may allow instances that are not warmed up or in a 500 status to be placed into rotation.

Important: Applying an App Setting will restart your Deployment Slot, so coordinate the appropriate time to apply the setting to minimize impact on your users as the Target/Prod slot will be restarted for all instances.