FleetKeeper, part 2: rings, exclusions and automatic stops

Part 2 of the FleetKeeper series. Start with part 1 if you missed the loop.

The scenario

The remediation passes on every test machine. Then it reaches production, where some devices have apparently not read the test plan. I would still ask whether the script is right, but the more useful operational question is how much damage a wrong answer can do.

FleetKeeper puts several controls around the script. Rings choose who goes first and waves set the opening dates. Exclusions keep devices out. If reported errors cross the configured limit, auto-halt stops the feature. These controls do different jobs; one cannot stand in for all the others.

Rings and waves, resolved on the server

Start with the feature catalog. This is where the ring policy lives:

// Tenant-specific group references and rollout timestamps are redacted.
"ringPolicy": {
  "rings": [
    { "name": "canary", "groupId": "<canary-group>", "priority": 0 },
    { "name": "pilot",  "groupId": "<pilot-group>", "priority": 1, "startsAt": "<wave-start>" },
    { "name": "prod",   "groupId": "<production-group>", "priority": 10, "startsAt": "<wave-start>" }
  ],
  "default": "prod",
  "overrides": {},
  "cacheTtlMinutes": 60
}

Rings map to Entra groups. Add startsAt and the ring becomes a dated wave. The endpoint does not know its ring; on each plan call the server resolves it. An explicit override wins first, then transitive Entra group membership with the lowest priority number, then the default.

The less obvious behavior is in failure and future-wave handling:

catch (Exception ex)
{
    _logger.LogWarning(ex, "[RING] Graph lookup failed for ring '{Ring}' group '{Group}' — skipping",
        ring.Name, ring.GroupId);
    continue;
}
...
// Wave-plan gate: ring not yet active → fall through to next priority.
if (ring.StartsAt is { } start && start > now)
{
    skippedFuture.Add($"{ring.Name} (starts {start:u})");
    continue;
}

A Graph failure does not throw at the endpoint; the lookup degrades to the default. A ring whose wave has not opened is skipped, letting resolution fall through instead of activating it early. There is a final check too: if the default ring has a future date, the resolver returns no ring. The default is not permission to ignore the calendar.

Membership snapshots have a cache TTL. A typical plan call can use a dictionary lookup rather than spend a Graph round trip, at the cost of seeing group changes up to one cache TTL late. I would keep that delay in the rollout procedure. Otherwise somebody will move a device between groups and immediately declare the policy broken.

The gates

Finding a problem does not grant permission to fix it. At remediation-plan time, each pending remediation still has gates to pass:

Figure 1: the gates applied at plan time.

if (halted.Contains(entry.Name))
{
    _logger.LogInformation("[PLAN-GATE] Suppressing {Feature} for {Device}: auto-halt active", state.RowKey, deviceId);
    continue;
}
...
// Blob-backed exclusions.json (VIPs added AFTER the flag was set are protected here)
if (await _exclusionStore.IsExcludedAsync(deviceId, context?.AzureAdDeviceId, entry.Name))
{
    _logger.LogInformation("[PLAN-GATE] Suppressing {Feature} for {Device}: exclusions.json", state.RowKey, deviceId);
    continue;
}
...
if (state.RemediationGivenUp)
{
    _logger.LogInformation("[PLAN-GATE] Suppressing {Feature} for {Device}: remediation given up", state.RowKey, deviceId);
    continue;
}

Notice when the exclusion check runs. A device added to the list after being flagged is still protected because the planner checks again, not just when the flag was raised. It also rechecks the target filter; a device outside current assignment rules is left out of the next plan. Each suppression writes a reason in a [PLAN-GATE] log line. For me, that reason is essential: “it did not run” otherwise starts a very long meeting.

The big red button, automated

Each feature can declare auto-halt policy in the catalog. The defaults are a 20% error-rate threshold, a 60-minute window, and at least 50 samples. Reported outcomes feed the feature's counter:

entity.SampleCount++;
if (isFailure) entity.FailureCount++;

if (!entity.Halted && entity.SampleCount >= Math.Max(1, policy.MinSamples))
{
    var rate = entity.SampleCount == 0 ? 0 : entity.FailureCount * 100 / entity.SampleCount;
    if (rate >= Math.Clamp(policy.ErrorRateThresholdPercent, 1, 100))
    {
        entity.Halted = true;
        entity.HaltedAt = now;
        entity.HaltedReason =
            $"auto-halt: {entity.FailureCount}/{entity.SampleCount} failures ({rate}%) over {policy.WindowMinutes}m window";
    }
}

The minSamples guard means one unlucky device does not immediately halt the feature. Once halted, the first plan gate suppresses it for everyone, with the trigger recorded in the reason string. It stays halted until a human investigates, fixes the issue, and resumes it.

The input deserves attention as well as the threshold. Outcomes are device-reported and counted per feature, not weighted by device. A faulty or compromised client can influence that fleet-wide error rate. Monitor the signal and make recovery deliberate. I would rather require a person to resume than let the system decide that yesterday's failure has probably become less embarrassing overnight.

On-demand actions, without a back door

Service Desk actions and the preview AI-agent MCP integration use a per-device action queue, merged into the plan. This deliberately bypasses groups and rings for an explicitly requested action. It does not bypass the safety gates above.

The HTTP and MCP functions have a read-only Graph role for looking up devices and state. A separate function holds the privileged role and calls Graph to trigger on-demand remediation, with sync as fallback. That function is not internet-exposed; it takes work from Service Bus.

This limits MCP's direct Graph privileges. It does not make a queued request harmless. The project's MCP guidance requires the preview endpoint to be gated with the MCP system key and either Entra Easy Auth or an APIM OAuth scope. A caller-supplied actor is only advisory until derived from a validated auth token. Authorize each request and scope it to the intended device: the privileged backend executes accepted work outside normal ring/group targeting. The privilege split alone does not establish prompt-injection resistance.

Limits worth stating

• Auto-halt belongs to the feature, not a ring. A bad remediation stops for everyone, intentionally. A failing canary does not by itself keep production closed; wave dates do that job.
• Reported outcomes are counted globally for the feature. Before resuming after a burst of failures, inspect where the reports came from and how many samples there were.
• Group-based rings need the service identity's Graph application permissions, and their view of Entra membership is only as fresh as the cache TTL.
• MCP remains a preview integration, not a solved prompt-injection boundary. Deployment still needs authentication, authorization, and per-device scope.

Next

The rollout controls sit on top of an infrastructure and identity model. Part 3 follows that Azure side, including mutual TLS trust and the supply-chain checks for distributed scripts.