Panorama Multi-Tenant Blast Radius
A managed service provider operates 68 PA-440 and PA-450 firewalls across 14 tenants. Panorama is an HA pair of PAN-5220 appliances with one log collector group per region across three regions. Each tenant has its own device group and template stack; two tenants share a common shared services template. The provider has a 4-hour maintenance window and a change advisory board. A recent change to the shared template caused 11 firewalls to fail commit with object not found errors. The provider rolled back by pushing the previous template, but two tenants lost logging for 40 minutes because their collector group was also modified. The provider wants to reduce blast radius, support tenant-specific overrides, and ensure a failed commit cannot affect other tenants. Current Panorama commit is set to commit all with no validation.
1. Redesign the Panorama hierarchy to reduce blast radius while preserving shared policy.
Read the worked answer
I would separate shared baseline policy from tenant-specific policy using a device group hierarchy: a shared parent device group for common security policy and shared services, and a child device group per tenant for overrides. For templates, use a template stack per tenant with a shared baseline template at the bottom and a tenant template above it, rather than editing the shared template for tenant changes. Use template variables for tenant-specific values such as IP addresses and zones. This preserves shared policy while preventing one tenant change from requiring a shared-object edit.
The two tenants that share a template should still have separate stacks that reference the shared template; they should not share a mutable stack if their change windows differ. I would also remove any tenant-specific objects from the shared device group and place them in the child group. The goal is that a commit for one tenant references only objects in that tenant's branch of the hierarchy, so a missing object fails that tenant's commit rather than 11 firewalls.
2. How should commits and log collector changes be staged to prevent a failed commit from affecting other tenants?
Read the worked answer
I would stop using commit all. Use selective commit and push to a canary firewall or a small tenant first, validate the commit result, and then expand in batches. Before pushing, run configuration validation and check for unresolved references in the target device group and template stack. Keep collector group changes in a separate change step from policy or template changes, because the recent incident combined them and extended the outage.
For log collectors, assign collector groups per tenant or per small tenant cluster rather than sharing a regional group across many tenants. If a regional collector group is required for cost reasons, isolate collector configuration changes to a maintenance window and verify logging on one firewall per tenant before proceeding. Panorama configuration versions and exported configuration backups should be taken before each batch so rollback is a known operation, not an improvisation.
3. What rollback and validation controls would you require after the incident?
Read the worked answer
I would require a pre-change export of Panorama and collector configurations, a documented rollback owner, and a tested rollback procedure that restores both policy and logging. The rollback should be rehearsed in a lab or with a canary tenant, including the time to push the previous template and confirm that collectors are receiving logs.
Validation after each push should include commit status per firewall, unresolved-reference checks, a policy match test for a known flow, and log ingestion verification from each tenant. I would also add change windows per tenant class, a canary stage, and a post-change review that compares the actual blast radius with the planned one. Finally, alert on commit failures and on missing log streams so that a partial failure is detected within minutes rather than after users or compliance teams report it.