The Catalog That Ate GitHub's Rate Limit
Fabrikam's catalog is fed by a script that writes 450 type: url locations into a single locations.yaml, one per repository. Every five minutes the catalog refreshes all of them. The GitHub token is a personal access token with 5,000 requests per hour. The catalog has 9,800 entities and a processing queue that regularly reaches 12,000 tasks. Users report stale entities and HttpError: API rate limit exceeded in backend logs. Only 120 of the 450 repositories actually contain catalog-info.yaml; the rest return 404 on every refresh. Some teams also register the same component manually through the catalog UI, producing duplicate ownership displays. The organization wants new repositories discovered automatically within 30 minutes, without asking teams to edit a central file, and wants removed repositories to disappear from the catalog.
1. Replace the script with a catalog provider. What provider configuration and schedule would you use, and what would you do about the 330 repositories without catalog-info?
Read the worked answer
Use the GitHub catalog discovery provider under catalog.providers.github with the organization, catalogPath: /catalog-info.yaml, a branch filter for the default branch, and a 30-minute schedule with a short timeout. Authenticate with a GitHub App installation rather than a personal token, so the limit is tied to the installation and does not disappear when an employee leaves. The 330 missing files are normal probes; lower the frequency and, if your provider version supports repository or topic filters, use them to narrow the scan. Do not keep a central generated locations file as the primary source. Keep catalog.locations for exceptions outside the organization, and let the provider own discovery and deletion. Test the schedule against a small organization first and watch the request count per crawl.
2. Two registration paths are creating duplicate entities. Explain why and how you would resolve it without losing ownership data.
Read the worked answer
The duplicates come from two locations producing the same entity ref: the static type: url location and the provider both emit component:default/<name>. The catalog does not merge two competing definitions; you get conflicts, last-writer behavior, or duplicate ownership displays. Pick one source of truth. Freeze the script, migrate ownership and annotations into each repository's catalog-info.yaml, then remove the static locations in batches. Verify backstage.io/managed-by-location points at the provider after each batch. If a team needs an exception, register it only through a location with a clear owner and exclude it from discovery. Preserve historical ownership by diffing the old location metadata against the repository file before deletion, and keep a rollback copy of the generated locations file during migration.
3. How do you make removed repositories disappear, and what is the risk in doing so?
Read the worked answer
Deletion is controlled by the provider's emitted mutations plus the catalog orphan strategy. For migration, start with the orphan strategy set to keep so a provider failure does not wipe entities; once you trust the provider, switch to delete so repositories removed from the organization disappear. Test with a staging organization and a deliberately failed provider run. The risk is a partial or failed crawl being interpreted as mass deletion; monitor the number of deletions per run and alert on a threshold. Also check that renamed repositories do not appear as delete-plus-create in the same cycle, because that can break ownership and relations until the next crawl. Keep a database backup before the first delete-enabled run.