Personal API keys, project-scoped deployments, and a Frankfurt residency clause
A regional insurer has three analytics squads building models in watsonx.ai. Each squad created its own project and its own API key, and every key belongs to a named employee's IBM Cloud account. Last month a squad lead left; her API key was deleted with her account, and two production scoring jobs that authenticated with it began returning 401 errors. Separately, the fraud squad wants to promote a Granite-based classifier to production and share it with the claims team, but deployments created inside a project are invisible to anyone outside that project. Compliance also requires that inference traffic carrying policyholder data stay in the Frankfurt region, while the development team sits in Toronto and copied whatever endpoint URL appeared in the sample notebooks. The platform lead must define a target operating model covering identity, scoping, and regional endpoints before the next release, and wants the decisions documented rather than another round of ad-hoc fixes.
1. What identity design should replace the personal API keys, and what specifically makes the current pattern fragile?
Read the worked answer
The current pattern ties a machine workload's authentication to a human's lifecycle. An IBM Cloud API key is bound to the identity that owns it: when the user is removed, the key is invalidated, and every consumer that reused it fails at once. There is also no clean attribution - audit logs show a person, not a workload.
Replace it with a service ID per environment per squad. A service ID is a non-human identity that owns its own API key and carries explicit IAM access policies scoped to the watsonx.ai service instance (or resource group). Concretely:
- One service ID per environment (dev/test/prod), with least privilege - writer/consumer roles on the specific deployment space, not account-level administrator.
- The access policy must cover both the platform resource and the project/space resource the workload touches; granting only one of the two is a common cause of authorization failures.
- Store the key in a secrets manager, inject at runtime, and rotate on a schedule plus on any suspected exposure.
- Keep human administrator access separate and audited (break-glass), so no workload depends on it.
With that in place, an employee leaving changes nothing about production. The trade-off is more objects to manage, so codify service ID creation and policy bindings in infrastructure-as-code rather than creating them by hand.
2. Where should the fraud model be deployed so the claims team can consume it, and what changes relative to deploying inside the project?
Read the worked answer
Promote the model out of the project into a deployment space. A project is an authoring boundary for collaborators; a space is the collaboration and runtime boundary where deployments live. A deployment created inside a project is visible only to that project's collaborators and is intended for testing.
Steps and consequences:
- Save the trained model as an asset and promote it from the project to a space (or, from notebook code, store the model and create a deployment that references it and the target space ID).
- The space must be associated with the same machine learning service instance used for scoring, and it carries a governance stage (for example pre-production versus production) that the model inventory can track.
- Create an online deployment if claims staff need sub-second scoring per case; use a batch deployment if scoring happens nightly over a file of claims.
- Grant the claims team consumer or viewer roles on the space, not on the fraud squad's project - this is the key isolation win, since they never see training notebooks, data assets, or other models.
- Publish the space URL plus deployment ID as the contract; the scoring payload must match the model's input schema, so document the expected fields.
Net effect: one stable endpoint, multiple consuming teams, and access control that reflects runtime rather than authoring needs.
3. What do you have to verify to satisfy the Frankfurt data-residency clause, and where do hardcoded endpoints typically leak?
Read the worked answer
Residency is a property of the whole path the data travels, not just the model call. Start by confirming the watsonx.ai instance is provisioned in the Frankfurt region, that the project and the production space are associated with that instance, and that runtime code authenticates against that region's endpoint URL. Sample notebooks frequently hardcode a US endpoint; that single string silently sends prompts abroad even when the dashboard says Frankfurt.
Then audit every hop the payload makes:
- The project's Cloud Object Storage bucket, since project and model assets live in regional object storage.
- Any watsonx.data or vector store used for retrieval, if prompts are augmented with documents.
- External embedding, reranking, or third-party APIs, which are the most common accidental cross-region calls.
- Evaluation and monitoring pipelines that persist prompt/response logs for quality review.
Also confirm model availability: not every foundation model is offered in every region, so a model that works in Toronto may need a substitute in Frankfurt - check before promising the claims team a specific model ID.
For evidence, review the account resource list filtered by region, inspect outbound traffic from the scoring job, and add a startup assertion that fails the job if the configured endpoint does not match the expected region. Document the assertion so audit can see it is enforced, not just intended.