AM and IG: Zero Downtime upgrade strategy using a blue/green deployment
Introduction
The standard deployment for ForgeRock Identity platform consists of multiple ForgeRock products such as IG, AM, IDM and DS. As newer ForgeRock versions are released, deployments using older versions need to be migrated before they reach their end of life. Also, newer versions of ForgeRock provide features such as Intelligent authentication, latest OAuth standards etc which helps businesses to implement complex use-cases
Problem Statement
Traditionally, ForgeRock upgrades are handled via a rolling upgrade strategy using an in-place update. This strategy doesn’t suit all deployments due to below constraints:
- Many deployments don’t allow any downtime. This means production servers can’t be stopped for upgrade purposes.
- Some deployments follow immutable instances approach, this means no modification is allowed on the current running servers.
To resolve these constraints, a parallel deployment approach or a blue/green strategy can be leveraged for upgrading ForgeRock servers.
Solution
This blog provides a high level approach of using blue/green methodology for updating ForgeRock AM servers and its related components like DS-ConfigStore, DS-CTS, AM-Agents and IG servers. We plan to cover similar strategies for DS-UserStores and IDM in future blogs.
In order to upgrade ForgeRock deployment, we need to first analyze the dependencies between various ForgeRock products and their impact on upgrade process:
- What versions of DS/OpenDJ are compatible with AM/OpenAM?
- What versions of Policy Agents are compatible with AM/OpenAM?
- What versions of AM/OpenAM are compatible with IDM/OpenIDM?
Given the dependencies between ForgeRock products, it is generally advisable to upgrade AM before upgrading DS, AM-Agents etc as new versions of AM support older versions of DS, AM-Agents but vice-versa may not be true.
Note: There can be some exceptions to this rule like:
- Web Policy Agents 4.x are compatible with AM 6.0 but not with AM 6.5. This means the order of upgrade shall be existing version to AM 6.0 => AM Agent 4.x to 5.x => AM 6.0 to AM 6.5.x
- If AM-IDM integration is used, then both AM and IDM needs to be upgraded at the same time.
Upgrade Units
ForgeRock platform deployment can be divided into 4 units so that upgrade of these units can be handled individually:
- Unit 1: AM and it’s related stores (DS-Config and DS-CTS)
- Unit 2: AM-Agents / IG
- Unit 3: DS-UserStores
- Unit 4: IDM and its datastore
The order of upgrade used by our approach shall be Unit 1=>Unit 2=>Unit 3=>Unit 4
Unit 1: AM Upgrade
Prerequisites/ Assumptions
- This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade or you are already using a blue/green deployment.
- In the above diagram, the blue cluster reflects an existing AM deployment (like 13.5.x version) and green reflects a new AM deployment (like 6.5.x version)
- There are N+1 AM servers and corresponding Config stores deployed in your existing deployment. This means N servers are used for production load and one server is reserved for maintenance activities like backup, upgrades etc. If there is no such maintenance server, then you may need to remove one server from production cluster (thereby reducing production load capacity) or install an additional node (AM server and corresponding Config store) for this upgrade.
- No sessions in CTS servers are replicated during blue/green switch; therefore, users are expected to re-authenticate after this migration. If your business use-cases require users to remain authenticated, then these sessions (like OAuth Refresh tokens) need to be synced from the old to the new deployment. Mechanisms like ldif export/import or using IDM synchronization engine can be leveraged for the syncing selective tokens from old to new deployment. Also, refer AM release notes on session compatibility across AM versions.
- Review release notes for all AM versions between existing and target AM deployment for new, deprecated features, bug fixes etc. E.g. for OpenAM 13.5 to AM 6.5 upgrade, review the release notes for AM 5.0, 5.5, 6.0 and 6.5 versions.
Upgrade Process
- Unconfigure replication for DS-3 config store. This ensures that the upgrade doesn’t impact existing AM deployment.
- Upgrade AM-3 in-place using AM upgrade process. Note: you may need to handle new AM features in this process like AM 6.5 secrets etc.
- Export Amster configs from AM-3.
- Transform Amster export so that Amster export is aligned with new green deployment such as DS hostname:port etc.
- Install AM, DS-Config and DS-CTS servers. Import Amster export into new green cluster. Note: For certain deployment patterns such as ForgeRock immutable deployment, Amster import needs to be executed for each AM node. If a shared config store is used, then Amster import needs to be executed only once and other nodes are required to be added in existing AM site.
Switchover to new deployment
6. After validating that the new deployment is working fine, switch the load balancer from blue to green. This can also be done in an incremental way. In case of any issues we can always rollback to the blue deployment.
Note: Any configuration changes made after blue’s Amster export should be applied to both blue and green deployments so that no configuration change is lost during switchover or rollback.
Post Go-live
7. Stop AM servers in the blue deployment.
8. Stop Config and CTS DS servers in blue deployment.
9. Deprovision the blue deployment.
Unit 2: AM-Agent/IG Upgrade
AM-Agent
Prerequisites/ Assumptions
- This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade or you are already using a blue/green deployment.
- In the above diagram, the blue cluster reflects an existing AM-Agent deployment and the green reflects new AM-Agent deployment.
- A parallel base green deployment for Protected app servers has already been created.
- Create new Agent profiles for green deployment on AM servers.
- This approach assumes both old and new AM-Agent versions are supported by AM deployment version.
- Refer to the release notes for latest and deprecated features in the new AM-Agent/IG version, like AM-Agent 5.6 release notes.
Upgrade Process
- Install AM-Agents in the green deployment. Update agent profiles on AM server (created in #4 above) for new agents deployed in green deployment to match configurations used in agent profiles from the blue deployment. For certain AM versions, this process can be automated by retrieving existing Agent profiles and using these results to create new Agent profiles.
Switch over to the new deployment
2. After validating that the new deployment is working fine, switch the load balancer from blue to green.
Post Go-live
3. Stop App servers in the blue deployment.
4. Remove blue agent profiles from AM deployment.
5. Deprovision the blue deployment.
IG
Prerequisites/ Assumptions
- This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade or you are already using a blue/green deployment.
- In the above diagram, the blue cluster reflects existing IG deployment and the green reflects the new IG deployment.
- This approach assumes both old and new IG versions are supported by AM deployment version.
- Create new Agent profiles for the green deployment on the AM servers required for IG servers.
- Refer to the release notes for the latest and deprecated features in the new IG version, like IG 6.5 release notes.
Upgrade Process
- Update the IG configs in the git repository as per changes in the new version. You may create a different branch in your repository for the same.
- Deploy the new green IG deployment by leveraging updated configurations.
Switchover to the new deployment
3. After validating that the new deployment is working fine, switch the load balancer from blue to green.
Post Go-live
4. Stop the IG servers in the blue deployment.
5. Deprovision the blue deployment.
Conclusion
Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant approach for minimizing downtime for ForgeRock deployment upgrades. It is always advisable to practice upgrade strategy in lower environments like dev, stage before moving to production environment.
Depending on the complexity of your deployment, there can be multiple things to be considered for these upgrades such as customizations, new FR features, migration to containers etc. It is always recommended to break the entire upgrade process into multiple releases like “base upgrade” followed by “leveraging new features” etc.
