How Feature Flags Help Build a Software Factory

Let's have a look at the usual procedure for development. First is planning, then clarifying the requirements, creating tasks in tracker, and finally developing. As soon as the tasks are ready, they are deployed to the test environment for verification, and the release branch is stabilized. Then, at last, comes the release, and the team can finally get real feedback from actual users.In the world of modern development, where the Time-to-Market indicator is crucial, this is unwieldy. It is one of the key indicators of DevOps that allows you to maintain the IT-products competitiveness – the period required to prepare the intended product/feature set for delivery to users. The sooner you get feedback from users, the sooner you can fix errors. You will spend less time on bad ideas, and increase resources available to devote to the successful ideas.

In order for updates to reach production faster, each iteration must include one feature. That is why you need to shorten the life of branches.

Problems with long-living branches

• Conflicts between commitments (“merge hell”). Postponing the release, integrating with an external system, and other external factors may cause the code to become inoperable.
• Difficulties with code sharing. Other team members may need code from the new branch if the features depend on each other. They have to make another branch to access the code.
• Problems with the test environment. If there is only one test server, but feature branches are numerous, you will not be able to test tasks in parallel.
• It's hard to roll back the changes. Problems after release are common, and if a new feature branch causes crashes, developers have to choose between a hotfix and a rollback, go into the source code, and re-upload the solution.

How does the Trunk Based Development approach solve these problems?

Trunk Based Development is a method of developing a code based on a single main branch. Unlike the Gitflow approach, TBD allows developers to add new modules directly to master. Secondary feature branches can also be created, but they have a short lifetime.

Trunk Based Development assumes only one branch for development, which is called a trunk. At any time, this branch can be deployed to the production. The development still goes into the separate feature branches, but now these branches live no more than two days.

All changes to the trunk are added via pool requests. The changes are small, so the review process does not take long. Getting a new code into the trunk starts the processes of automatic building, testing, and deployment to the necessary environments.

But how does one develop in a single branch if some features are not ready yet, and the release is tomorrow? Here Feature Flags come to our aid.

How Feature Flags work:

In essence, this is an IF-block that runs a piece of code when a certain condition is met. The simplest thing is when the developers themselves decide whether to enable or disable the code. There can be more complex parameters, for example, on schedule or only for users with a certain level of access. Vice versa, the feature is disabled if the system load exceeds the specified threshold.

Image

At the switching point, we refer to the toggle router which determines the state of the feature. To understand which flag is needed, the router refers to its configuration and context. The former defines overall strategy for the inclusion of the flag (basic work conditions), while the latter includes any additional information (e.g., the name of the user who sent the request, current time etc.).

What Feature Flags can provide

•  
• Time-to-Market reduction: the team is constantly releasing new code, large tasks are divided into small ones, and as a result, development proceeds at a consistent, quick pace.
• Using operational feedback for product development: the earlier the release reaches users, the sooner the team has real data about the performance of their product.
• Continuous delivery of features with consistent quality: the ability to disable non-working code reduces the risks in the released version.
• Testing new features in combat conditions: feature flags allow you to gradually implement services, controlling the risks when releasing to the real audience.
• The ability to develop multiple software versions simultaneously: TBD and feature flags allow you to offer different features to different groups of users, while the same team can support all these software versions.

How to use Feature Flags

Technically, feature flags work the same way, and they can be divided into the following categories by the method of application:

• Release toggles: they hide features that are not ready yet, reduce the number of branches, and detach the feature launch from the deployment date. These are the main types of flags.
• Experimental toggles: these are used for A/B testing. They allow you to adapt features to target different user groups. Thus you can deploy the new service to X% of the audience to evaluate the load or collect the first reviews.
• Permission toggles: these open access to paid features or closed administrator functions. Such flags can live for a very long time, be very dynamic, and change their state with each request.
• Ops toggles: these disable resource-intensive functions. For example, this way you can adjust the app's performance on weaker smartphones or insure against a drop in performance when launching new functionality. The flag will disable the module before it causes a critical failure.

How to manage flags

• Proprietary solutions: LaunchDarkly, Bullet-Train, and Unleash. Each product offers its own advantages, and each one is convenient in its own way. However, you will have to pay for licenses, and the flexibility of the configuration depends on the system developer.
• Open source solutions: Moggles, Esquilo. These are free solutions, but for them to work for you, you will need to work on them. In addition, you will have to find a product with a set of features that suits you.
• Own management system: this is the option that we use. This is a one-of-a-kind solution that completely satisfies us.

How Flags work for us

• Feature Flags Portal: Web + REST API application for manipulating the state of flags. Works directly with the flag storage.
• Feature Flags Storage: the persistent store with the settings for flags and their statuses.  
• Kubernetes ConfigMap: a K8s ConfigMap resource that is built based on data that is stored in FF-Storage in a convenient format for the end application. Changing these flags via FF-Portal also results in changing FF-configmap.
• Microservice: uses FF-configmap as the configuration source when the application starts. When changing FF-configmap, the microservice reloads its configuration.

The application reads the flag configuration from FF-ConfigMap, which is mounted to the Pod as a file. If you change ConfigMap, K8s will update the file, and the application will react to the file change and overload the flag configuration.

Flags are changed through the portal, sending an integration message to the bus-line when the status is updated. The Config Updater component updates flag values in FF-ConfigMap via the K8s API.  

What about testing

The question arises, how does one test a product with feature flags? At first glance, flags complicate this process. If there are a lot of switches, then the number of possible states increase dramatically.

The flags do not always depend on each other. Because of this, we are testing two limiting cases for the release: 1) all new flags are disabled and 2) all flags are enabled.

Practice shows this to be sufficient.