Introducing Lab Assistant

When multi-project workflows are necessary, the hand-offs can lead to repetition. To alleviate that, Lab Assistant will automate the hand-off for a multi-project workflows.

Multi-project workflows

If your organization’s policies are compatible with the constraints and gates built into GitLab, the workflow is amazingly smooth and effective. The baked-in controls are deliberately set to keep decisions focused on a couple of events:

1. Merge when code is ready
2. Tag when deployment is ready

From these two deliberate actions, code + pipeline statuses + job results allows most other behaviors to be automated. The assumption is that the individual authorized to approve code merges is the same individual who can tag releases.

This falls apart where there are multiple stakeholders who have serial, manual processes where no individual is responsible for the outcome, so several individuals need to perform a single aspect. In this situation, pipelines can only go so far and tags and branches become exponentially more complex than is reasonable. If branches, tags, and environments are protected against the developers, maintainer access is also withheld which greatly reduces autonomy and velocity.

Creative project -> deployment project

The deployment project acts as a locked-down parallel to the “wild west” style creative project. In adding these additional gates, a whole merge request workflow can be engaged for review and approval. Even if the only change is ticking a version number or container tag.

The most common example of this is a product or application that shares a set of environments with other applications. These are typically tied to a big relational databases where business logic is mixed with schemas and data. These systems are typically very tightly coupled to other internal and external components so any changes to these environments require a good amount of coordination.

The merge request workflow is a great way to consolidate this change control functionality into one place. Creating long-running branches that match these environments allows tracking the changes to the environment over time. Using a tagging pattern for promotions from one to another may be better depending on the pace of changes and amount of players.

The biggest gap in how GitLab expects code to flow is that a merge request from one project to another acts as a way to move that source code from a fork back to the original project. In these scenarios, the source code isn’t usually relevant to the change since some compiled JARs or a set of SQL queries are the unit of change. It’s not about lines of code changed, it’s about an approved package of artifacts.

From there on out, it’s a very familiar pipeline design with automated and dynamic jobs to confirm and test aspects and then ultimately deploy the new code to the next environment.

How does Lab Assistant alleviate this?

At the hand-off from the code project to the deployment project, there’s a bit of gray area that can be handled many different ways. The most obvious is to have a cross-project pipeline that kicks off and sends an artifact across that is built and deployed. No merge request but the pipelines and statuses all flow nicely. This typically doesn’t feature enough coordination to reassure the team responsible for the long-lived test environments.

The next increment-toward-manual would be to have the change management process create a merge request that references the packages that are changed. Since that information would theoretically be entirely contained in an application repository being tagged with a new release, the activity seems ripe for automation. Basically, a user would go to a deploy repository, create a branch, find the old tag, replace it with the new tag, commit that update, create a merge request, and then start the approval flow. Some steps are automated in the manual approach, like a pipeline running validations and alerts to the deployment project owners about the new merge request.

In a containerized or package-managed environment, it’s about updating a docker image tag or a Gemfile, pom.xml or some other version locker. In the more legacy style applications, this may be a commit hash for a bunch of SQL scripts that were “approved”. This leads to duplication where the signals are:

1. Code is updated in a creative MR
2. Code change is approved by product team and merged
3. Code changes are merged together to form a release via git tag
4. Git tag pipeline builds and tags artifacts
5. Someone decides to go to change management project to create an MR that points to the git tag
6. Pipelines run and provide feedback and review
7. Approvals for deploying to environments are tracked
8. Deployment is done and evaluated (and rolled-back if needed)

So these could all be distinct activities, but the product team approving the merge and tagging the commit and creating the MR to update it in the environment all seem like the same behavior to me.

If there are enough stakeholders that each one would want to reserve each threshold, this may not be important or particularly helpful.

On the other hand, if there is a small group that would handle all of those things, the duplication will be annoying.

Using lab assistant

You create a job that looks something like this. The pre-requisites are that you need to know the project identifier number for the destination project and a user identifier number for someone to assign the new merge request to.

create deploy mr:
  stage: release
  only: 
    - tags
  image: registry.gitlab.com/brownfield-dev/public/lab-assistant:latest
  script:
    - lab-assistant --target-project-id=19762418 --target-user-id=3548980

The target-project-id and target-user-id can be set in the arguments or added as environment variables to the project.

Environment variables are the same letters but with all caps and underscores instead of hyphens.

TARGET_PROJECT_ID=19762418
TARGET_USER_ID=3548980

The rest of the information it gets comes from GitLab CI built-in environment variables.

What will be added?

The point of this is to stay relatively simple. So far it just creates the merge request in the other repo without applying much decision support logic. The ability to add this logic was the point of creating the project, so I’ll be adding some of that.

Actually commit something

It may be nice to be able to identify what to change and use the “auto remediate” workflow to identify the reference and update the version.

More advanced rules

Right now the pipeline rules tell the job whether or not to run. The job will always create a branch and merge request if it runs. The good news is that the branch creation API will fail if the branch already exists. The bad news is that if the commit hash changes, the branch will get a new name. There is a good case for adding more complex rules to make sure the MR isn’t in WIP status or that it only works on master or that the commit was done by certain individuals.

Errors and edge case handling

Right now it either works or it doesn’t. There’s no case where the job would run to completion and it wouldn’t perform its actions but the status would come back green. If an MR already exists, for example, the job will fail. This would be better to handle gracefully since the activity was completed, just already happened. Having it leave a comment on the MR or add a commit to it with the new tag commit hash would also be helpful since it would invalidate any in-flight approvals for the destination project.

Feel free to make suggestions

Make issues or merge requests against the repo if you are interested in adding a capabiltiy to the Lab Assistant. It’s basic and tiny now so any increases in scope will probably require some re-architecting.

Who will support it?

Great question. This is completely unsupported and I’d expect people to pull in this code and customize it to their liking. If it becomes a highly complex tool that mediates all sorts of interactions, it will certainly fall into disrepair and need to be rewritten.