Tweet New Posts Automatically-ish

This is not related to brownfield technology, but it is something that can be done to help with handbook updates and other non-code sorts of projects. This recipe has some interesting ingredients such as GitLab CI, a Twitter API warpper for Go, Twitter Apps, GitLab Pages workflow, A Docker Container, and the route-map file.

GitLab CI

GitLab’s CI mechanism is running the pipelines that deploy this pages application. Since it’s already doing a lot of heavy lifting, it seemed easy enough to add a job to tweet about the recent changes. Since my workflow has one new post per merge request, I wanted it to identify the new post, tweet the commit message and then a public link to that post.

From an application requirements perspective, it’s pretty easy and obvious, but taking a diff of directories and re-working the URLs is a bit of a hassle. I noticed that GitLab has a CI variable called CI_MERGE_REQUEST_CHANGED_PAGE_PATHS which seems like it would already be populated with this information. After a brief foray into the docs, I found the enabling file called .gitlab/route-map.yml.

Route map

The route-map.yml file is just a list of arrays with regular expression patterns (docs here). This functionality is built around the assumption that one will be creating a review app for a real web application. If your change impacts the login screen, the merge request will have a link directly to the login screen. If it impacts an uploader or user profile or other page in the app, the links could bring you to those features. As a GitLab Pages repo, this one is quite a bit simpler.

The one for this project has 2 items in it, one for the “pages” directory and one for the “post” directory. Since hugo’s rendering engine doesn’t do much with them, the translation is clear.

# pages, but drop the extension
- source: /content\/page\/(.+?)\..*/
  public: 'page/\1/'

# posts, but drop the extension
- source: /content/post/(.+?)\..*/
  public: 'post/\1/'

The source regex takes the “content/” directory out of the path and removes the extension since the authoring side is all markdown. The pages host will map either .html or no-extension to these same files, so I’m simply dropping the extension.

Twitter API warpper for Go

NOTE: In the first draft, this was a shell script. I’ve reimplemented that as a go application for container size efficiency and functionality reasons.

Dalton Hubble’s Go implementation of the Twitter API is way overpowered for what I am doing in this pipeline job, but that’s the beauty of Go. A tried-and-true, widely used library that has a chance of being updated for twitter API changes and security issues is far more maintainable than a boutique shell script that will behave differently in different shells.

Of the go-twitter library’s many features, we only make use of “Auth” and “Status”.

A docker container

I put a tweet shell script directly in the blog repo at first, but realized that it was a broadly useful capability and didn’t require the rest of the code to operate. Rather than cloning the whole repo and performing actions, it made sense to move the tweet script to its own “deploy image” of sorts. Deploying a Tweet to Twitter actually enables some other interesting things that I’ll touch on in the security section later.

So the go app twitter was wrapped in an Alpine container and now lives in a publicly accessible repo at https://gitlab.com/brownfield-dev/public/meep-meep (v1.0 was debian-slim with a shell script, v2.0+ is golang) and the container can be used in image: registry.gitlab.com/brownfield-dev/public/meep-meep:v2-0.

Twitter apps

In order to get the various keys needed to operate the Twitter API, I had to go through registration and create a bot. My “bot” just tweets so there wasn’t a lot of paperwork. Nothing about hostile user practices or data mining. The goal is to have values attached to an account that can fill in these environment variables.

TWITTER_ACCESS_TOKEN
TWITTER_ACCESS_TOKEN_SECRET
TWITTER_CONSUMER_KEY
TWITTER_CONSUMER_SECRET

GitLab Pages workflow

The GitLab Flow approach is how we work with our handbook and other documentation that is built for GitLab so it was the most comfortable pattern to adopt for this blog. The pretty graphics and lengthy descriptions exist, but the gist is that any changes are started as a merge request, the change is supposed to be small, the change is supposed to be merged quickly.

Since I have a merge request object for each blog entry, I created the CI Pipeline job to generate the tweet contents there. Since I’m relying on the commit message for the body of the tweet, this means I’ll have many unacceptable tweet previews such as “fixed typo in title [link to blog post]” that shouldn’t go out.

My first implementation featured a job on Master that actually does the tweeting. This way the job has “protected branch” status and requires merge approvals before it can even be built. Even then, I made it manual because the merge commit is tougher to manage than the preview commit. Copying and pasting the preview tweet from the MR artifact into the production job worked okay, but wasn’t automatic enough.

The last iteration for now

Since that first tweet went out, I’ve refined the workflow a bit. I abandoned the protected branch approach to reducing tweeting ability and shifted to environments. This allowed me to move the tweet job back to the Merge Request where our helpful review job artifacts were available for referencing in meep -path and we know exactly what the commit message was so the merge commit won’t create trouble.

I have noticed the merge pipeline seems to have access to the merge request’s last pipeline run’s artifacts. It may make sense to move it back to master if the artifact passing works.

Environments and deploy jobs

The other GitLab feature that I use for this site is environments which doesn’t make a lot of sense for GitLab Pages sites due to them having a magic pages deploy job that shows up and does all the environment provisioning and content transfer. The way I started using environments was for local development. I created a “local” environment with a base URL of “http://127.0.0.1/” so that the route map would point me toward my hugo server job directly from the merge request. This is hostile for co-contributors but since I don’t have any yet, it’s a nice quality-of-life enhancement.

Environments as a security feature

The tweeting functionality requires my, the maintainer, sending my access tokens and keys and secrets to the runner. If someone were to contribute a post to my blog, and put in a goofy or offensive commit message, having the tweet job in the merge request would allow them to manually start it and tweet using the brownfield_dev account.

In the pipeline definition, the tweet job now sports an environment node that looks like:

  environment:
    name: twitter
    url: 'https://twitter.com/Brownfield_dev?x#'

The project configuration now has a matching configuration in the settings that prevents anyone except for me from triggering jobs that deploy to the twitter environment.

NOTE: The authentication environment variable names had TWITTER_ added to the beginning since that screenshot.

The combination of these two things prevents malicious tweets from going out via a contributor on the project. Anyone I add as a maintainer would naturally have access to change the environment protection, or retrieve the variables directly.

Final workflow

1. Create an issue with the article topic
2. Create a merge request to draft the article
3. Run the hugo new post command to stub the new article and draft away
4. Use commit messages that follow the “if this merge is accepted, it will” assumed prefix so stuff like “add meep workflow draft” and “resolve spelling and grammar problems”
5. Review everything the local: 127.0.0.1:1313 environment (links and flow), once it’s perfect, make the tweet commit
6. Check generated tweet artifact which is linked from the merge request as tweet preview
7. Merge the merge request into master branch to publish the new article
8. After new post is live, check tweet preview link to be sure it works
9. Run job on last MR pipeline to actually send the tweet

How to use the merge request

Several features in the merge request are enabled for a project where the above steps have been taken. This rather complex screengrab tries to summarize what all the new buttons and links do.

The commit message for the screen grab was just fixing a mistake I made locally and pushed with a prior commit so tweeting that is no good.

The pipeline definitions for reference

Like all good recipes, the life story part has to be scrolled past in order to get to the nuggets of usefulness.

I’m leaving the whole file in just to be clear that nothing else is happening in the project that is consuming meep-meep.

image: registry.gitlab.com/pages/hugo:latest

variables:
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - build
  - review
  - deploy
  - tweet

test:
  stage: build
  script:
    - hugo
  only:
    - branches
  except:
    - master

pages:
  stage: build
  script:
    - hugo
  artifacts:
    paths:
      - public
  only:
    - master

review:
  image: alpine
  stage: review
  script:
    - echo "${CI_COMMIT_MESSAGE}" > tweet.txt
    - echo "${BLOG_URL_STUB}${CI_MERGE_REQUEST_CHANGED_PAGE_PATHS}" >> tweet.txt
  only:
    - branches
  except:
    - master
  environment:
    name: local
    url: http://127.0.0.1:1313
  artifacts:
    expose_as: 'Tweet preview'
    paths: ['tweet.txt']
  variables:
    GIT_STRATEGY: none

tweet:
  stage: tweet
  image: registry.gitlab.com/brownfield-dev/public/meep-meep:v2-0
  script:
    - meep -path tweet.txt
  only:
    - branches
  except:
    - master
  when: manual
  environment:
    name: twitter
    url: 'https://twitter.com/Brownfield_dev?x=#'
  dependencies:
    - review
  variables:
    GIT_STRATEGY: none