There is a lot of talk circulating about the exit of the Air Force and DOD’s first “Chief Software Officer”. Having first-hand experience with the plans, processes, and outcomes allows me to write from a detached observer perspective. What can be learned by those embarking on improving a massive, Brownfield organization?
What happened anyway?
For anybody reading this without a whole bunch of existing background knowledge, here’s a far-too-brief summary.
- A gentleman was somehow anointed
himself“Chief Software Officer” (see note at the bottom) of the US Air Force - He brought in private sector success (survivorship bias) from the
Europeantrans-atlantic marketplace - Worked on several fronts to try to improve software development within the military
- Exited the role recently and expressed himself in an article
I’m not going to go through the article since it’s his view of things and we need to look at the whole picture. Asking any single human about their experiences with the CSO or any initiative he touched may yield quite an array of responses. The remainder of this article is an attempt to focus on efforts and outcomes.
The impasse
The boiled-down aspiration of all of the efforts that came out of the CSO was to bring creativity into the military. It may not have seemed like this was the goal, and may not have been among their intentions, but here it is.
| Current State | Desired State |
|---|---|
| Software takes years to make and is low quality | Software development is easier and has better outcomes |
| Security holes prevent updates which are made to plug security holes | Shift security left so software is already secure |
| The people in charge of large enterprise software systems aren’t software people | They won’t have to be software people since the tools keep things secure |
On the left, we have an environment that experimentation is impossible. Nobody can build a prototype since it doesn’t meet the requirements and any software that is modified needs to traverse the waterfall to get deployed. The waterfall is fundamentally flawed and slow.
On the right, we have the DevOps, Velocity, Agile way of working. It’s just tools, right? Give them the tools and they’ll be able to use them.
Who is they though?
They is not military-employed software developers. Not airmen or soldiers or even military-affiliated civilians. If the military had internal software expertise, changing how they work is (by comparison) trivial.
Nearly all software the government needs to write comes from contractors because writing software isn’t among the things government people do. In the middle of thousands of government software procurement contracts, suddenly all of the CMMI waterfall nonsense that had to be in the contract to win it is being modified by a set of delivery patterns and tools.
Of course, the procurement folks don’t have any problems pushing back on the CSO and saying “next time, we can make the requirements have this stuff, but not this time”.
Who else could they be?
The other source of software used by the military is products that were built and released and purchased. Not the development effort or time but just the software, as it is. When the government wants to use some software, it goes through an ATO process. This includes product software, so at times they government may request changes to the software to meet some requirement to grant the Authority to Operate and thus pay for the licenses or support contracts.
This group is easier to push around without needing powers granted by Congress. The interface to the public sector software world is a select few budget-holders on the federal side and a select few public sector salespeople who are always hungry.
The public sector salespeople are accustomed to having a government representative say “it would be swell if your software met this ATO requirement by next release so we could buy $4,000,000 worth of licenses”. The software product would adapt to the needs. The procurement would go smoothly. Everybody wins eventually. It’s actually really painful to delay new features to add a mechanism that disables accounts after 35 days.
The CSO found himself in the position where he could make these sorts of demands but never controlled much of a budget. Other people had the budgeting done years in advance and there isn’t any extra money sitting around in the military or any other part of the government.
The demand: New distribution channel
As mentioned before, the biggest hurdle to receiving money from the government is achieving and maintaining the Authority to Operate that software. The big demand was to use a specific software distribution channel where artifacts and Dockerfiles were sent through a blessed pipeline and out pops authorized software.
I’ve seen this pattern emerge every time there are software developers following one master and some new master trying to make devops and even talked about it at an event. The same outcome happened here which, I’m learning, isn’t actually as bad as I thought.
| Intended Outcome | Actual Outcome |
|---|---|
| All software follows the approved patterns so it’s super secure and has fewer flaws and gets deployed faster | Software gets deployed faster as developers figure out what reduces the feedback |
The time between development and software usage does go down once (if) projects hammer themselves into the shape required by the pipeline. The incidental reduction in granularity of the SBOM is a rule beating system defect but it’s present in the regular ATO process as well.
The outcome: Different compliance
Before the CSO’s blessed (or cursed depending on your perspective) pipeline, the government’s ATO process for software could be slow and always lead to waste and delay. The value of that waste and delay was an individual reviewed the stuff and said “okay, you can run this.” That individual took on the responsibility for that software and most of them take it seriously. As stated previously, there isn’t a lot of software development expertise in the military so these people rely on mostly self-reported things or poorly configured scan results (or well-tuned scan results that some other contractor runs).
Now there’s a pipeline job that does the poorly configured scan and says the software is okay. Since the pipeline is all tied back to a single individual (see note at the bottom), the typical protective nature of the bureaucracy doesn’t kick in when a package goes sideways. The pipeline isn’t corporeal enough to blame and fire or rally around and point fingers elsewhere.
I doubt anybody will pick up the target for this pipeline so don’t have a lot of faith in the new distribution channel staying in place for long.
Culture change is hard
It’s tough to know whether this was seen as a culture change activity or simply an ego-driven “see how much better it can be” preview of private sector development.
Many of the complaints in his exit article, I read with utmost sympathy. I’ve been saying similar things to the departments of state, commerce, labor, energy, and defense for years. I’ve been an outsider pointing in and saying “that’s crazy” to people who are doing the work. They largely agree but “it’s this way for [security, fairness, tradition, etc]”. It’s interesting to hear that the same thing happens at the top. This leads to the conclusion that it’s systemic dysfunction rather than an error or something malicious.
Who fixes a systemic dysfunction?
When Charity or other smart people talk about the Socio-Technical system that makes up an organization, imagine that at the millions-of-people and millions-of-computers scale.
The only recent example of high-performance software development for the Air Force required chopping off 99% of the Air Force from that effort. It was run out of a loft in Boston, given a Star Wars name, and allowed to flourish. And even that, once they succeeded in one outcome, the smallest bit of legacy organization influece ruined future attempts.
The people doing the work have a culture of checklists. If they did all the steps, that was their job. If they missed a step, they get yelled at. Software development doesn’t have steps. It’s artistic and creative. It requires design expertise the bring the right functionality to the users in the right way. User-centered design is a major component of the Power to the Public stories on Public Interest Technology. The same rules and needs apply for warfighters and support civilians.
Leverage points
Normally the people who control the leverage points are the ones who need to take action or be persuaded to act (or retire) for the organization to change its culture. Rewarding frantic efforts or profitable outcomes creates feedback loops that lead to more frantic efforts of profit instead of healthy employees and healthy growth.
- Arms race Pipeline scanners need to be better tuned for each product: technical expertise in the government
- Drift to low performance Being wasteful with time and resources doesn’t have any consequences
- Tragedy of the Commons The resources are all “the government” so if designs are terrible, just write an operating guide or checklist
These are all ancillary flaws that need to be addressed but aren’t fundamental. The reasons that these things are the way they are is the budgeting and procurement processes.
Cash rules everything around everyone
The only leverage point that can possibly have an impact is wielding the budget with deliberate and drastically different priorities. There’s a lot of nuance to these sorts of things and budgeting always becomes the target for corporate capture and politicking. The steps I would take if I were in Congress granting these budgets:
- No software development projects are allowed to be outsourced unless they have a clear decommissioning date within the next 3 years
- Expand the federal civilian force with developers and designers, or tap into the enlisted and officer ranks
- Enlisted and Officers need to know the domains they’re working in and overseeing
- All software created for and by the government is open source by default
- Tie contractor payments to positive outcomes, not crystal ball requirements
Digging into that last one a bit, if contractors had to deliver the positive outcome required to get their money, they wouldn’t bid unless they had a solid plan. This would be a nice way to lead procurement teams toward the notion that some things can’t or shouldn’t be outsourced. Any business systems that are expected to work for more than a year need to be built and maintened by employees so that the software can keep being built forever. This isn’t a failing of Agile work either. It’s simply the admission that any software worth writing needs to be worth staffing until decommissioning.
With these 5 changes in place, a slew of new systemic failures will emerge. The big contractors will adapt to them and keep winning work that they can barely deliver. If the outcome-based-payments are embraced, they should stop bidding at some point.
It’s also incredibly difficult to fight the over-classification issues for “sensitive work”. Much like ATO, a person needs to put their livelihood on the line to get things declassified and open sourced. Poor software development practices may lead to passwords and salts and sensitive information getting out. And who wants to read 5 million lines of code just to be fired for some hard-coded admin backdoor being discovered?
Closing thoughts
It does not seem that the military will be capable of adapting to the needs of the 21st century without some egomaniac running roughshod through the organization rebuilding many org units and shutting down many others. This maniac needs congressional support and to be given budgetary control for the units they intend to revolutionize. Without controlling the money, it’s not going to work and any progress made will be quickly erased.
That said, the military is probably one of the toughest places to enact such changes. The intelligence folks have come a long way but the previously stated list of state, commerce, labor, and energy are having a lot of trouble. Even energy, the supercomputer people.
Godspeed public servants.
Response from Mr C himself
Hmm. This is an interesting take.
You have some of your statements not exactly correct though. I didn’t “anoint” myself CSO. My boss did.
The pipeline you mention, Iron Bank, was never tying back to a single individual. And it will definitely survive.
Your European comment is interesting. I had several businesses in the US prior to becoming CSO.
Overall, lots of great points.
I made ome adjustments to the article based on this response. I’m still not sure what a breakdown in the pipeline is going to lead to. Can’t drag a yaml file in front of congress to answer the same question over and over. I’ll be interesting when it’s tested.