I've written before about the importance of velocity, and I think in tech we sometimes ask product teams to be fast in a vacuum. we treat velocity as a matter of discipline or willpower, but the truth is that a team's speed is often the result of technical decisions made in the past, and reversing those takes time.
in this post I want to walk through what we lived through at atlas, going from almost no releases to multiple releases a week (4 to 5 a week for the past 6 weeks). I want to share my diagnosis of why fast releases weren't possible when I started working with this team, why they are now, and what we did to get here.
everything I'm about to describe was a learning curve for me. I'd never been through a process like this and I didn't have a plan when I started. I had a clear sense of where I wanted to go and I navigated by intuition along the way.
I want to write this down so I don't forget it, and because it might be useful to others. for that reason this post is going to be long and I'm not trying to make it entertaining. it's meant to be a catalog of everything we did and everything I now know you'd have to do to get here.
why our release process used to be slow
it took me a while to figure out why our product team had so much trouble shipping to production. I could see the symptoms of a team in trouble (low interaction, lots of bugs, reactive problem-solving, no roadmap, projects in progress with painfully low priority) but I didn't understand the causes.
here are some of the reasons I now see for why releases were slow:
- a codebase loaded with tech debt. our team had been through a lot of changes in product and business direction. that, combined with the wrong kind of velocity (shortcuts everywhere, 900-line functions), no tests at all, and no typescript, made the codebase genuinely unmaintainable.
- a non-scalable data model. months of direction changes and very little strategy left us with a data model as bad as the codebase. the data we were storing wasn't reliable, and there were basic cases for our business that the model couldn't represent (everything was a patch; lots of duplicated, inconsistent info).
- a deeply demotivated team. the two points above explain how the team was feeling at this point. any feature we wanted to ship was traumatic, 80% of the day was spent fixing bugs (and there was no clear process for that either), there was barely any communication (literally very few messages on slack), and a lot of disconnect from the rest of the company and the business teams.
- no strategy. this is by far my biggest takeaway from the whole thing and I'll come back to it. in short, there was no clear product strategy, short-term or medium-term. that wasn't the product team's fault. I now know that to build a strategy, there has to be a north star coming from the business side. the symptoms that we were missing strategy were that the rest of the team kept asking us for a roadmap that didn't exist, or that we kept asking ourselves whether we needed a head of product. what we actually needed was a clear objective and clear strategies for the product.
the first few months
the four points above were as frustrating for me as they were for the team. I wanted to think about new features but I could see there was a lot of behind-the-scenes work to do first. these are some of the things we did in the first few months to improve our release velocity:
1. the team needed a win. the first thing we did was a project from scratch: a new app for suppliers. the goal was clear: our product designer had spent months designing screens nobody was building. the frontend team was burnt out from patching an app that couldn't take any more. we needed a blank canvas. a new app, a new component library, a project we could be proud of. we also needed to give backend room for the big changes ahead. we started here and a couple of weeks later we had an app fully designed and 100% built on the front end. it was an early proof of how fast this team could move with better tools. this project never got a backend integration and never shipped, but we hit the goal: a better-designed, better-UX app built 100% by this team.
meanwhile, the backend team needed something to change so we could take on more ambitious work.
2. we needed to redo our data model. the problems were obvious. my hard signal was that the data we were storing was wrong and often contradictory (i.e. two queries that should have returned the same result returned different ones). there was also a softer one: when I looked at the db tables, my sense was that every dev had added columns for whatever they were building without thinking about the whole (the result was a lot of duplicated, inconsistent data). changing the data model is a huge project, and we got plenty of things wrong here. the first is that we couldn't trim scope. we had to design a better model, implement it, migrate the old data, and deploy it. all 4 were extremely hard, and the project took 6 months (we'd planned 1). the payoff was huge once it landed: the data now reflects how we think about the business. our data model genuinely supports the data it has to store, no hotfixes, no patches, no new columns every time we don't know where to put something. (I also have learnings from the decisions we made: we optimized for clarity at write time but didn't think enough about reads.)
3. we needed more and better talent. this is the most sensitive part of this post but I want to be transparent. our team had some fantastic people and some who weren't. for me, parachuted into this team, the challenge was figuring out who was on which side. I tend to think problems with people usually fall into one of two buckets: context (this person is brilliant but the conditions they're in won't let them shine) or individual (this person doesn't work well because of a lack of seniority or agency). this team had a lot of "context" problems, so to see how people actually worked, I gave almost everyone a from-scratch project where they could think and propose as if nothing else existed. some shone, others struggled.
the happy part of this process for me was learning how to hire. with the team in place, we got to work on the next milestone.
4. we had to rewrite the backend from scratch. this wasn't an obvious or easy decision. our backend was what I described earlier: a rat's nest. enormous functions (900 lines or more for very simple things), patches everywhere to handle specific cases, 0% test coverage, no typescript, no real way to test or qa anything. deciding to rewrite came down to asking ourselves which would take longer: a new team spending time to understand everything that was already there, or rewriting from scratch. we went with the second one by pure luck and it turned out to be the right call. today our backend has 100% test coverage, several months after going live, and the architecture is scalable (there are always issues and things to improve, but we're standing in a very different place). we did it gradually, and it ended up taking clearly less time than adding tests and rewriting the old code would have.
5. a clear process for reporting and fixing bugs. all of this had to happen while we kept operating day to day, which meant we were going to keep having bugs. I remember at the time our entire non-technical team was very frustrated with the product team. they were frustrated because things were broken and the team was barely responsive. we did two things here. the first was literally pulling every dev request out of slack, ranking them by priority, and properly fixing the top 5 most frequent ones. that mattered because if devs kept spending 90% of their time on bugs, they'd never get to bigger projects. the second was putting devs on a weekly on-call rotation. the goal was to have a clear owner when a request came in. that gave us some order, but this point is still painful today and I haven't yet figured out the ideal balance between feature work and bug work.
6. the team needed clear expectations and a career path. beyond the motivation that comes from working on projects you're proud of, the other thing that matters to me is that teams have a clear north star. it felt important for this team to know what was expected of them, both technically and on the business side: top-tier devs technically, and deeply involved in business decisions. setting the rules of the game mattered for two reasons: it gave us a framework for spotting who wasn't a fit, and it let us set clear rules for rewarding the ones who were (in our case: monthly performance reviews and a chance to move up the ladder every 6 months).
with all of this in motion and starting to bear fruit, I had the biggest realization of my year. we needed a PM (or PMs), even though I didn't know it yet.
why we started hiring PMs
the idea that the team needed a PM came from an unexpected place. once a few months had passed and the team was running much more smoothly, we suddenly had room to think about features. and we'd been going a long time without releases (because we'd been doing so much under-the-hood work to get things in shape). the need for a PM came out of needing to think about what we actually wanted to do with the product, i.e. what the product strategy was.
quick context on where I'm coming from: I coded for 10 years and never once had a good experience with a PM. on the teams I was on, the PM was someone who handed down a list of tasks with no context and pushed you to finish them by a date. I resisted having PMs at Atlas for years, and I think that was because I didn't understand the role.
the other thing I noticed is that when there are no PMs, someone else does that work anyway. in our case it was the devs: planning, estimating, etc. that's not bad, but it's not the best use of their time. and even if they can plan and estimate, they don't own or develop the strategy, so a piece is still missing.
I got convinced we needed a PM by listening to Lenny's podcast and by starting to ask myself a question: where do I want the product to go, and how do I get us there? as I dug in I learned that a good PM ties both of those together and takes care of the most important thing: strategy.
how we started thinking about strategy
thinking about strategy has been one of the more rewarding things this year has given me. I'll define product strategy the way I think it's most commonly understood: a strategy is a series of consecutive steps that, if executed, get you to a goal.
the best example for me is Tesla's:
Goal: Tesla's mission is to accelerate the world's transition to sustainable energy
Strategy: Build a high-priced sports car → Use that revenue to build an affordable car → Use that revenue to build an even more affordable car → While doing so, also provide zero-emission electric power
the real shift in how we thought about product at Atlas was when we stopped doing things just because, and started ordering our features and ideas behind a big goal.
out of this, we started writing documents (i.e. PRDs) with 4 sections: project mission, vision, strategy, and features. the important part for me is that the four are consecutive. vision flows from mission, strategy from vision, and features are the natural result of executing the strategy.
I could write a whole post about this, but the important takeaway is that we organized what to do on the product side behind a clear objective, and that gave us more order and more meaning.
how we got weekly releases back
after many months, the team started running more smoothly. everything I described above helped us recover release velocity, and although we have new challenges now and the calm is always fragile (because businesses grow and nothing stays still), today we have foundations to build on and we got our release pace back. the team ships 4 to 5 new (meaningful) features to production every week.
I don't think there's a silver bullet when it comes to working with people. these processes are messy, and what worked today probably can't be replicated tomorrow. what matters to me here is to share how we untangled a process that felt very chaotic, with the caveat that it took many months and we made plenty of mistakes along the way.
I hope it's useful if you're going through something similar, and remember there's always light at the end of the tunnel :)