After having a major system uplift, there are four words I’ve heard over and over again which are stalking my dreams and turning them into nightmares lately. “We didn’t test that.” Well, that doesn’t seem too dire really, but the repercussions of this are huge and cross every team that worked on the uplift in the first place.
Who is impacted?
Developers - In some cases, these are the unsung heroes of your uplift and what is their reward after it is completed? To keep their noses to the grindstone and work a few more weeks off on the fires that keep cropping up. Doesn’t sound like a real promising job huh?
Projects - Not just the project people, which I'll get back to, but you're now putting off being able to work on your next projects. It doesn't matter if it is old production fixes or currently pipelined new projects, delays will drain your political capital because you are now missing target deployment dates and the business thinks you are pretty much worthless for not catching all the things you are now responsible for fixing. Project personnel start feeling burned out just like your developers because they've been on the project forever, and the bigger the project the longer you'll want to keep them around to play clean up for all the new debris coming up.
Support - How many fires do you get on a daily basis? Multiply that by a factor of 2 or 3 after doing an uplift like this for about a month, and you’ll see what kind of volume you’ll be slowly crushed under when you have people in the new system doing their daily work which they never really replicated in the test environment.
DBAs - During the project and throughout the implementation we've been wearing them thin, and they are now getting no rest by making them look at applying patches and identifying system issues which should’ve been caught months ago so they aren't able to instead be able to focus on things like building new monitoring tools. Fundamental things like missing critical DB bugs can easily happen because everything in the system is being thrown against the wall without a whole lot of investigation possible for each item.
How can we do better?
We need to be more proactive about showing the business and project teams what has been run in the last 30, 60, 180 days in your system and by whom. Having these snapshots would be able to show the majority of the daily, monthly and quarterly reporting goals that the project needs to meet.
We can do a better job about documenting out our system flow, so we know what all our parts and pieces are which are required to provide full service to the business. This way you won't get surprised by a report pushing output to a third party in-house group, to a server you didn't even know was connected to your system.
I'll probably add more as I encounter them, but what would you add to this list?
No comments:
Post a Comment