Photo by Osman Rana (opens in a new tab) on Unsplash (opens in a new tab)
Non-member link (opens in a new tab).
If you’ve ever worked in a large organization such as an institution of higher learning, you know that some systems are like ancient city plumbing. Everyone uses them, but understanding how they truly work requires becoming an archaeologist literally, piecing together clues from scattered, outdated documentation. The thought of touching them sends a shiver down the spine of even the most seasoned engineers.
For us, that system was the student photo upload utility. A seemingly simple multi-user application built 15 years before the decision to decommission with a massive footprint, deeply entangled with everything from ID cards to official student records. Recently, I had the thrill (and terror) of leading the project to decommission it and deploy a replacement as a feature of a modern app. This is the story of that deployment, a manual, meticulous, and ultimately successful mission.
The Beast We Were Up Against
The old system wasn’t just a simple web app written in .net with an SQL server database running on an IIS server. It was a system widely depended on and used at least once by any and every single student that passed through the institution. Its integration with core services were numerous and every connection was a potential point of failure. It interfaced with two main services
-
ID Card Printing. New students uploaded their photos to generate their identity credentials for physical premises access and identification.
-
The Student Records System. The uploaded photo wasn’t just stored in a simple folder; it was written directly as a BLOB into a database table.
The old process was fragile with a user interface that didn’t meet WCAG 2.2 AA accessibility standard and an experience that generated a high volume of support tickets for trivial issues like users having no visibility into their submission status, a problem a simple notification would have solved. It provided no backup and offered zero analytics. The server it ran on had also reached end of life, which posed a significant security vulnerability and a potential breach of the country’s General Data Protection Regulation (GDPR) regulations. We needed something better.
The Strategy: Respect the Touchpoints
We knew our new system wouldn’t be a simple “lift and shift.” Our core philosophy was: “The database is the contract. We will not break it. The directory where the photos from the current system are uploaded to is our final destination, we will not change it.”
Our new system would be built as a feature of a modern PHP app. It would have a slick UI, face detection integration to suggest the likelyhood of suitability to the students before their photo is uploaded for human review, proper file validation, and real-time notification. But its most important job was to mimic the old system’s final output: writing that image to the exact same table and file storage, in the exact same format (jpg) and without exceeding the average file size and dimensions of the old system’s output.
This decision was our key to a low-risk deployment. We weren’t asking a dozen other teams to change their code. We were changing the how, not the what. The ID card printers and legacy apps would never know the difference.
The Project Team
The project team was a lean, cross-functional squad and comprised of:
-
Two Full-Stack Developers — myself and a colleague
-
One Database Developer
-
One Database Administrator (DBA)
-
One Product Owner
-
Three Testers
Go-Live: A Symphony of Focused Silence
Before any production code could be touched, our process was governed by a formal Request for Change (RFC). This dry, procedural document became the single source of truth for our entire operation. It meticulously documented every pre-go-live task, the precise sequence of go-live steps, and the post-launch validation checks. Most critically, it detailed our backout plan. This RFC was our ticket to the production environment.
I highly recommend working from the office if you can on days like this as it helps with effective communication.
The hour of deployment, however, felt less like a tech rollout and more like a heist movie executing a meticulously planned blueprint. Given the components involved, the deployment was scheduled to span over 5 hours. The team had decided to work from the office that day and was on a Teams call with loads of snacks present on the desks (we knew we may not get to lunch depending on how launch goes — pun intended). The approved RFC open on one half of the screen, a shared terminal on the other and our checklist on a second screen.
We followed the steps in the checklist religiously which included testing the various touchpoints. The test on the final touchpoint failed.
What?! This was meant to be our final validation before opening the floodgates. Instead, we were staring at a full stop. The tension on the call snapped from focused silence to controlled panic.
We immediately notified the product owner, requested to extend the RFC window by another 3 hours and went into troubleshooting mode. The photo was in the destination directory. It was in the database. The new system had done its job perfectly. The problem had to be downstream.
After a frantic one and half hour of checking logs and permissions on the file directory, one of the developers pinpointed the issue. The permission of the files from the new system were different from that of the former.
We had found our culprit. Our diligent DBA, following every best practice for security, had rightly updated the code to set more restrictive permissions. The old system, in its insecure way, granted universal read-write-execute access to every file. The legacy printing software, built in a different era with different assumptions, did not have the appropriate user-level permissions to read files with the more secure setting.
We had done the right thing, and it had broken everything.
A quick, tense discussion ensued and we reached a pragmatic solution: we would temporarily revert the permission level to to be less restrictive for the go-live to ensure compatibility, with a Jira (an agile project management and issue tracking tool) ticket created on the spot to task the DBAs with properly configuring the user permissions on the printing service.
A few moments later, the command was run, the permissions updated. We held our breath and retried the failed test. It passed! The relief was palpable. We had successfully navigated the final, unexpected hurdle. The app was opened to the public, and the deployment was, finally, a success.
This last-minute scramble was the perfect reminder that even with a flawless plan for the systems you control, the legacy ecosystem you integrate with will always have the final say. It taught us that “working” isn’t just about your code functioning in isolation; it’s about your code functioning within the complex, often imperfect, reality of the entire IT landscape.
Lessons Learned from the Trenches
-
Understand the “Why” Before the “How”: The most valuable work happened weeks before deployment, mapping every single touchpoint. The technical deployment was just the final act.
-
The Database is Often the API: In legacy environments, the database schema is the contract. Honoring that contract is the fastest path to a successful, non-breaking change.
-
Communication is a Feature: This would have failed if we had just built it in a vacuum. Treating other teams as stakeholders was crucial.
Decommissioning a beloved (or at least, heavily used) legacy system is a rite of passage. It was a powerful reminder that sometimes, the biggest innovations aren’t about the shiniest new technology, but about the careful, considerate execution of a plan that respects the complex ecosystem you operate in.
Now, who’s ready to talk about containerizing it all?
What was your most nerve-wracking legacy deployment? Share your stories in the comments below and don’t forget to leave some claps😊*.*
Thanks for reading! Read more about me here (opens in a new tab).
Support writers. All our non-profit’s offerings here (opens in a new tab).
Click here to subscribe to the IT Chronicles newsletter. (opens in a new tab)
Click for Wordsmith, Mystery Writing, Write Like Stephen King, more (opens in a new tab)
Click on the image for more IT Chronicles — Art by Susan Brearley and Night Cafe