Robert Nye, an information technology specialist with Mississippi State University, expected his Feb. 24 trip to the East Lee Boulevard DPS office to be relatively quick. Stopping by with an appointment on his way to campus, he figured he’d be in and out without disrupting his schedule.
“I hadn’t been before since I’m a new resident, but my fiancée went in the fall last year,” he said Friday. “She was like, ‘Yeah I was in and out in 15 minutes!’ and so I was expecting to be able to get work in no time. That was definitely not the case.”
Instead, he found a line of people out the door.
“While I was standing in line, at least two other people with appointments came up behind me,” he said. “They let the three of us inside to get tickets, but the room was already full with maybe a dozen people. They said the system had shut down and they’d call me, which they did, four hours later, to say that their system was still down. I never got another call back.”
DPS Commissioner Sean Tindell told The Dispatch Thursday that Nye’s experience was hardly uncommon last week. A software update from international tech firm Idemia crippled the department’s servers, shutting down operations statewide Feb. 20 and 21.
While workers managed to get the system back online over the weekend of Feb. 22, things were still dramatically slower than normal, with wait times of two hours or more grinding DPS operations to a halt.
Affected services included driver’s licenses, but also other forms of identification, firearms permits, commercial driver’s licenses and learners permits, even registration for sex offenders. If it used the department’s servers, it was effectively on hold.
Tindell said the department managed to get the technical side of things pretty close to normal over the course of the week, estimating that DPS was at 85% capacity heading into the weekend. By Tuesday Columbus’ DPS office was reporting no unusual delays, and Mississippi Highway Patrol Sgt. Derrick Beckom confirmed he wasn’t aware of any ongoing disruptions in Columbus or Starkville.
Still, Tindell warned that the effects of the outage might be felt for up to two weeks more. Even once the system itself was back up and running, every day of slow services generated dozens of backlogged customers that now need to be addressed.
“If you can wait and you don’t have to come in the next week or two, maybe put it off for two or three weeks and give us time to work through the backlog,” he said. “We’ll get you taken care of. If you can renew online, do that. Just be patient and try to make an appointment or do it online before you physically come into one of our stations.”
Tindell said DPS asked about simply rolling back the update or creating a backup beforehand, but were informed by Idemia that it just wasn’t possible for a systemic change this large.
“That did come up in some conversation before the software update was initiated,” he said. “For some reason, Idemia did not believe it was possible to have a rollback available for this kind of software update. There was a point where Windows 95 just wouldn’t work anymore. There was no rolling back to the old software once the button was hit. That’s what we were told.”
What went wrong?
Tindell said DPS is still working to identify what exactly went wrong. He was circumspect about whether incidents like this were avoidable going forward. He said engineers are focused on just getting the system operational again right now, with a postmortem debrief with Idemia coming but not yet scheduled.
“They’re a multinational company who’s been on contract with us for nearly 10 years now,” he said. “We’ve had some issues in the past but nothing quite like this. Everybody’s had to do updates on their personal devices, sometimes you have issues. This time we had an issue that just happened to be statewide in a very critical agency that does a lot of credentials and licenses for folks.”
He suggested that in the future DPS might preemptively shut down its offices for large updates that can’t be rolled back, telling people to get their appointments in before the deadline or wait a week or two for the system to be given the OK.
That kind of communication is exactly what Nye says he wished he’d seen when making his appointment. He made his reservation online on Feb. 21, after the outages had started and DPS had started making public statements that it was having trouble, but he never got any kind of alert that his appointment might face issues. He even got an automatic reminder on Sunday to show up.
He did eventually get what he needed, coming back Tuesday and waiting around 45 minutes. By then staff had begun telling people they couldn’t take appointments.
“The local staff were very good at communicating to the people that were there,” he said. “It was just frustrating that I didn’t see any public announcements from DPS.”
You can help your community
Quality, in-depth journalism is essential to a healthy community. The Dispatch brings you the most complete reporting and insightful commentary in the Golden Triangle, but we need your help to continue our efforts. In the past week, our reporters have posted 35 articles to cdispatch.com. Please consider subscribing to our website for only $2.30 per week to help support local journalism and our community.
You can help your community
Quality, in-depth journalism is essential to a healthy community. The Dispatch brings you the most complete reporting and insightful commentary in the Golden Triangle, but we need your help to continue our efforts. In the past week, our reporters have posted 35 articles to cdispatch.com. Please consider subscribing to our website for only $2.30 per week to help support local journalism and our community.
