An internal review of technical outages that caused significant delays at airports and international land borders this fall has exposed critical flaws with the Canada Border Services Agency’s IT services.
The review found neither the CBSA nor Shared Services Canada (SSC) is prioritizing solutions to dated technology that should be declared “a top government risk.”
It goes on to say the CBSA’s response to the outages was inadequate and inconsistent across the country, and that parts of the IT system are fragile.
The review, ordered by Public Safety Minister Gary Anandasangaree following the outages between Sept. 28 and Oct. 5, lists 12 action items to be completed by next fall.
“I think the plan clearly makes that statement that we have a problem and we are going to fix it,” said Stephen Laskowski, president of the Canadian Trucking Alliance.
He said CBSA IT outages have been a common occurrence over the last decade.
“To be quite frank with you, it’s only escalated in severity with this fall perhaps being one of the worst outages, where we had truck drivers 24 to 36 hours stuck at the border.”
CBC News has put in a request to speak with Anandasangaree about the review’s findings.
What caused the 2 outages
The outages happened after two separate, planned IT changes: a database upgrade and a firewall patch.
A person with SSC did not apply the necessary patch to CBSA databases ahead of a routine upgrade Sept. 28 that caused “significant corruption of live traveller and commercial data.”
That led to “cascading system failures and service outages” at inspection points inside international airports and land borders, the report says.
It created delays for air travellers because airlines needed to manually look people up on Transport Canada’s “no-fly list.”
The outages prevented importers from submitting manifests for shipments entering Canada electronically, leading to a week-long backlog at highways, marine ports, and air and rail yards.
It left some truck drivers waiting days to get into Canada.
“Significant technical intervention has resolved the majority of the data corruption,” but as of Oct. 25, “some recovery efforts continue,” says the report.
Patch led to communication breakdown
While teams were dealing with the database issue, an emergency security patch was installed Sept. 29 on a CBSA system that helps airlines determine who is on the “no-fly list.”
But the patch caused a communications break between CBSA and airlines that led to a full outage lasting seven hours on a Monday afternoon.
“The critical security patch was applied to the CBSA’s systems by SSC without proper notice being provided to the CBSA,” says the report, “leaving no possibility of co-ordination with airlines for readiness or to find a more quiet time to implement.”
The review found SSC was unaware that airlines would need to make changes on their end to keep those communications open.
Airlines entered outage protocols, but manually processing names led to delays.
“The change was thought by SSC to be non-disruptive,” says the review, “but clearly was not.”
Action plan has March, October deadlines
The review, posted online on Friday, outlines “critical lessons” designed to address the root causes of the outages, with deadlines of last November and next March or October.
CBSA and SSC will create a joint IT change management process alongside industry partners to “re-establish trust and to avoid unexpected service disruptions.”
Truckers looking to enter Canada from the Blue Water Bridge or Ambassador Bridge faced hours-long delays as a result of a now-resolved systems outage that saw staff unable to process electronic forms. Many in trucking are saying delays are growing more frequent and want to see systems improved.
The government will also seek a cultural change to “improve personal and collective accountability” after finding that the SSC is “not sufficiently aware of the real-world business impacts of CBSA system outages.”
CBSA and SSC will also “bolster the IT environment by identifying and correcting single points of failure” as a way of making the system more resilient.
Industry groups push for action
The Canadian International Freight Forwarders Association (CIFFA) has told Anandasangaree that the IT outages are a “deteriorating situation at our borders, which is severely impacting trade.”
The group, which represents supply chain companies across Canada, said there were 117 system outages between Oct. 6 and Nov. 27 when they wrote to the minister about the problems.
“We are growing increasingly concerned with the lack of communication from CBSA to vital Trade Chain Partners attempting to bring forward issues of concern so we can work together towards solutions,” wrote CIFFA executive director Bruce Rodgers.
Rodgers told CBC News that he’s met with the CBSA to discuss the review this month and that “time will tell” if the action plan will work.
“We’re aware of the challenges but at the same time, trade cannot continue to fail,” said Rodgers.
Laskowski said the Canadian Trucking Alliance has been pitching IT upgrades to the Liberal government as a priority nation-building project.
He called the soon-to-open Gordie Howe Bridge an example of a welcomed investment to busy international crossings, but noted a new crossing won’t help ease problems caused by old technology.
“It’s going out and buying a brand new car and finding out the manufacturer put a 10-year-old engine and transmission in your brand new car,” said Laskowksi.
“It looks great, but without the new transmission and engine, well, it’s not going to work the way it should.”
Mark Weber, president of the Customs and Immigration Union that represents CBSA officers, told CBC News that officers are using systems from the ’90s that are prone to outages.
“When the system is down, we’re not getting the information on potential security lookouts or flags,” said Weber.
The internal review said those security lookouts were being communicated to officers manually.
“While targeting data was not available during some of the outage,” says the report, “border services officers used their training and experience, as well as indicators obtained on the ground, to conduct risk assessments on both people and goods, consistent with contingency plans.”
