Previously Published in The Messenger
The September 2022 cyber attack left the County, then governed by County Executive Steve Bellone (D-West Babylon), reeling with months of downed operations and services and a matter and timeline of response that earned criticism from other governmental officials, namely the special committee of the Suffolk County Legislature.
The special committee released their final, sixty-four page report on the cyber intrusion on Thursday.
Special Committee Chair Anthony Piccirillo (R-Holtsville) began the brief meeting by thanking the special committee, Presiding Officer Kevin McCaffrey (R-Lindenhurst) for his “leadership,” “creating” the special committee, and selecting Piccirillo as the chair. He also thanked Special Counsel Richard Donoghue, former acting deputy U.S. Attorney General, as well as the County employees of multiple departments who endured interviews and the stymied response time by the County.
“I want to thank all of our county employees, some who were put in very stressful situations during the cyber attack,” said Piccirillo, adding that “from the beginning” the investigation was not a “political witch hunt,” rather an effort to investigate the deficiencies in the County’s Information Technology (IT) department and recommend solutions to prevent another attack of such a magnitude.
“We’ve done a thorough job, we’ve followed the facts,” said Piccirillo.
“The report demonstrates that this was not a single point of failure scenario. Rather, it was a combination of fragmented and, in some cases, outdated systems, inadequate staffing, planning and training, and insufficient attention to the warning signs of the attack that was underway,” said Special Counsel Donoghue. “Those things led not just to the initial breach, but to the extended recovery period suffered by the County.”
The full report is now available online, here.
The committee is composed of Chairman Piccirillo, Presiding Officer McCaffrey, Minority Leader Jason Richberg (D-West Babylon), and Legislators Rob Trotta (R-Fort Salonga), Jim Mazzarella (R-Moriches), and Tom Donnelly (D-Deer Park).
The Executive Summary
On September 8, 2022, a cyber criminal using a ransomware strain known as BlackCat, also known as “ALPHV” or “Noberus,” launched an attack on Suffolk County’s IT system. Reports indicate that the hackers gained access to the County’s system months before the ransomware attack was launched. The hackers accessed, encrypted, and stole a “significant” amount of Suffolk County’s data, including “network maps, budgets, credentials, passwords, and other government information,” the report reads. Personal information of County residents, employees, and retirees, such as Social Security numbers and driver’s license information, was also compromised. Such data is now available on the Dark Web, a branch of the Internet only accessible with a specific type of browser and auxiliary software that is notorious for criminal usage due to the anonymity this corner of the web provides.
The criminal group initially demanded a $2.5 million ransom payment to restore the County’s access to its data, later reducing that ransom to $600,000, which County officials refused to pay.
Then-IT Commissioner Scott Mastellon testified in front of the special committee in October 2023 that the County spent more than $16 million on response and remediation efforts. Since then, the report says that estimates now appear to be $25 million in total costs.
The bipartisan special committee was formed on October 21, 2022, to investigate the attack. Since then, the committee has interviewed numerous employees and officials, hired forensic and cyber security professionals to “provide independent expert review and advice,” gathered documentary evidence, and held seven public hearings.
Key to the executive summary is the committee’s concession that no municipality can “completely shield” their IT sector from intrusion and exploitation, but that the 2022 hack was of such a large scale, significance, and duration, that the attack was “largely attributable to inadequate planning, preparation, coordination, and training by and of Suffolk County personnel,” tantamount to a “failure of leadership.
The Key Findings
The report outlines ten key findings of the report.
- Insufficient coordination between the different IT teams ultimately impacted the County’s cybersecurity readiness.
- The absence of a cyber-attack response and recovery plan “significantly hindered” the County’s response to the attack, increasing response time and costs.
- A “pass-through” within the County’s perimeter firewalls for data traffic put the entire County at risk. The destination of data from the pass-through was the Suffolk County Clerk’s office, then run by Clerk Judy Pascale (R-Moriches).
- Prior to the 2022 attack, the County’s overall cybersecurity was not sufficient to withstand such an attack.
- Firewalls that had reached end-of-life status were continually used.
- The lack of a Chief Information Security Officer (CISO) hindered the County’s preparation and response to the attack.
- The County’s insufficient staffing and training also contributed to the magnitude of the attack.
- County personnel failed to “sufficiently heed” warning signs of an impending cyber-attack leading up to the September breach.
- IT Commissioner Mastellon failed to report on the cybersecurity risks to the Legislature, as required by Suffolk County law.
- IT personnel were aware of Bitcoin mining activities in the Clerk’s office prior to August 2021. The report stipulates that it is unclear whether the mining activities contributed to the attack.
The “Pass-Through”
The report finds that IT personnel created a “pass-through” for Internet traffic destined for the County Clerk’s office could traverse the firewall without inspection. There remains “bitter disagreement” between the Clerk’s office and IT office about who was responsible for the pass-through. The special committee could not determine whether the Clerk’s office requested the pass-through, only that the IT department had the specific ability to create it. The firewall being circumvented was a Dell SonicWall that had reached end-of-life in 2019.
The report reads: “Knowing that the Clerk’s Office was protected only by an end-of-life departmental firewall, no IT professional should have sought, or agreed to, the creation of a ‘pass-through’ in the County’s perimeter firewall for traffic destined for the Clerk’s office firewall.” The report notes a February 2022 CyberDefenses Report that corroborates this finding.
A June 2023 testimony of then-head of the Clerk’s office’s IT team Peter Schlussler revealed that the aforementioned out-of-date SonicWall was the only device protecting the Clerk’s office from the World Wide Web.
The Breach
As early as 2019, the County was aware of a specific type of software used to collect and manage information about system activity. The so-called Log4j software is a popular and widely-used software due to its simplicity and no-cost download and usage. This software was eventually used in the 2022 breach.
IT personnel in Suffolk County, as early as December 2021, undertook “extensive” efforts to patch Log4j vulnerabilities across County domains. While the report finds that such efforts were “largely successful,” vulnerabilities to the software were not fully eradicated, despite the Clerk’s office being involved in those patching efforts.
Existing vulnerabilities were made known to the County IT personnel no later than February 2022, but since the perpetrators had already gained access to the Clerk’s domain as of December 19, 2022, previous remediation efforts likely would have had no effect in preventing the cyber-attack.
On June 21, 2022, an FBI Special Agent spoke directly to the County’s IT Security Coordinator Brian Bartholomew about evidence suggesting that malware might have been operating within Suffolk County’s domains, based on suspicious traffic within the New York State Court system. The FBI agent relayed that such activity could have been linked to the Suffolk County Clerk’s domain. Mr. Schlussler reported “nothing on our side” regarding the County’s cybersecurity vulnerabilities.
In August 2022, emails from the IT security team highlighted potential breaches and detection of suspicious activities. One email reveals perpetrators’ attempts at stealing “account names and passwords” and an attempt to “extract credential material from the Security Account Manager databases.”
Schlusser emailed the IT security teams regarding the alert: “Please offer some background. I have no insight to what this is about.”
On September 1, a file was found on the server in the Clerk’s office that none of the personnel had downloaded.
A September 7 email from Mr. Schlussler to Coordinator Bartholomew reads: “Brian, we need to deal with this asap [sic]. 3rd cortex today, with the last two being malicious.”
Cortex refers to the aforementioned security alerts.
Hours after the September 7 emails were exchanged, the perpetrators began encrypting data on Suffolk County servers across the County environment and posting ransom demands.
“Had the red flag warnings in late August and early September 2022 been recognized and acted upon, the September 8, 2022, ransomware attack may have been averted,” the report reads.
Recovery and Remediation
Then-Executive Bellon declared a County State of Emergency in the wake of the hacks that was extended from September 2022 to December 2023. The emergency declaration allowed the Executive to issue no-bid contracts without endorsement from the Legislature, among other powers.
Remediation entailed extensive replacement and upgrading of IT hardware and software across the entire County environment. The County’s main website was unavailable for five months after the attack. Vendors providing essential services, such as childcare, could not receive vouchers and payments. Emergency 911 operations were affected and County residents were unable to pay outstanding traffic tickets. County employees whose personal information was compromised received credit monitoring services at the County’s expense.
The report also outlines that County employees were “obligated to work long and unreasonable hours” in the wake of the attack, and that intra-County communication was relegated to personal phones, emails, and Cloud-based channels. IT and County leadership also deferred to outside vendors rather than County IT professionals who were “more familiar” with the County’s IT systems.
The report also finds that County data was “unnecessarily destroyed by outside vendors and that the remediation efforts were otherwise not well coordinated.”
Recommendations
The committee outlines seven steps it deems “necessary” to ensure future expectations will be met.
First, different IT teams across the County, as well as elected officials who supervise them, must work “closely and collaboratively” to prevent another attack. Cybersecurity issues must be elevated to appropriate levels and addressed sufficiently. Additionally, “political issues should never be permitted to interfere with the County’s cybersecurity defenses.”
Second, a County CISO should be appointed for a set term, to be determined by the Legislature. Furthermore, all Internet traffic destined for County domains must pass through approved firewalls and no department should use modems or other devices to circumvent County firewalls.
Third, the County CISO should “assemble a cross-department Cyber Incident Response and Recovery Team comprised of specific IT professionals identified by name from every IT team in the County.”
Fourth, the County CISO must “issue a comprehensive cyber intrusion response and recovery plan for the entire County,” specific to all departments and environments. The CSIO and IT Commissioner should coordinate with the Fire, Rescue, and Emergency Services (FRES) Commissioner to ensure the cyber response plan is “integrated” with the County’s Comprehensive Emergency Management Plan.
Fifth, the CISO and IT Department must provide an IT Risk Assessment Report to the County Executive, County Legislature, and other County leaders as the law requires.
Sixth, the County must increase the size of the IT Security team and “significantly” improve the skillsets of the team, stressing retention of County professionals, rather than deferring to outside vendors.
Seventh, and finally, the County should seek Cyber-Breach Insurance. County Executive Ed Romaine (R-Center Moriches) made this a core tenet of his 2023 campaign platform and told The Messenger that the County should be able to obtain insurance by the end of the year.