From overwhelmed to obsessed: one security professional’s EDR journey

Creating, maintaining, and growing an effective cybersecurity program for a rural hospital has been a tough task. When I joined the security team at Union Hospital of Cecil County (UHCC), we had already deployed Red Canary’s managed detection and response solution. The main security analyst who was previously handling Red Canary was leaving, so I was given a crash course in what to do to remediate alerts and what not to do. (For example: DO NOT ban explorer.exe.)

The training was brief and I was left with plenty of questions, but since I was handed a bunch of other systems, I was forced to move on, knowing I would need to come back. I knew what endpoint detection and response (EDR) was on paper—college gave me a textbook definition—but I didn’t really understand how it worked and how Red Canary could take this data and generate alerts.

The next time I sat down to look at my Red Canary portal, I was overwhelmed. I ended up contacting my Red Canary incident handler, Michael Haag, which was the best decision I could have made. He broke down the features and backend workings so it made sense. He also suggested I look at the Red Canary blog and check out Atomic Red Team, Red Canary’s open source testing framework. One Friday I started looking into his suggestions—which began an obsession for me.

I started spending big chunks of time looking at past detections from Red Canary and researching what different processes did. I also executed a few Atomic Red Team tests, which provided me with new and interesting alerts. My early days with Red Canary went from “overwhelmed” to “obsessed.” The blog and free content made a huge difference. It helped me quickly improve my knowledge, saved me time, and allowed me to get even more value out of the product.

Partnering to drive better outcomes

Red Canary has been an important partner in combating cybersecurity challenges common to a rural hospital. A few of the substantial changes we’ve seen in our security program include:

Time saved

Our staff now spends less time looking at logs, which has allowed Red Canary to become a huge force multiplier. Now that time can be used to make other improvements and continue working on the elusive documentation all small teams struggle to accomplish.

Increased confidence

Another huge change is the confidence we’ve gained during an alert or incident. It is very helpful to have an experienced outside group you can contact when you receive an alert. Coming up with a verdict on your own can be overwhelming, so having easily accessible, knowledgeable assets is important. It is also helpful to have a rundown of the chain of events, de-obfuscated code, and explanations when receiving alerts. This enables the responding individual to start valuable triage anywhere, anytime, which allows a small team to live their lives outside of work.

Peace of mind

Red Canary gives us peace of mind on the quiet days as well. Quiet days without alerts can sometimes cause paranoia for a security professional. Even though EDR does not have visibility into everything, it helps prevent the feeling I am missing something and wondering why I have not received an alert recently. If you don’t have a group like Red Canary giving you access to a large and wide-ranging pool of data, you can easily fear that the only reason you are not receiving alerts is because something is not configured properly.

We’ve prevented burnout by using the base playbook package, which allows call notifications to be delivered for different types of alerts. The call notifications for alerts are set up so Red Canary continues calling until it is acknowledged, which ensures that the responding party fully wakes up and looks at the alert. This is a noticeable difference from the typical email alerts that most applications provide.

Driving continuous improvements

At some point, our security team began looking at the Red Canary alerts we received not as successes, but as partial wins. Our thinking is this: while it is a win that the threat was detected, the fact that it was detected means that something malicious was executed. Our ultimate goal is to receive no alerts and have every touch base phone call simply be a confirmation that our data is being received. To reduce the number of alerts, we look at our detections and think: what existing tools in our environment can we use to prevent or disrupt that same type of alert?

We’ve done a lot of work to educate IT staff and management on the many pieces needed to run a security program. With all the detections and saves, it was believed that Red Canary was the only security solution our program needed. We needed to explain to the key parties within UHCC our theory on partial wins and that Red Canary was only a piece of the complete puzzle. There are other things we still need, such as encryption, two-factor, and firewall. We still require other products to prevent attacks.

Red Canary does not prevent attacks, but it’s important to note that it does help bring these issues to the surface. Once issues are brought to the surface, we can look at our detections and see why they have become detections and not notifications that events were stopped.

Advice for overcoming common challenges

While my story is about working on a hospital’s security team, many of the same challenges and successes can be applied to medium-sized businesses in other sectors. Here are a few of the challenges we overcame and some advice for other teams facing them.

Limited manpower

This is the most common problem teams of my size face. My advice is to change your perspective and look at limited manpower as an advantage because there are usually less approvals needed for a change. Less approvals needed means that small teams can make up for the lack of staff by evolving quicker than our larger counterparts can. We try to implement changes and restrictions that larger companies can only hope to do after a long approval process. While changing your mindset is helpful, it still does not change the fact that there are only so many work hours in a week. To counter the natural limits on time, we attempt to prioritize and bundle changes into groups to reduce the burden.

Limited budget

Most companies do not get the full utilization out of their products. Many small security teams do not have time to master a product, so they get the tool running and reporting, then set and forget. I try to revisit my products frequently and explore settings and features to make sure that we are getting the most out of our products. I make it a priority to read different blogs to gather as many ideas as possible that we may be missing. There are many group policy changes that can make a large impact on security.

Feeling stuck or defeated

It is easy to get hyper focused on the next change so that we forget how much we grew in a year. Make sure you look back sometimes and remember how far you have come. When I look back to the person I was when I felt overwhelmed by EDR, to the person I am now after a year of reading and education, it gives me confidence in my ability to continue growing.

Closing thoughts: cybersecurity in healthcare

The growth of cybersecurity in the last five years in the healthcare industry has been incredible. Cybersecurity has gone from an afterthought to an emphasis. For example, breaches used to be an article in the tech section of a news site; now they’re on the front page. This attention has helped drive home the sad necessity of quality cybersecurity programs for healthcare-oriented organizations. It has also become a way for non-technical people to be introduced to cybersecurity terms and trends. Because of this exposure, some healthcare workers are expecting changes and value said changes that come.

Healthcare still has plenty of room to improve in regards to cybersecurity. Medical devices are still being sold that are not compliant with HIPAA and are built on soon to be out-of-date operating systems. These very necessary medical devices become soft targets for attackers. There is also much progress needed in educating healthcare workers about the necessity of cybersecurity. Currently, many healthcare workers view cybersecurity as a compliance requirement that only impedes their abilities to keep up with an increasing demand for a fast patient experience. Hopefully, as cyber threats become more prevalent, some healthcare workers will be able to make the connection between their reliance on technology and the need to keep these technologies available.

The patient’s care should always come first in the healthcare industry. Is there any room in healthcare for cybersecurity? It is important to remind everyone in healthcare from administration to management to staff that by not taking the necessary steps to protect a healthcare organization from cyber threats, we are putting our patients at risk. The reliance on electronics in healthcare industry continues to grow as technology improves and staff with paper chart experience retire. It is important to provide confidential, accurate, and reliable access to electronic resources in order to provide the best care for our patients. Cybersecurity is critical for this goal.