10 Things I have learnt doing Vulnerability Management part time
Part of my current role is Vulnerability Management. Before this role, I had zero experience in this area so I have had to learn on the…
Part of my current role is Vulnerability Management. Before this role, I had zero experience in this area so I have had to learn on the job. I was upfront in my interview about my lack of experience and knowledge gaps but my manager trusted me to learn as I go. I wanted to share some of the things I have learnt so far in the past year. None of this is gospel but just things that would have been helpful to me at the beginning.
- Know your estate
- Not every scan is the same
- You cannot resolve all Vulnerabilities
- Prioritisation vs Coverage
- Look for help
- Make remediation a process that fits into remediation teams’ workflow
- One solution can address many vulnerabilities.
- It’s ok to say No.
- Your VM tool could be wrong
- Zero Days — Keep Calm and Carry On
1. Know your estate
Whether you have inherited a vulnerability management tool/programme or configuring one from scratch, you must have an idea of what assets you are scanning.
What is in your estate? Laptops, Desktops, Servers, Networking gear. What OS are they? What are your most important Assets?
Work with people who know about your infrastructure to figure out what exactly you should be scanning. For us, the infrastructure team have invaluable knowledge on this.
As a general rule of thumb, things hosted in your organisation’s environment should be scanned. Your organisation is responsible for the remediation of any vulnerabilities within your estate. If you are scanning things in the cloud, you may need to get permission.
Within the past year, I have gone through an exercise where I reviewed the assets we were scanning. With the help of the infrastructure team, I found that the list of assets/IP addresses we were scanning and the types of scans running on them was not accurate or appropriate for our current estate. This is because while our infrastructure had changed over the past few years, our scanning setup in our tool did not. Our scanning strategy was not setup in such a way that it could take into account the dynamic nature of our estate e.g. moving from cloud to cloud, dropping Certain OS’ and adopting others, decommissioned infrastructure etc. Your scanning strategy must be able to cater for changes in your estate.
2. Not every scan is the same.
Scans are configurable. They can be configured to do broad and comprehensive checks or very specific vulnerability related checks. You may want to scan a web server with a different scan config than that of a networking switch.
Knowing this is important for two things:
- Scanning your entire estate with the same scan config is probably not the most effective scanning strategy. It will cost more in terms of resource consumption and scan times and could produce results that are puzzling.
- If you know what you are scanning for, verifying remediations becomes much quicker. I had an issue where a TLS vulnerability was fixed but subsequent scans kept picking up the same vulnerability. I went back and forth with the vendor and even went as far as saying their detections are wrong! only to later figure out that the scan config I was using wasn’t even looking for the issue we resolved. So it was reporting an old historical finding that doesn’t exist anymore. You could argue this is a tooling issue but the point is to be aware that not all scans are the same.
3. You cannot resolve all Vulnerabilities
When you open your VM tool, you will likely see that you have hundreds or thousands of vulnerabilities outstanding. I would say this is normal. There are new vulnerabilities published almost every day. Most orgs don’t have the resources to have a dedicated Vulnerability Management team. And even when they do, that team wouldn’t usually have the ability to remediate things. Rather, they can flag items to the teams who can do the actual remediation e.g. network admins, server admins, service desk.
Smaller orgs might have a smaller estate so in theory less to resolve. But the problem is a lack of resources to work on remediation.
Larger orgs may have a dedicated Vulnerability Management team but the estate is too big to fix everything.
Bottom line — it’s normal to have a lot of vulnerabilities reported but it’s important to have a process or plan to work on them.
4. Prioritisation VS Coverage
It is logical to prioritise vulnerabilities that are “Critical” but the Criticals-First strategy brings some problems:
- Teams forget about non-critical vulnerabilities. If your teams only work on Criticals, how will you ever tackle the less severe items? There will always be Critical Vulnerabilities released monthly so you will be occupied with those.
- Context — Is a critical rated vulnerability actually a critical for your estate?
- How do you surface priority items while still trying to address items deemed less of a priority? This is a problem I have been trying to resolve this year. One of the things I am doing is gathering a list of assets that are deemed absolutely critical. I have asked around the company for input on this from a variety of teams, not just IT. I want to have business context as well as technology context when assessing vulnerability risks. Which assets do we really care about? Ones that if they went down would cause reputational damage, financial damage, loss of customers, loss of trust, disruption of core business and technology functions.
5. Look for help.
It is impossible to know the ins and outs of every vulnerability you come across. You may have strong Windows knowledge but not much on Linux. You may know a lot about browser security but not much about networking gear. It’s easy to feel overwhelmed with the amount of information we deal with. You may even feel out of your depth. That’s ok. When you do feel like this, ask for help. I don’t know the in-depth technical aspects of every vulnerability I come across. I do my own research, consult others, study, ask a community forum:
- Vendor customer forums — often someone has already asked your question here as they have the same detections in place for their estate and likely seeing similar things to you.
- Twitter/Mastodon
6. Make remediation a process that fits into the remediation teams’ workflow.
Make remediation easy for the teams that will actually do the work. If they work on tickets, automate vulnerability findings into tickets in their queue. Be careful not to overwhelm them though. It needs to be a sustainable process.
When I first started in this role, the remediation teams had hundreds and hundreds of auto generated vulnerability tickets. And the amount of tickets was growing daily. It simply wasn’t sustainable for them to work on these tickets. You need to make remediation a process that is sustainable and effective instead of playing whack-a-mole. What that process looks like may be different for different organisations. I have one here which drip feeds tickets with solutions to our remediation teams. We have weekly meetings to discuss progress and so far we have managed to address a lot of our highest risk vulnerabilities. I am not saying this is the best way to do things but it has somewhat worked for us. We still have challenges we need to overcome and there is a lot of room for improvement in efficiency and volume.
7. One solution can address many vulnerabilities.
In the early days, when I opened our vulnerability management tool I could see so many vulnerabilities I would start to feel overwhelmed. I would think there is so much information to get through and VM is not my full time role, only a small part of it. Where should I start? One thing I quickly realised is that one solution can address multiple vulnerabilities. The tool we use has a feature that can group solutions that address multiple vulnerabilities. I just needed to tell it the scope. e.g. I want to know all the solutions required to tackle Critical Vulnerabilities across our Linux Server estate. There were 68 Vulnerability instances but just 4 solutions to address them all. This doesn’t seem so overwhelming anymore. This can easily be relayed to stakeholders to ease any sense of alarmist reading of metrics.
8. It’s ok to say No.
This one goes for anything. With VM, I have a lot of stakeholders. Management, remediation teams, normal users, committees. Stakeholders obviously have a right to ask questions, question reporting and metrics, send you requests and improvement suggestions. However, they are often things that can be addressed later or you simply do not have the capacity to work on them. You don’t always need to drop everything and work on those requests straight away. You need to be empowered to say “Thanks. I can’t do that right now but I will look at it at a later date”. I make a ticket for myself in the team backlog which we then prioritise accordingly in planning. They are valid requests but I just don’t have the capacity to do them all right now.
9. Your VM tool could be wrong.
I am not saying you should question everything your tool finds but there are occasions where your tool could be reporting incorrectly. Your tool is software and lines of code. Mistakes can be made. There have been several occasions where detections in our tool have been incorrect which lead to false positives being reported or vulnerabilities not being reported at all where they actually did exist.
Look out for anomalies in charts, trends or something you didn’t expect to see in your reporting. It’s good practice to question anomalies. Our stakeholders regularly do this when we send out reports and although it can sometimes be frustrating, more often than not they help to spot something we may have missed. VM, like Security, is a team sport.
When you notice an issue with what your tool is reporting, escalate and engage your tool support contacts straight away. Provide proof as to why you think something is wrong. Keep in mind everyone makes mistakes, developers, analysts and vendors so give them a chance to fix it. Obviously, if this keeps happening or the mistake is something catastrophic, you might want to look at a different vendor.
10. Zero days — Keep calm and carry on.
I cringe writing such a cheesy line but it’s true. When a zero day is published, I find there is a lot of sensationalist, click bait reporting that makes it sound like the world is going to end tonight. Sure, the Zero day published might be extremely serious but don’t panic. You can only do what you can do physically and mentally. Get a grip of the details. Your tool may take some time to have detections in place so do your own investigation straight away. Ask around in your company, look through your software/hardware inventory if you have one.
Some questions to think about:
- Is anything in your estate actually affected?
- If yes, what is the scope?
- Is the Zero day specific to a certain OS, software version?
- Are there any fixes available?
- Are there any mitigating actions you can take?
It’s ok not to have all the answers straight away. Get the appropriate teams involved and come up with a plan. You can only work with the information you have available to you unless you are a malware engineer. It’s not a bad idea to look through any available POCs though.
Keep an eye out for updates on vendor security bulletins, Twitter/Mastodon and security websites such as bleepingcomputer, theregister, darkreading etc.
