Vulnerability assessment: how to tell if the critical findings in your report are real

What a vulnerability assessment is, and what it sees from outside

A vulnerability assessment is a check that looks for the known weaknesses of a system and ranks them by severity. The kind that most often lands on a business owner's desk is the external, automated one: a scanner asks the website the questions anyone on the internet could ask (which files it serves, which certificate it uses, how email is configured) and tries to interpret the answers. Quascan works this way, without ever entering the company's systems.

The limit is the interpretation. Many sites answer in ways the scanner does not expect: the friendly page where an error should be, Cloudflare's anti-bot screen covering the real site, the email configuration written on the domain without the www while the scanner looks for it on the one with the www. Faced with an answer like that, a scanner can declare the check inconclusive, or it can guess. Guessing almost always produces an alert, because a report full of red looks more useful than one that admits what it could not see. And a report full of red makes the fix easier to sell.

It went wrong for us in several different ways:

• a company with its email protected as strictly as possible was told that protection was missing entirely, because the scanner was looking for it at the wrong address
• a name server slow to answer turned into email protection declared absent
• a certificate installed without one of its intermediate pieces ended up in the report as a site without HTTPS, in red, while the padlock was right there in the browser
• an application that answers any address with its home page was credited with WordPress files it has never had

The first fix was wrong too

To recognise courteous servers we added a simple check. Before looking for files, the scanner asks for a made-up address and keeps the answer aside; if a "found" file then has roughly the same length, it is discarded. The site from the beginning went from 71 to 95, with no criticals left. We also recomputed every score we use to compare sites with each other, because they had been built on the dirty numbers: the same principle as measuring on real data before trusting a comparison.

A month later, two reports that had gone out to clients were wrong again. One said "database copy exposed". That server inserted a random code into every page, different on each request, so two identical answers always had different lengths. The check written to prevent false alarms had no way of noticing.

The rule: a critical finding needs proof

In September we went through every check with a single rule. A problem goes into the report when there is proof. For an exposed file, the proof is having read its content and recognised it as that exact file. For a certificate that fails verification, the scanner has to say what is wrong: expired, issued for another name, incomplete chain.

When a check cannot reach a conclusion, because the server does not answer or shows an anti-bot screen, the result goes into a separate section of the report, the inconclusive checks. It stays there, and never becomes an alert. The report is shorter now, and your own technician can verify every line in it.

For an internal audit we would make the opposite choice. With a team that can check in five minutes, a false alarm costs a ticket closed at once and a missed problem costs far more, so reporting everything is right. A report sent to management has a different reader: whoever receives it cannot check an alert alone and passes it to the IT lead or to the website supplier. If the first critical that person opens is false, the whole document loses credibility, including the true things written in it. For the same reason we prefer systems that declare when they cannot answer: we cover it in the note on a switch for every fragile integration.

Vulnerability assessment or penetration test: which one you need

They are two different jobs answering two different questions. A vulnerability assessment asks which known weaknesses are present, and returns a list ranked by severity. It is largely done with automated tools, costs less and can be repeated often, for example after every major release. A penetration test asks how far a real attacker would get: a person tries to exploit the weaknesses, one after another, and documents the paths that work. It takes more time and more expertise, and is done less often.

For a company starting from scratch, the sensible order is almost always the same: an assessment first, then verification of the criticals it produces, then a penetration test on the parts that matter most, such as a business system exposed online or the customer area. A critical from an automated assessment is a hypothesis. A penetration test, or a manual check, tells you whether it can be reached.

NIS2 for small and mid-sized companies: is a vulnerability assessment enough?

On its own, no. NIS2, transposed in Italy by Legislative Decree 138/2024, asks for security measures that are mostly invisible from the internet: who has access to what, how backups are made and tested, what happens during an incident, who is accountable inside the company. An external scan of the website covers a small slice of that list. It is useful for finding doors left open and for building evidence over time, and on its own it does not prove compliance.

In Italy, organisations added to the NIS list in 2025 have 18 months from the national cybersecurity agency's (ACN) notice to adopt the baseline measures: for most of them the deadline falls in October 2026. The exact date is on the notice they received. There is also an effect that reaches companies outside the list: organisations subject to NIS2 must keep their suppliers' security under control, and the questionnaires travel down the chain. For many smaller companies the first request for a vulnerability assessment comes from a client, or from a tender in the public sector.

In that case it pays to hand over a report that survives a second reading. A document with false criticals, passed on to a client, opens questions that will then have to be closed one by one.

What to ask whoever sends you the report

These questions fit in one email, and the answers make sense without being technical.

• For each critical finding, what is the proof? If the answer is that the server responded in a certain way, the proof is missing.
• Does the report separate problems found from checks that failed? If everything is green or everything is red, with nothing in between, something was guessed.
• Does the report say what it cannot see? Backups, staff access and procedures are invisible from outside. For NIS2 that is the part that weighs most.
• Is the author of the report also the one selling you the fix? If so, have someone else check the criticals before approving the spend.
• Does the same check, run again today, give the same result? An alert that comes and goes on every scan depends on how the server answers.

If you have a report in hand and do not know how far to trust it, we read it with you and tell you which criticals have proof behind them. It is part of our work in cybersecurity, for companies in general and for those working in the security sector. When security has to be built into a new business system or application, it enters the custom software project from day one. How the scanner is built is told in the case, and Quascan is also listed among our products.