[TECH] Analysis of vendor acknowledgement according to CVE

["Steven M. Christey" <coley@linus.mitre.org>]

Elias Levy gave some recent statistics from the Bugtraq database on
vendor notification.  Following his lead, I've hacked up a different
set of statistics that I culled from CMEX, which these days is
basically the candidate database.  I can't provide good breakdowns
based on who did the initial disclosure.

The short story is that according to CMEX, about 50% of the candidates
that were announced between November 1999 and July 2000 don't have
vendor acknowledgement.  Most candidates that become official entries
have some sort of vendor acknowledgement.

Candidates were taken from the RECENT-01 through RECENT-29 clusters.
These clusters encompass candidates that were publicly disclosed
between November 24, 1999 and July 20, 2000.  More recent clusters
were omitted from the analysis because they are too new to determine
the impact of vendor acknowledgement on Board voting.  However, 42% of
the candidates from clusters RECENT-30 through RECENT-35 did not have
vendor acknowledgement.

Note that how I determine vendor acknowledgement is probably pretty
strict relative to what others might consider sufficient
acknowledgement.  I described the approach at the last Board meeting,
but details are provided at the end of this report.

Summary
-------

Total candidates: 690 (wow!  at least 690 new issues in 8 months!)

Candidates still active: 443

Candidates that became official entries: 247

Candidates with insufficient votes: 162

Candidates with some sort of vendor acknowledgement: 341 (51% of 690)


Types of Acknowledgement
------------------------

- Advisory by vendor or response team (e.g. CERT/CC or AusCERT)

- Acknowledged by vendor in some other fashion (e.g. patch, README,
  email followup)

- Announced by reliable source (e.g. an established security vendor
  such as L0pht, ISS, BindView, etc.)

- No acknowledgement / Could not find acknowledgement

A vendor advisory is a formal announcement by an established vendor
who has set up a web page to contain all their advisories.  Basically,
these are OS vendors and application vendors such as Allaire who are
used as "official" references for CVE.

If a security vendor is the first announcer, but the affected software
vendor posts an advisory, then it counts as a vendor advisory.  A
response team advisory is assumed to contain some sort of vendor
acknowledgement.


Candidates that became entries
------------------------------
 110 Advisory by vendor or response team
  63 Acknowledged by vendor in some other fashion
   6 Announced by reliable source, e.g. L0pht, ISS, BindView, etc..
     - Normally implies vendor acknowledgement.
  68 No acknowledgement / could not find acknowledgement

Total: 247
Percentage with No/unknown acknowledgement: 28%


Candidates that have sufficient votes to become entries
-------------------------------------------------------

[Note: many of these candidates are being held back by content
decisions or other factors, thus they might not become entries.]

  59 Advisory by vendor or response team
  60 Acknowledged by vendor in some other fashion
  6  Announced by reliable source, e.g. L0pht, ISS, BindView, etc..
     - Normally implies vendor acknowledgement.
  96 No acknowledgement / could not find acknowledgement

Total: 221
Percentage with No/unknown acknowledgement: 43%


Candidates with insufficient votes
----------------------------------

[These candidates do not have sufficient votes to be ACCEPTed by the
Board.]

   2 Advisory by vendor or response team
   4 Acknowledged by vendor in some other fashion
   0 Announced by reliable source, e.g. L0pht, ISS, BindView, etc..
     - Normally implies vendor acknowledgement.
 156 No acknowledgement / could not find acknowledgement

Total: 162
Percentage with No/unknown acknowledgement: 98%


The stats for candidates with insufficient votes is pretty
illuminating in terms of the impact on CVE.  Approximately 60
candidates had voting conflicts, which for various reasons removes
them from the statistical pool.


Here are the overall stats:

 179 Advisory
 137 Acknowledged
  25 Announced by reliable source
 349 No acknowledgement

Total: 690
Percentage with No/unknown acknowledgement: 51%


Severity vs. Acknowledgement
----------------------------

These stats were taken from a little-used field in CMEX that describes
the specific type of flaw (buffer overflow, dot dot, etc.)  The field
isn't filled in half the time, partially because it's not an essential
field, and partially because some bugs are difficult to classify.
Note that it is only filled in for the purposes of these sorts of
analysis of CVE - it is not distributed anywhere else.

The following severity values were assigned.  Broad strokes were
applied - e.g. if you have a buffer overflow, then it's assumed that
you can execute arbitrary commands; if you have a dot dot problem,
it's assumed you can read files; etc.  So, this data is clearly
suspect.

SERIOUS = execute commands or read files
MIXED = type of flaw may or may not be serious depending on implementation
NOT-SERIOUS = information gathering / etc.
UNKNOWN = flaw type not available

For entries:
  60 ACK SERIOUS
  28 ACK MIXED
   3 ACK NOT-SERIOUS
  88 ACK UNKNOWN
  28 NOACK SERIOUS
   2 NOACK MIXED
   2 NOACK NOT-SERIOUS
  36 NOACK UNKNOWN

For candidates:
  50 ACK SERIOUS
  24 ACK MIXED
   1 ACK NOT-SERIOUS
  85 ACK UNKNOWN
  89 NOACK SERIOUS
  46 NOACK MIXED
   4 NOACK NOT-SERIOUS
 138 NOACK UNKNOWN
			
		


Monthly breakdown of acknowledgement
------------------------------------


Month   Some ACK    No ACK
-----   --------    ------
199911    14          17
199912    47          45
200001    31          33
200002    48          38
200003    25          30
200004    32          40
200005    58          48
200006    58          71
200007    28          25



Appendix: Determining Vendor Acknowledgement for CVE
----------------------------------------------------

Here are the basic guidelines I use for determining if a candidate has
vendor acknowledgement.  Where there is acknowledgement, I've started
annotating CMEX with the "method" of acknowledgement, but that data
isn't particularly reliable.

Sufficient acknowledgement:

 - Vendor posts an advisory to *Bugtraq or web site

 - Vendor posts an email followup to a *Bugtraq post

 - Vendor explicitly describes issue on vendor web site

 - Vendor is quoted in established media (e.g. cnn.com)

 - Vendor explicitly describes issue in README or Changelog

 - Vendor provides a patch whose description clearly indicates the bug
   fix (even if they attempt to obscure it).  This one can get fuzzy
   because they might not be fixing the problem you think they are, so
   I'm usually careful to avoid calling this acknowledgement.

Insufficient acknowledgement:

 - Non-vendor claims that a patch is available (rationale: the patch
   may not actually fix the bug).  If the patch can be downloaded and
   the associated README/etc. obtained without running a program, then
   the README is examined for acknowledgement

   NOTE: this is *any* non-vendor, even an otherwise "trusted" source.
   I believe that patch information in some sources of vulnerability
   information may come from a cut-n-paste from the original
   announcement of the patch, regardless of the trustability of the
   source, and it isn't necessarily verified.

 - Discloser claims that the vendor was contacted or provides some
   quote

 - Vendor requires web site registration, or customer support number,
   to enter portion of web site that might contain acknowledgement