[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[TECH] Possible CVE Confidence Levels
Following is a proposal for different approaches to CVE confidence levels. This proposal includes 2 different options: - 3 levels of confidence (supports simplicity) - 4 levels of confidence (supports the "web of confidence") Thanks to Barbara Pease (of the CVE content team) and Andre Frech for developing these levels with me and writing various sections of this description. - Steve Background rationale -------------------- In the fields of biological and medical research, the process for confidence building for an investigator's work is formal and well established. First, a laboratory builds its own confidence that it correctly understands the nature of the problem and has given the best possible interpretation of experimental results by reproducing the results using similar or entirely different experimental techniques. The varied approaches increase confidence through the constancy and the consistency of the data. This would be analogous to serious testing and analysis where no source code is available, since biological systems under investigation are black boxes with only partial characterization. At that point, the laboratory team submits a paper to a reputable journal. The paper goes out for peer review by other researchers engaged in work in the same field. The reviewers often make suggestions for additional experimentation. The laboratory carries them out and then resubmits a revised paper. A lone publication within the research community is still not enough. Once the paper is published, other interested laboratories using the methodology section of the paper try to reproduce the results. Often they do additional work supporting the original contention. Those laboratories publish their papers just as rigorously refereed confirming the original publication. The original workers become established with the highest confidence in the literature, and people can accept the work and build on it to further elucidate a line of investigation. The vulnerability analysis community does not come close to this approach for establishing confidence. There is no formal peer review. There are varying methods for verifying that a vulnerability exists, but they are not standardized or even well-documented. In addition, the volume of work, and the speed with which the body of knowledge is growing, suggest that such an established academic approach is not feasible in the short term, if ever. 3 Levels of Confidence ---------------------- Andre, Barbara, and I considered more granular levels of confidence (between 5 and 6 levels), but they were too specific to our own individual perspectives and intended uses of those levels. They could be too difficult to easily describe to the average user. We arrived at 3 levels of confidence that were agreeable to all of us, with minor variations. Note that the levels of confidence are described from the perspective of the individual or organization who is establishing the level of confidence in the vulnerability information. Different entities may have different levels of confidence in the same report based on their own expertise, who they trust, etc. There is also the assumption that the entity who is establishing the confidence is experienced in information security analysis and/or testing. In the context of CVE, each voter could assign the amount of confidence that the voter has in the specific candidate. Highest level - CONFIRMED ------------------------- - a security-aware vendor has confirmed the problem - I or my organization has confirmed the problem - an organization trusted within the community for its experience in information security has confirmed the problem with testing. The core rationale behind the CONFIRMED level is that individuals or organizations experienced in information security issues have proven that the problem exists with hands on testing. This would be the highest confidence level based on the existence of vendor security advisories demonstrating astute security awareness and knowledge, in combination with either vendor patches or bug fix references, e.g. developers' README files. Vendor reports of default configurations creating a security problem should also be included. Reputable organizations that are experienced and knowledgeable in the field of information security could be included in this level. RATIONALE -- When duplicating a problem, one may instrument the systems while reviewing hardware logic and source code where relevant. Some bugs are exceedingly complex and defy a simple source code analysis. Even if source is available, the source code reviewer needs a certain level of expertise to analyze the problem. The affected software vendor may not have the expertise to recognize that there is a problem in the first place - or they may not even acknowledge that the problem exists. If a fix is available, it has to be tested to make sure that the problem was really fixed. This process can confirm both the complex cases and those fixes where source code analysis identified the problem in simpler cases. Preparing patches and bug fixing for releases is done with source code available, experience in the product and the opportunity to discuss the matter with others also experienced in the product. In the case of security vulnerabilities, as long as the vendor is knowledgeable in security matters, then there is reasonable confirmation that the bug fixed presents a real security problem. For a CONFIRMED level of confidence, we could include technical reports, email, verbal communication, and technical memoranda and advisories reporting well thought out tests from organizations other than the vendor, if their line of business uses such testing, and the organization is reputable. The organization would stand on its own reputation and extensive experience. Middle level - LIKELY --------------------- - multiple sources have confirmed the problem with testing, but their experience level is unknown (proof of testing is a shared exploit or test) - multiple sources report a problem, but without proof of testing - vendor reports a problem but there is no proof of testing (i.e no patch, work around or statement of tests is available). - vendor reports a problem, but vendor is inexperienced in information security and is clearly responding to recommended fixes from sources established at the LIKELY level, rather than the CONFIRMED level - exploit source code exists, but hasn't been tested by me or my organization - a reliable source (community-trusted) has reported a problem This middle level would indicate a reproducible but informal vulnerability testing where exploitation is reported by several people. The experience and background of the announcers (or those who replicate the issue) may not be known. Reports showing astute observation and speculation about a configuration problem may also be included. Vendor reports and fixes in which the issue is addressed without sufficient proof that it is exploitable, or in cases in which the vendor does not necessarily have the skills to confirm the issue themselves, could be recorded at this confidence level. RATIONALE -- The reporting mechanisms and methods of verification are informal but reasonable. There should be an exploit or test that people are using to confirm the original report. The technical content should show observation and insight, but since access to source code may not be possible and the experience level of the testers may not be known, and it cannot be confirmed with the available data. These references would more than likely be informal mailing list discussions, private communications to colleagues at other institutions about some hacking around with the problem, etc. The work is ad hoc but indicates that the reporters have some skills in identifying vulnerabilities. Vendor advisories that do not have patches available might be included here. The hope is that some discussion at least has taken place with knowledgeable people at the vendor site, but without a patch it is not clear that any tests were run, unless there is confirmation from a reliable source. One could also include vendor advisories and patches from vendors where the level of their security expertise is not clear. They may fix a bug that others at the same confidence level report, without insight as to whether or not it presents a security problem. Lowest level - UNCERTAIN ------------------------ - disputed without independent confirmation at the CONFIRMED level - announced without any references or discussion of testing - no references This level could include reports with poor references or no references. It could include reports in which no test or exploit is included, and where no one else reports that they duplicated the problem. This could include reports that are in dispute, but the evidence cited is not at the CONFIRMED level. One issue is the case in which a security-aware vendor disputes a claim from a CONFIRMED-level, trusted organization. RATIONALE -- there is insufficient evidence to prove that the issue is real. 4 levels of confidence ---------------------- An alternate approach is to establish another "middle" level of confidence, the TRUSTED level. The TRUSTED level basically separates the items in the CONFIRMED and LIKELY levels that deal with an organization's trust in third party reporters. Here's a simplified writeup of the 4-level approach. CONFIRMED: - the vendor has confirmed the problem - I or my organization has confirmed the problem TRUSTED: - I trust the entity who announced the problem - I trust an entity who said they confirmed the problem LIKELY: - multiple sources have confirmed the problem, but they are not trusted - there is a plausible exploit that hasn't been tested by me or my organization UNCERTAIN: - disputed - announced without any confirmation - no references With a TRUSTED level, there is no need for the community to decide which organizations are to be trusted at the CONFIRMED level. Pro's and Con's of 3-level versus 4-level confidence ---------------------------------------------------- If there is not a TRUSTED level, then the 3-level approach is easier to describe to consumers of vulnerability information. It can avoid "politicizing" which organizations can be trusted and more cleanly separates *who* is to be trusted from *how* those trusted entities are used in assigning confidence. If there is a TRUSTED level, then each organization that applies a confidence level to vulnerability information could effectively "formalize" which organizations they trust. If a person trusts organization X, and X trusts Y, then the person might be more willing to trust reports from Y. This could be used to more cleanly represent a "web of trust" or "web of confidence" and differentiate between the vendor, the entity that assigns the confidence to the issue, and all other third parties.