Re: Question about robots.txt

["kseifried@redhat.com" <kseifried@redhat.com>]

On 13/05/17 01:55 AM, jericho wrote:
> 
> On Thu, 11 May 2017, Coffin, Chris wrote:
> 
> : > That said, after Kurt's mail in December of 2015... in the last ~ 
> 30 - 
> : 60 days, I noticed that MITRE finally changed that. Google is now 
> : indexing and caching the CVE pages.
> : 
> : We made the change to allow indexing back in Feb of 2016, which was 
> a 
> : few months after Kurt had pointed out the issue. We apologize to 
> all for 
> 
> Something I cannot prove, because I don't screenshot my daily 
> "missing 
> CVE" searches as far as Google results go. But I would swear this is 
> not 
> the case. We'll have to agree you have your official statement, and I 
> have 
> my 'anecdotal' evidence as someone who searches on new/missing CVE 
> IDs 
> every day.
> 

Nope, archive.org has their robots.txt going back to 2001 with pretty
much daily records:

https://web.archive.org/web/*/cve.mitre.org/robots.txt

> : not replying to the original thread at that time. Dan also 
> mentioned the 
> : same in a response to you back in April of this year 
> : 
> (http://common-vulnerabilities-and-exposures-cve-board.1128451.n5.nabble.com/Re-CVENEW-New-CVE-CANs-2017-04-23-19-00-count-1-td722.html#a727).
> 
> You see, I don't have to read that. Kurt mailed in Dec 2015, you say 
> Dan 
> replied to me in April 2017. Use your numbers. My point stands about 
> MITRE's promise of following up.

You can check archive.org to see the exact day it changed.



-- 

Kurt Seifried -- Red Hat -- Product Security -- Cloud
PGP A90B F995 7350 148F 66BF 7554 160D 4553 5E26 7993
Red Hat Product Security contact: secalert@redhat.com