RE: Question about robots.txt

[jericho <jericho@attrition.org>]

On Tue, 8 Dec 2015, Coffin, Chris wrote:

: We made the choice a long time ago to not allow indexing of the 
: cve.mitre.org web site. At least part of that decision was simply 
: resource constraints ? when CVE was in its toddler years, search 
engine 
: indexers were very resource intensive.

That 'decision' was based on crap excuses, even back then. =) As 
someone 
who ran two sites over the time MITRE ran CVE, and intensively watched 
logs on one of them (attrition.org, since 1998-10-07), search engines 
were 
NOT resource intensive back then. Attrition staff talked about that 
issue 
and didn't block any of our content in robots.txt because search engine 
spam was present, but not heavy. For those interested in Internet 
history...

forced ~$ more /home/admin/util/list.filter
72.14.203.104
forced.attrition.org
images.search.yahoo.com
casualgamer.org
myspace.com
stumbleupon.com
f-mai.gif
f-bak.gif
f-att.gif
thefiles.gif
panopta.com
divinelanguage.com
forced ~$ grep -i google /home/admin/util/list.*
/home/admin/util/list.bot:googlebot.com
/home/admin/util/list.bot:Feedfetcher-Google
/home/admin/util/list.filter-old:google.com
/home/admin/util/list.filter-old:google.co.jp/search
/home/admin/util/list.filter-old:google.de
/home/admin/util/list.filter-old:google.fr
/home/admin/util/list.filter-old:google.co.uk
forced ~$

"list.filter-old" is from 2003-08-25. The limited set of Google domains 
should be very telling, given the year and traffic generated.

We actually *stopped* filtering Google at some point, while ignoring 
Yahoo 
early on. Why? Because they were simply not hammering sites and causing 
any undue burden, to a random desktop machine bought at the local 
computer 
store. Those were "ignore displaying those entries in our log parser", 
not 
"block them from reaching our web server" via iptables.

That was Attrition when it was run on a ~ $500 box bought in 1998 and 
hosted on a consumer link, compared to MITRE's resources and CVE 
contract 
money from the government at the time. So to be clear, MITRE's answer 
in 
2015, is based on people forgetting what it was like in 1997 - 1999.

That said, after Kurt's mail in December of 2015... in the last ~ 30 - 
60 
days, I noticed that MITRE finally changed that. Google is now indexing 
and caching the CVE pages.

Thank you, as a long-time taxpayer funding MITRE's projects, including 
CVE, to the tune of $1,487,334,000 in MITRE income last year. Good to 
see 
you making these small changes to help the industry.

: We are currently re-examining this policy and will keep the Board 
: posted.

Except... you didn't. Just like you didn't ask us about the 3k+ 
RESERVED 
fiasco that got several of us talking about this morning, figuring out 
how 
we'd handle it. When NVD spoke up, we all collectively said "hell yeah!"

The fact that NVD called you out, and has since said they will be 
'ignoring' those IDs, is also very significant in CVE history. This is 
the 
first *real* break that NVD has had from CVE ever. There have been 
other 
breaks the last year+, but they were more pedantic and favored NVD over 
MITRE/CVE, based on the time of entries becoming public (e.g. NVD 
published before MITRE did).

Brian