| T.R | Title | User | Personal
Name | Date | Lines |
|---|
| 3807.1 | | CFSCTC::SMITH | Tom Smith MRO1-3/D12 dtn 297-4751 | Tue Jul 02 1996 20:05 | 1 |
| 3807.2 | A Standard for Robot Exclusion: | RANGER::WASSER | John A. Wasser | Mon Jul 08 1996 11:07 | 7 |
| 3807.3 | what Content-type? | AUSS::GARSON | DECcharity Program Office | Thu Feb 13 1997 01:33 | 10 |
| re .*
The previously referenced standard (which seems to have moved anyway)
does not specify what MIME Content-type is expected for robots.txt.
I am using a cut down HTTP server for a couple of legacy wrapper
situations and it hard-codes "text/html" whereas I would expect a full
function HTTP server to specify something else. Does anyone know
what Digital's internal crawlers expect in the way of MIME Content-type?
Will text/anything do? Do they even check the MIME Content-type?
|
| 3807.4 | I'd expect text/plain | HOUBA::MEHERS | Damian, http://bigbird.geo.dec.com/ | Thu Feb 13 1997 02:48 | 1 |
|
|
| 3807.5 | | AUSS::GARSON | DECcharity Program Office | Tue Feb 18 1997 23:34 | 6 |
| re .3
Partly answering my own question...the new draft of this standard (as
yet unfinished) says explicitly that the Content-type must be
"text/plain". It does not however say what happens if the Content-type
is otherwise.
|
| 3807.5 | | AUSS::GARSON | DECcharity Program Office | Thu Feb 20 1997 01:30 | 8 |
| re .3
Partly answering my own question...the new draft of this standard (as
yet unfinished) says explicitly that the Content-type must be "text/plain".
It does not however say what happens if the Content-type is otherwise.
A possible workaround that was suggested to me offline is to use a
proxy server.
|