| T.R | Title | User | Personal
Name | Date | Lines |
|---|
| 3934.1 | | RANGER::WASSER | John A. Wasser | Tue Aug 06 1996 13:34 | 11 |
| 3934.2 | | VAXCPU::michaud | Jeff Michaud - ObjectBroker | Tue Aug 06 1996 13:49 | 21 |
| 3934.3 | | RANGER::WASSER | John A. Wasser | Wed Aug 07 1996 10:54 | 20 |
| 3934.4 | Typos -- Corrected in current build | NETRIX::"[email protected]" | Chris Lord | Wed Aug 07 1996 18:12 | 8 |
| 3934.5 | | VAXCPU::michaud | Jeff Michaud - ObjectBroker | Wed Aug 07 1996 20:28 | 52 |
| 3934.6 | additional notes | TUXEDO::ROSENBAUM | Rich Rosenbaum | Fri Aug 09 1996 00:31 | 14 |
| 3934.7 | | MR1MI1::VILCANS | | Fri Aug 09 1996 14:18 | 9 |
| 3934.8 | Warning: Crawler on the Intranet is following links w/query arguments! | VAXCPU::michaud | Jeff Michaud - ObjectBroker | Mon Mar 24 1997 16:30 | 27 |
| > - at some point the crawler will have the (optional) ability to
> crawl link URLs that contain "?" query arguments. This is necessary
> to support Lotus Domino servers which use query syntax for
> perfectly ordinary pages (well not _completely_ ordinary, they are
> dynamically generated from Lotus databases).
Looks like this support is now there. Someones trying to index
my site, including the generated url's that contain query
arguments. It's going to take them probably 9 million hits
(or more to get all the combinations of authors and notesfiles
and personal names) to complete.
The user agent that I'm getting hit by is:
User-Agent: AltaVista Intranet V1.0 sbu_antony [email protected]
and I've sent mail to them letting them know that they better
have alot of disk space (and time, at the rate they are going
I compute it's going to take them 5+ years to finish :-).
Can't the crawler only following links with query arguments if
the server *is* a "Lotus Domino" server?
I'd add support for .txt files to my http server so I can support
a robots.txt, but it's too late now as it appears the crawler
only tries to fetch robots.txt once, no matter how many links or
how much time has past ...
|
| 3934.9 | | teco.mro.dec.com::tecotoo.mro.dec.com::mayer | Danny Mayer | Tue Mar 25 1997 08:49 | 8 |
| > I'd add support for .txt files to my http server so I can support
> a robots.txt, but it's too late now as it appears the crawler
> only tries to fetch robots.txt once, no matter how many links or
> how much time has past ...
How about cutting the connection so that it has to start again?
Danny
|