| Title: | Internet Tools |
| Notice: | Report ALL NETSCAPE Problems directly to [email protected] . rnet? Read note 448.L for beginner information. |
| Moderator: | teco.mro.dec.com::tecotoo.mro.dec.com::mayer |
| Created: | Fri Jun 25 1993 |
| Last Modified: | Fri Jun 06 1997 |
| Last Successful Update: | Fri Jun 06 1997 |
| Number of topics: | 4714 |
| Total number of notes: | 40609 |
We have need for a subset of spider functionality - namely, crawling the net and returning articles - but we do not need any indexing done. Does anyone know of any software out there that does this ? Our understanding of the AV spider software is that you can't get at just the functionality we need (ie, it seems to have the whole thing - spider, indexing, etc. - as a single package that you can't separate out) Thanks for any help /Jong
| T.R | Title | User | Personal Name | Date | Lines |
|---|---|---|---|---|---|
| 4591.1 | Netscape LiveWire's Site Manager, ForeFront's Web Whacker | LGP30::FLEISCHER | without vision the people perish (DTN 381-0426 ZKO1-1) | Fri Apr 04 1997 11:46 | 15 |
re Note 4591.0 by AIAG::KIM:
> We have need for a subset of spider functionality - namely, crawling the net
> and returning articles - but we do not need any indexing done.
Netscape LiveWire's Site Manager will do this for one site,
i.e., suck up all the pages and make a local copy in the same
directory structure and with fixed links.
There are actually quite a few tools on the market that more or
less do this for people who want to read offline (or read at
speeds greater than real-time fetching will allow). One of
the better tools is Web Whacker -- see http://www.ffg.com/ .
Bob
| |||||
| 4591.2 | AIAG::KIM | Fri Apr 04 1997 14:27 | 4 | ||
Re: .-1 Thanks. /Jong | |||||
| 4591.3 | crawl a range of sites ? | AIAG::KIM | Mon Apr 07 1997 13:02 | 9 | |
Having briefly looked at the ForeFront WebWhacker, my impression is that it is a pretty good and light-weight tool that satisfies the need for off-line browsing and content-delivery. However, I'm looking for a little more general crawler which at minimum is capable to crawl a range of sites (for instance, *.pko.dec.com) and return the data. Unfortunately, I don't see the equivalent functionality available with WebWhacker. Any comments? Thanks /Jong | |||||