> >The good news is that I'm starting to see some instances of "Item
> >already checked" fla****ng by. This would mean that I probably don't
> >have to wait another 16 hours or more for final results and that
> >there may not be 3,000 broken links by the time it's finished
> >checking the site...
>
> What you mean, I think, is that you're doublecounting some of the
> "repeated" broken links where the same link exists in many places and
> of course each one is broken? Some things that are im****tant.... how
> HUGE the site is, how many links there are!!! And is 3% large or
> small? I think that's about what I would have thought it was. I
> think that's neither large nor small but about what such a huge site,
> originally built over ten years ago, has as a legacy.
There turned out to be a nasty bit of recursion in the site - items
that are in the "DipPouch" folder on the site are physically located in
the root folder... as is the link (in the filesystem sense) that
redirects the traffic there. It's a clever thing to do in a couple
ways. It was just inconvenient for this project. In the end, I had to
'break' that link in order to successfully crawl the site with the
spider. Otherwise I would have gotten ever deeper levels of URLs that
look like:
"diplom.org/DipPouch/DipPouch/DipPouch/DipPouch/DipPouch/DipPouch/..."
The final result is that there are 37776 links to 5892 unique targets
(including images). There are 4824 good links and 945 bad links; The
number for 'bad links' unfortunately including those links that needed
to be tem****arily disabled. I'll be contacting the maintainers
individually with specifics on their sections as soon as I can generate
re****ts.
So I've got a tool that can help find the broken links (with some human
intervention), but the statistics are more obviously useless than is
normally the case (and the recursion makes it difficult to *****s the
size of the site too).
Chris
|
