Web Crawler…

Thought for the day..week.. month…  I need a good web spider… 

  • grub – used by wikia ©
  • heritrix – used by internet archive (java)
  • websphinx – random edu project (java)
  • larbin – (C++) pretty good and simple, best starting point.