Skis / Toys / Fun

Thought for the day..week.. month…  I need a good web spider… 

• grub – used by wikia (C)
• heritrix – used by internet archive (java)
• websphinx – random edu project (java)
• larbin – (C++) pretty good and simple, best starting point.

Tags:   crawler