Enterprise Search Consulting

링크: https://crawler.dev.java.net/

아파치 라이센스이며 루씬과 마찬가지로 라이브러리 형태이다.

간단설명
---------------

• Smart and easy framework thats crawls a web site
• Integrated Lucene support
• It's simple to integrate the framework in own applications
• The crawler can start from one or from a list of links
• Two crawling models available:
  • Max Iterations: Crawls a web site through a limited number of links: Fast model with a small memory footprint and cpu usage.
  • Max Depth: A simple graph model parser without recording in and outcoming links. Fast as the max interations model.
• Accept filter interface to limit the links to be crawled
• Core accept filters available: ServerFilter, BeginningPathFilter and RegularExpressionFilter
• Combining the accept filters with AND, OR and NOT possible
• Plugable http connection libraries HttpClient (default) and HTMLParser (optional)
• Own listeners can be added in the parsing process
• The framework is not a GUI based tool to mirror a website and browse the site offline!

느낌
---------
    - 간단하게 웹로봇을 만들려고 할때 유용한 라이브러리가 될 것 같다.
   - 대용량 로봇은 힘들 듯....
    - 역시나 우리나라에서 많이 사용되는 javascript형태의 링크는 지원을 하지 않는 것 같고 향후 readmap에도 없다.
    - 중복페이지(혹은 중복서버) 검출방법이 살짝 궁금하다는....
      (단순히 url만으로 처리하는지? 아니면 네임서버 분석까지 하는지... 등등...)