It seems to be trendy lately to make tools which can create custom or more specific word lists for password cracking, just last week we posted about the web application The Associative Word List Generator (AWLG), which crawls the whole web to look for associated words with a given topic.
This application is more towards creating custom word lists from a specific domain by crawling it for unique words. Basically you give the application a spidering target website and it will collect unique words. The application is written in Ruby and is called CeWL, the Custom Word List generator. The app can spider a given url to a specified depth, optionally following external links, and returns a list of words which can then be used for password crackers such as John the Ripper.
IF you combine the info output by CeWL and AWLG with the standard wordlists for password cracking - you should have a fairly comprehensive set.
By default, CeWL sticks to just the site you have specified and will go to a depth of 2 links, this behaviour can be changed by passing arguments. Be careful if setting a large depth and allowing it to go offsite, you could end up drifting on to a lot of other domains. All words of three characters and over are output to stdout. This length can be increased and the words can be written to a file rather than screen so the app can be automated.
Version 2 of CeWL can also create two new lists, a list of email addresses found in mailto links and a list of author/creator names collected from meta data found in documents on the site. It can currently process documents in Office pre 2007, Office 2007 and PDF formats. This user data can then be used to create the list of usernames to be used in association with the password list.
Installation
CeWL needs the rubygems package to be installed along with the following gems:

• http_configuration
• mime-types
• mini_exiftool
• rubyzip
• spider