  
	
  
  
	
	  htdig
	
	
	  ht://Dig Copyright © 1995-2004 
THANKS.htmlThe ht://Dig Group 	  Please see the file 
COPYINGCOPYING  for
	  license information.
	
	
	
	  
		
		  Synopsis
		
	  
	  
		htdig [
options] [start_url_file]
	  
	
	
	  
		
		  Description
		
	  
	  
		Htdig retrieves HTML documents using the HTTP protocol and
		gathers information from these documents which can later be
		used to search these documents. This program can be
		referred to as the search robot.
	  
	
	
	  
		
		  Options
		
	  
	  
		
		  
			-a
		  
		  
			Use alternate work files. Tells htdig to append 
			.work
 to database files, causing a second copy of
			the database to be built. This allows the original
			files to be used by htsearch during the indexing run. When
			used without the "-i" flag for an update dig, htdig will
			use any existing .work files for the databases to update.
		  
		  
			-c 
configfile		  
		  
			Use the specified 
configfile file instead of the
			default.
		  
		  
			-h 
maxhops		  
		  
			Restrict the dig to documents that are at most 
			maxhops
 links away from the starting document.
		  
		  
			-i
		  
		  
			Initial. Do not use any old databases. This is
			accomplished by first erasing the databases.
		  
		  
		      -m 
url_file		  
		  
		         Minimal. Index only the URLs listed in
			 
url_file and no others.
			 A file name of "-" reads from STDIN.
			 See also the 
start_url_file argument.
		  
		  
			-s
		  
		  
			Print statistics about the dig after completion.
		  
		  
			-t
		  
		  
			Create an ASCII version of the document database. This
			database is easy to parse with other programs so that
			information can be extracted from it for purposes other
			than searching. One could gather some interesting
			statistics from this database.
			
Each line in the file starts with the document id 
			followed by a list of
			
\tfieldname:value.
			The fields always appear in the order listed below:
			
			
			
 fieldnamevalue			
 u URL 			
 t Title 			
 a State (0 = normal, 1 = not found, 2
			= not indexed, 3 = obsolete)
			
 m Last modification time as reported
			by the server
 
			
 s Size in bytes 			
 H Excerpt 			
 h Meta description 			
 l Time of last retrieval 			
 L Count of the links in the document
			(outgoing links)
			
 b Count of the links to the document
			(incoming links or backlinks)
			
 c HopCount of this document 			
 g Signature of the document used for
			duplicate-detection
			
 e E-mail address to use for a
			notification message from htnotify
			
 n Date to send out a notification
			e-mail message
			
 S Subject for a notification e-mail
			message
			
 d The text of links pointing to this
			document. (e.g. <a
			href="docURL">description</a>)
			
 A Anchors in the document (i.e. <A
			NAME=...)
			
		  
		  
			-u 
username:password		  
		  
			Tells htdig to send the supplied username and password
			with each HTTP request. The credentials will be encoded
			using the 'Basic' authentication scheme. There 
			HAS
 to be a colon (:) between the username and
			password.
		  
		  
			-v
		  
		  
			Verbose mode. This increases the verbosity of the
			program. Using more than 2 is probably only useful for
			debugging purposes. The default verbose mode (using
			only one -v) gives a nice progress report while
			digging. This progress report can be a bit
			cryptic, so here is a brief explanation. A line
			is shown for each URL, with 3 numbers before the
			URL and some symbols after the URL. The first
			number is the number of documents parsed so
			far, the second is the DocID for this document,
			and the third is the hop count of the document
			(number of hops from one of the start_url
			documents). After the URL, it shows a "*" for
			a link in the document that it already visited,
			a "+" for a new link it just queued, and a "-"
			for a link it rejected for any of a number of
			reasons. To find out what those reasons are,
			you need to run htdig with at least 3 -v options,
			i.e. -vvv. If there are no "*", "+" or "-" symbols
			after the URL, it doesn't mean the document was
			not parsed or was empty, but only that no links
			to other documents were found within it. With
			more verbose output, these symbols will get
			interspersed in several lines of debugging output.
		  
	          
		        
start_url_file		  
		  
		         A file containing a list of URLs to start indexing
			 from, or "-" for STDIN.  This will augment the default
			 
attrs.html#start_urlstart_url 			 and override the file supplied to
			 [-m 
url_file].
		  
		
	  
	
	
	  
		
		  Files
		
	  
	  
		
		  
			
attrs.html#config_dirCONFIG_DIR /htdig.conf
		  
		  
			The default configuration file.
		  
		
		
		  
			
attrs.html#database_dirDATABASE_DIR /db.docdb
		  
		  
			Stores data about each document (title, url, etc.).
		  
		
		
		  
			
attrs.html#database_dirDATABASE_DIR /db.words.db,
			
attrs.html#database_dirDATABASE_DIR /db.words.db_weakcmpr
		  
		  
			Record which documents each word occurs in.
		  
		
		
		  
			
attrs.html#database_dirDATABASE_DIR /db.excerpts
		  
		  
			Stores start of each document to show context of
			matches.
		  
		
	  
	
	
	  
		
		  See Also
		
	  
	  
		
htmerge.htmlhtmerge ,
		
htsearch.htmlhtsearch ,
		
attrs.htmlConfiguration file format , and
		
http://www.robotstxt.org/wc/norobots.html		A Standard for Robot Exclusion
.
	  
	
	
	Last modified: $Date: 2004/06/12 13:39:13 $
  
