

==========================
PhpDig version 1.6.1 Documentation

==========================


Last update : 2003-03-15

Note : Spelling corrections by Brien Louque and John Zastrow.


1. Table of contents
==============================

1. Table of contents
2. Where to find the lastest PhpDig version ?
3. PhpDig Features
4. Installation
5. Configuration
6. Update PhpDig
7. Indexing with web interface
8. Indexing by command line interface
9. Templates
10. Insert PhpDig in a website
11. Inside PhpDig
12. Getting help with PhpDig

2. Where to find the lastest PhpDig version ?
==============================


At address : http://phpdig.toiletoine.net/

3. PhpDig Features
==============================


3.1. HTTP Spidering
-------------------------


PhpDig follows links like any web browser,
to build the pages list to index.


Links can be in an AreaMap or frames. PhpDig supports relocations.
Any syntax of HREF attribute is followed by Phpdig.
Simple javascript links like window.open() or window.location()
are followed too.


PhpDig does not go out the root site you define for the indexing
(but see Domain Indexing option).
Spidering depth is chosen by user.


All html content is listed, both static and dynamic pages.
PhpDig searches the Mime-Type of the document.

3.2. Full Text Indexing
-------------------------


PhpDig indexes all words of a document, excepting small words
(less than 3 letters) an common words, those are definded in a text file.


Lone numbers are not indexed, but those included in words are indexed.
Words with underscores are included as well.


A count of word occurances in each document is also stored.
Words in the title can have a more important weight in ranking results.

3.3. File types wich can be indexed
-------------------------


PhpDig indexes HTML and text files by itself.
PhpDig could index PDF, MS-Word and MS-Excel files if you install
external binaries on the spidering machines to this purpose.

PhpDig is configured using catdoc, xls2csv
and pstotext programs.


You could find catdoc and xls2csv at this url :
http://www.45.free.net/~vitus/ice/catdoc/. Choose the
0.91.5 version. The "stable" version have trouble with some
encodings and does not include xls2csv program.


You could find pstotext at this url :
http://research.compaq.com/SRC/virtualpaper/pstotext.html.


The author does not offer support on those tools. Contact the authors
of those if you have trouble in compiling and/or installing them.


Of course, you can use another tools to do the job. Choice is yours.
Output format must be plaintext.

3.4. Other features
-------------------------


PhpDig Tries to read a robots.txt file at the server root.
It searches meta robots tags too.


PhpDig can spider sites served on another port than the default 80.
Password protected sites can be indexed giving to the robot an username
and valid password.
Be Careful ! This feature could permit to an unauthorized
user reading protected informations. We recommend to create a specific
instance of PhpDig, protected by the same credentials than the restricted
site. You have to create a special account for the robot too.


The Last-Modified header value is stored in the database to avoid redundant indexing.
Also the <META> revisit-after tag.


If desired, the engine can store textual content of indexed documents.
In this case, relevant extracts of found pages are displayed in the results
page with highlighted search keys.

3.5. Display templates
-------------------------


A simple template system adapts search and results page to
an existing site look.
Making a template consists only in insert few xml-like tags in an html page.

3.6. Limits
-------------------------


PhpDig can't perform an exact expression search.


Because of the time consuming indexing process,
the Apache/php web server which performs the process must not be safe_mode configured.

This limit can be turn :
- Using distant indexing with MySql TCP connexion and FTP connexion ;
- Launching indexing process in a shell command. This can be made by a cron task.


Spidering and indexing is a bit slow.
On the other hand, search queries are fast, even in an extended content.


4. Installation
==============================


4.1. Prerequisites
-------------------------


PhpDig requires a Web server (Apache is my preference)
with Php (module or cgi), and a MySql database server.


The following configurations were tested :
PHP 4.0.3pl1 cgi, Apache 1.3.14, MySql 3.23.28, Windows 2000 ;
PHP 4.0.5rc1 module, Apache 1.3.20, MySql 3.23.28, Windows 2000 ;
PHP 4.0.5 module, Apache 1.3.20, MySql 3.23.39, Linux with 2.4 kernel ;
Php/4.1.1, Apache/1.3.20 (Win32), Windows 2000 ;
Php/4.1.2, Apache/1.3.23 (Unix) mod_ssl/2.8.7, Linux kernel 2.4.3 ;
Php/4.3.0, Apache/2.0.44 (Unix) OpenSSL/0.9.6g, Linux kernel 2.4.18.

4.2. Scripts installation
-------------------------


Unzip the archive in a directory and configure Apache to serve it.
(it will be named [PHPDIG_DIR] in the following)
The engine did not need a dedicated VirtualHost to operate.
If PhpDig is installed on an Unix operating system server, set the file
permissions to writable on the following directories, for the suid Apache
server is running :


[PHPDIG_DIR]/text_content
[PHPDIG_DIR]/include
[PHPDIG_DIR]/admin/temp

4.3. MySql database installation
-------------------------


There are two ways to install the database.


- Php install script :
In your favorite browser, request the page :


[PHPDIG_DIR]/admin/install.php


And follow the instructions.
Choose if you want to create the entire database or only the table,
or update from a previous database version of PhpDig.
This script uses the form datas to complete the fields of the
"[PHPDIG_DIR]/include/_connect.php" script and copying it to
"[PHPDIG_DIR]/include/connect.php".


- Manual installation :
You have to create the database (You can choose any other name than
"phpdig") :


#mysql mysql
mysql> CREATE DATABASE phpdig;
mysql> quit

#mysql phpdig < [PHPDIG_DIR]/sql/init_db.sql

Verify that all tables are present :

#mysql phpdig
mysql> SHOW TABLES;

The database answer must be :

+------------------+
| Tables_in_phpdig |
+------------------+
| engine           |
| excludes         |
| keywords         |
| logs             |
| sites            |
| spider           |
| tempspider       |
+------------------+
7 rows in set (0.00 sec)

mysql>


After the database was created, copy the "[PHPDIG_DIR]/include/_connect.php" file
to "[PHPDIG_DIR]/include/connect.php" and edit the new one.
Replace the values "<host>", "<user>", "<pass>",
and "<database>" to your database server URL, the username, the password
to connect to it (if required) and the name you give to the phpdig database.
If you want to prefix all table names by a custom string,
replace "<dbprefix>" to it and rename tables in any MySql client.
If you do not, empty this value.

In a local installation, the values
"localhost", "root", and "" are sufficient in most cases.


To verify the install is complete, open the main page
[PHPDIG_DIR]/index.php with your favorite web browser.
The search form must be visible.


5. Configuration
==============================



After installaton, the engine will work without modification
to the configuration file. The configuration steps depend on your needs.
Don't forget to change the administration login and password
if you use a Php compiled in an Apache dynamic or static module.


Notice : Authentification doesn't operate with a CGI php. In this case,
uses an .htaccess file in order to protect the [PHPDIG_DIR]/admin directory.



All configuration parameters are in the
[PHPDIG_DIR]/include/config.php file.
Each of them is followed by a comment explaining it purpose.


In the following, all statements are lines of the config.php file.

The values are default values.

5.1. Configuring administrator access
-------------------------

Change the following constants. If you don't want to see
a clear password value, use the Apache authentification functions.


define('PHPDIG_ADM_AUTH','1');     //Activates/deactivates the authentification functions
define('PHPDIG_ADM_USER','admin'); //Username
define('PHPDIG_ADM_PASS','admin'); //Password

5.2. Configuring robot and engine
-------------------------

Change following variables and constants.

define('SPIDER_MAX_LIMIT',20);          //max recurse levels in sipder
define('SPIDER_DEFAULT_LIMIT',3);       //default value
define('RESPIDER_LIMIT',4);             //recurse limit for update

define('LIMIT_DAYS',7);                 //default days before reindex a page
define('SMALL_WORDS_SIZE',2);           //words to not index
define('MAX_WORDS_SIZE',30);            //max word size

define('PHPDIG_EXCLUDE_COMMENT','<!-- phpdigExclude -->');
                                        //HTML comment to exclude
                                        //a part of a page
define('PHPDIG_INCLUDE_COMMENT','<!-- phpdigInclude -->');
                                        //HTML comment to stop exclude of
                                        //a part of a page

define('PHPDIG_DEFAULT_INDEX',true);    //phpDig considers /index or /default
                                        //html, htm, php, asp, phtml as the
                                        //same as '/'
define('PHPDIG_SESSID_REMOVE',true);    // remove SIDS from indexed URLS
define('PHPDIG_SESSID_VAR','PHPSESSID');// name of the SID variable

define('TITLE_WEIGHT',3);               //relative title weight
define('CHUNK_SIZE',2048);              //chunk size for regex processing

define('SUMMARY_LENGTH',500);           //length of results summary


define('TEXT_CONTENT_PATH','text_content/'); //Text content files path
define('CONTENT_TEXT',1);                    //Activates/deactivates the
                                             //storage of text content.
define('PHPDIG_IN_DOMAIN',false);            //allows phpdig jump hosts in the same
                                             //domain. If the host is "www.mydomain.tld",
                                             //domain is "mydomain.tld"

5.3. Configuring external binaries
-------------------------


Each external tool is defined by three constants :
- INDEX (true or false) : Activate this file type indexing ;
- PARSE (path) : Executable path ;
- OPTION (options) : Options of program.


define('PHPDIG_INDEX_MSWORD',true);
define('PHPDIG_PARSE_MSWORD','/usr/local/bin/catdoc');
define('PHPDIG_OPTION_MSWORD','-s 8859-1');

define('PHPDIG_INDEX_PDF',true);
define('PHPDIG_PARSE_PDF','/usr/local/bin/pstotext');
define('PHPDIG_OPTION_PDF','-cork');

define('PHPDIG_INDEX_MSEXCEL',true);
define('PHPDIG_PARSE_MSEXCEL','/usr/local/bin/xls2csv');
define('PHPDIG_OPTION_MSEXCEL','');

5.4. Configuring templates
-------------------------

Change following variables and constants.

$phpdig_language = "en";                        //GUI language

$template = "$relative_script_path/templates/phpdig.html";  //Template file path
define('HIGHLIGHT_BACKGROUND','yellow');         //Highlighting background color
                                                 //Only for classic mode

define('HIGHLIGHT_COLOR','#000000');             //Highlighting text color
                                                 //Only for classic mode

define('LINK_TARGET','_blank');                  //Target for result links
define('WEIGHT_IMGSRC','./tpl_img/weight.gif');  //Baragraph image path
define('WEIGHT_HEIGHT','5');                     //Baragraph height
define('WEIGHT_WIDTH','50');                     //Max baragraph width
define('SEARCH_PAGE','index.php');               //The name of the search page

define('SUMMARY_DISPLAY_LENGTH','150');          //Max chars displayed in
                                                 //description
define('DISPLAY_SNIPPETS',true);                 //Display text snippets
define('DISPLAY_SNIPPETS_NUM',4);                //Max snippets to display
define('DISPLAY_SUMMARY',false);                 //Display description
define('PHPDIG_DATE_FORMAT','\1-\2-\3');         // Date format for last update
                                                 // \1 is year, \2 month and \3 day

define('SEARCH_DEFAULT_LIMIT',10);      //results per page
define('SEARCH_DEFAULT_MODE','start');  // default search mode (start|exact|any)

5.5. FTP configuration (if necessary)
-------------------------


PhpDig doesn't indexes FTP sites.


Lot of PhpDig users install it on shared web servers, and on those, Php is always
configured with safe_mode activated.
On those shared hostings, access to the crontab is also not allowed.


Another instance of Php, on a distinct server is the solution.
In my case, a linux server installed at my home and plugged on
a cable connexion runs the update process for
the demo version of PhpDig.
Your host must permit you to connect to your MySql database through TCP/IP.


And what about this famous FTP connection ?
It sends textual content of indexed documents to the proper
directory in the remote server.

If you deactivate the FTP function (in case of low-bandwidth connections,
like by modem), only the summary stored in the database and not the exact document is displayed on the results page.

FTP parameters are the following.

define('FTP_ENABLE',0);              //Activate/deactivate the ftp connection
define('FTP_HOST','<ftp host dir>'); //FTP server name
define('FTP_PORT',21);               //FTP port
define('FTP_PASV',1);                //Use passive mode (PASV), recommended
define('FTP_PATH','<phpdig root dir>'); //Path of the phpdig directory on server,
                                              //relative to the ftp rootdir
define('FTP_TEXT_PATH','text_content'); //Text content directory (default)
define('FTP_USER','<ftp username>');  //FTP username account
define('FTP_PASS','<ftp password>');  //FTP password account


6. Update PhpDig
==============================


6.1. Database update
-------------------------


The [PHPDIG_DIR]/sql/update_db_to[version].sql contains all required
SQL instructions to update your existing install of PhpDig.
Vous pouvez également utiliser l'interface d'installation en choississant
l'option de mise à jour de la base. Cette fonction n'opère que pour la
version immédiatement précédente de la base de données.

6.2. Scripts update
-------------------------


Save your configurations files, and just replace the existing scripts by the
new ones.


7. Indexing with web interface
==============================


7.1. Index a new host
-------------------------


Open the admin interface with your browser : [PHPDIG_DIR]/admin/index.php.
Just fill in the url field, PhpDig reconizes if it is a new host or an existing one.
You can also precise a path and/or a file, wich is the starting point of the robot.


Select the maximum search depth in levels and click on the "Dig This !" button.


A new page opens showing the indexing and spidering process.
If a double is displayed, it means that PhpDig has detected that the current document, with
a new url, is a duplicate of an existing one in the database.
Each "+" sign means that a new link was detected and will be followed at the next
spidering level.
For each level, PhpDig displays the number of new links it has found.
If no new link is found, PhpDig stops its browsing and displays the list of the documents.

7.2. Update an existing host
-------------------------


From the admin page, you can reach the update interface
by choosing a site and clicking on the [update form] button.
A two parts inteface appears.
On the left side of the screen is the client-side folder structure of the site.
The blue arrow displays the "folder" content, in order to reindex the documents individually.
The document's listing of a folder is on the right side of the screen.


On both sides, the red cross deletes all the selected branch or file,
including sub-folders in case of deleting a branch, from the engine.
The green check mark reindexes the selected branch or document if they were indexed
for more than [LIMIT_DAYS] days. It also search new links for documents wich are
changed.

7.3. Index maintenance
-------------------------


3 scripts are used to delete useless data in the PhpDig database.
The links are in the admin page.


Clean index deletes index records not linked to any page.
Useful if manual deletes are done in the database.


Clean dictionary deletes keywords which are not used by the index.
Useful for reducing the size of the dictionary,
particularly when a large site contains a great deal of technical words and is deleted from the
engine.


Clean common words must be run when new common words are added in the
[PHPDIG_DIR]/includes/common_words.txt file. It deletes all reference to those common words.


8. Indexing by command line interface
==============================



Le script [PHPDIG_DIR]/admin/spider.php could be lauched by the shell
in order to not overload the webserver.


Launching the script :


#php -f [PHPDIG_DIR]/admin/spider.php [option]


List of options :
- all (default) : Update all hosts ;
- forceall : Force update all hosts ;
- http://mondomaine.tld : Add or update the url ;
- path/file : Add or update all urls listed in the given file.


Examples :


#php -f [PHPDIG_DIR]/admin/spider.php http://host.mydomain.com
#php -f [PHPDIG_DIR]/admin/spider.php [File containing an urls list]


As any shell command, the output can be redirected to a textfile.
(If you want some logs.)


#php -f [PHPDIG_DIR]/admin/spider.php all >> /var/log/phpdig.log


The [PHPDIG_DIR]/admin/spider.php can be launch by a cron task too, in order to auto update
the index. The recommended periodicity is 7 days. The updated documents you want to see
immediately in the searches can be updated manually.
Those pages can contain a "revisit-after" metatag with a short delay.


9. Templates
==============================


9.1. Description of templates
-------------------------


Templates are HTML files containing some xml-like tags wich are replaced with
the dynamic PhpDig content.

See the provided templates source code as making templates example.


The <phpdig:results></phpdig:results> tags display the
results table : All content between the tags will be repeated as much
time there are results in the results page.


Two CSS classes are used by PhpDig :

.phpdigHighlight : <SPAN/> class for highlighting of search terms.

a.phpdig : <A/> class for phpdig results and navigation links.


All template tags look like : <phpdig:parametre/>.

Excepted the
<phpdig:results></phpdig:results> tag, all are stand-alone tags.

9.2. Tags outside the results table
-------------------------


phpdig:title_message   Page title

phpdig:form_head       Starting the search form
phpdig:form_title      Form title
phpdig:form_field      Text field of the form
phpdig:form_button     Submit button of the form
phpdig:form_select     Select list to choose the num of results per page
phpdig:form_radio      Radio button to choose the parsing of search keys
phpdig:form_foot       Ending the search form

phpdig:result_message         Num of results message
phpdig:ignore_message         Too short words message
phpdig:ignore_commess         Too common words message

phpdig:nav_bar         Navigation bar to browse results
phpdig:pages_bar       Navigation bar without previous/next links
phpdig:previous_link src='[img src]'   "Previous" icon
phpdig:next_link src='[img src]'       "Next" icon

9.3. Results table tags
-------------------------


phpdig:results       Contains results list

phpdig:img_tag       Relevance Baragraph
phpdig:weight        Relevance of the page (in percents)
phpdig:page_link     Result title and link to the document
phpdig:limit_links   Links of limitation to an host / path
phpdig:text          Highlighted text extract or summary
phpdig:n             Result ranking, starting 1.
phpdig:complete_path Complete URL of the document
phpdig:update_date   Last update of the document
phpdig:filesize      Size of the document (KiloBytes)



10. Insert PhpDig in a website
==============================


10.1. The index.php script
-------------------------


The index.php script is only an example of using PhpDig with the
same name template. This script can be inserted in any part of your website,
assuming the configuration files and libraries are included.


The $relative_script_path must contain the relative path of PhpDig's root
directory from the current script.
The phpdigSearch() must be called always as this :


extract(phpdigHttpVars(
     array('query_string'=>'string',
           'template_demo'=>'string',
           'refine'=>'integer',
           'refine_url'=>'string',
           'site'=>'integer',
           'limite'=>'integer',
           'option'=>'string',
           'search'=>'string',
           'lim_start'=>'integer',
           'browse'=>'integer',
           'path'=>'string'
           )
     ));
phpdigSearch($id_connect, $query_string, $option, $refine,
              $refine_url, $lim_start, $limite, $browse,
              $site, $path, $relative_script_path, $template);


The last parameter, $template, sets the way how PhpDig works :
Use 'classic' for the static look of PhpDig. All html tags are
included.
Use the $template variable to use a template.
The variable could be set in the config.php file or anywhere else, in
order to have a different look in distincts part of your website.
Use 'array' to do what you want with the search form and results.

10.2. Using 'classic' mode
-------------------------


There is nothing to do : All display is done by the phpdigSearch()
function. But you can't modify display with this.

10.3. Using templates
-------------------------


With templates described earlier, it is very easy to insert PhpDig in an
existing website look.
A template could be an entire HTML page as sample templates provided,
but only a part of it too. Only this part is described by the template and
will appear where you call the phpdigSearch() function.
You just have to add a .phpdigHighlight CSS class to you existing CSS.

10.4. Using PHP
-------------------------


using the 'array' mode, The phpdigSearch() function returns an
array containing both results and search form elements.
Use this to get all content of this array, giving the script a full
search results URL :


print '<pre>';
print_r(
phpdigSearch($id_connect, $query_string, $option, $refine,
              $refine_url, $lim_start, $limite, $browse,
              $site, $path, $relative_script_path, 'array')
);
print '</pre>';


And do what you want with the results (first in big, the following three
in medium size, and only the title of others at the right side, all is possible !).


11. Inside PhpDig
==============================


11.1. Spidering and Indexing
-------------------------


PhpDig reads the fist page entered for indexing and adds found links in a
list of links to follow.
When no more valid link is found by the robot, it stops the process.


Do decide what to do with a new link, PhpDig follows this procedure :


- It requests the HTTP header for the current URI. If the returned Mime-type
could be parsed by the robot, it continues its process.
If the server returns a redirection, PhpDig search if the redirection go
to another host or not.
Then the robot compares "last-modified" header with the previous stored
date. If they are the same, the URI is not processed.
At least, the robot compare the URI with the exclude list.


- If the document type is HTML, PhpDig reads the META Robots content to know
if it is allowed to index and/or follow links from the current document.


- Then PhpDig downloads the document in a temporary file.
In first the document is indexed : The text content is stored in a file in order
to display snippets on results page, then parsed in order to get the keywords.
For an HTML document, the exclude and include comments are searched
(PHPDIG_EXCLUDE_COMMENT and PHPDIG_INCLUDE_COMMENT constants) to exclude parts
from indexing.


- At least, PhpDig reads again the temporary file (in case of HTML document)
in order to extract new links. All links are tested and parsed to decide
those to index, those giving a 404 error, linking to another host and so on.


- Indexing process is exclusive : An host is locked by the spider during
indexing, update or delete. No operation on this host is permitted (excepting
search of course) as long as the host is locked.

11.2. Clearings on search
-------------------------


The search form is so simple that it not needs lot of explanation.
But it could be useful to know that :


- An AND operator is applied between each search key ;

- Putting a '-' sign before a word excludes it from the search results.
No document containing this word would be displayed ;

- Search is case-insensitive and accent-insensitive.


12. Getting help with PhpDig
==============================



A small messageboard dedicated to PhpDig can be found at :
http://phpdig.toiletoine.net/messageboard/

Ask there any questions you have about this script.


You can also mail the author at
phpdig@toiletoine.net



File created by XSLT parser Php 4.3.0 - Sablotron 0.97

