2 # Script to perform searching
3 # For documentation try 'perldoc /path/to/search'
7 # Copyright 2006 LibLime
9 # This file is part of Koha
11 # Koha is free software; you can redistribute it and/or modify it under the
12 # terms of the GNU General Public License as published by the Free Software
13 # Foundation; either version 2 of the License, or (at your option) any later
16 # Koha is distributed in the hope that it will be useful, but WITHOUT ANY
17 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
18 # A PARTICULAR PURPOSE. See the GNU General Public License for more details.
20 # You should have received a copy of the GNU General Public License along with
21 # Koha; if not, write to the Free Software Foundation, Inc., 59 Temple Place,
22 # Suite 330, Boston, MA 02111-1307 USA
26 search - a search script for finding records in a Koha system (Version 2.4)
30 This script contains a demonstration of a new search API for Koha 2.4. It is
31 designed to be simple to use and configure, yet capable of performing feats
32 like stemming, field weighting, relevance ranking, support for multiple
33 query language formats (CCL, CQL, PQF), full or nearly full support for the
34 bib1 attribute set, extended attribute sets defined in Zebra profiles, access
35 to the full range of Z39.50 query options, federated searches on Z39.50
38 I believe the API as represented in this script is mostly sound, even if the
39 individual functions in Search.pm and Koha.pm need to be cleaned up. Of course,
40 you are free to disagree :-)
42 I will attempt to describe what is happening at each part of this script.
47 This script performs two functions:
51 =item 1. interacts with Koha to retrieve and display the results of a search
53 =item 2. loads the advanced search page
57 These two functions share many of the same variables and modules, so the first
58 task is to load what they have in common and determine which template to use.
59 Once determined, proceed to only load the variables and procedures necessary
62 =head2 THE ADVANCED SEARCH PAGE
64 If we're loading the advanced search page this script will call a number of
65 display* routines which populate objects that are sent to the template for
66 display of things like search indexes, languages, search limits, branches,
67 etc. These are not stored in the template for two reasons:
71 =item 1. Efficiency - we have more control over objects inside the script, and it's possible to not duplicate things like indexes (if the search indexes were stored in the template they would need to be repeated)
73 =item 2. Customization - if these elements were moved to the sql database it would allow a simple librarian to determine which fields to display on the page without editing any html (also how the fields should behave when being searched).
77 However, they create one problem : the strings aren't translated. I have an idea
78 for how to do this that I will purusue soon.
80 =head2 PERFORMING A SEARCH
82 If we're performing a search, this script performs three primary
87 =item 1. builds query strings (yes, plural)
89 =item 2. perform the search and return the results array
91 =item 3. build the HTML for output to the template
95 There are several additional secondary functions performed that I will
98 =head3 1. Building Query Strings
100 There are several types of queries needed in the process of search and retrieve:
104 =item 1 Koha query - the query that is passed to Zebra
106 This is the most complex query that needs to be built.The original design goal was to use a custom CCL2PQF query parser to translate an incoming CCL query into a multi-leaf query to pass to Zebra. It needs to be multi-leaf to allow field weighting, koha-specific relevance ranking, and stemming. When I have a chance I'll try to flesh out this section to better explain.
108 This query incorporates query profiles that aren't compatible with non-Zebra Z39.50 targets to acomplish the field weighting and relevance ranking.
110 =item 2 Federated query - the query that is passed to other Z39.50 targets
112 This query is just the user's query expressed in CCL CQL, or PQF for passing to a non-zebra Z39.50 target (one that doesn't support the extended profile that Zebra does).
114 =item 3 Search description - passed to the template / saved for future refinements of the query (by user)
116 This is a simple string that completely expresses the query in a way that can be parsed by Koha for future refinements of the query or as a part of a history feature. It differs from the human search description in several ways:
118 1. it does not contain commas or = signs
121 =item 4 Human search description - what the user sees in the search_desc area
123 This is a simple string nearly identical to the Search description, but more human readable. It will contain = signs or commas, etc.
127 =head3 2. Perform the Search
129 This section takes the query strings and performs searches on the named servers, including the Koha Zebra server, stores the results in a deeply nested object, builds 'faceted results', and returns these objects.
133 The final major section of this script takes the objects collected thusfar and builds the HTML for output to the template and user.
135 =head3 Additional Notes
141 use strict; # always use
143 ## STEP 1. Load things that are used in both search page and
144 # results page and decide which template to load, operations
151 use C4::Languages; # getAllLanguages
153 use POSIX qw(ceil floor);
154 use C4::Branch; # GetBranches
155 # create a new CGI object
156 # not sure undef_params option is working, need to test
157 use CGI qw('-no_undef_params');
160 my ($template,$borrowernumber,$cookie);
162 # decide which template to use
164 my @params = $cgi->param("limit");
165 if ((@params>=1) || ($cgi->param("q")) || ($cgi->param('multibranchlimit')) ) {
166 $template_name = 'catalogue/results.tmpl';
169 $template_name = 'catalogue/advsearch.tmpl';
172 ($template, $borrowernumber, $cookie) = get_template_and_user({
173 template_name => $template_name,
176 authnotrequired => 0,
177 flagsrequired => { catalogue => 1 },
181 =head1 BUGS and FIXMEs
183 There are many, most are documented in the code. The one that
184 isn't fully documented, but referred to is the need for a full
190 # FIXME: URI re-writing should be tested more carefully and may better
191 # handled by mod_rewrite or something else. The code below almost works,
192 # but doesn't quite handle limits correctly when they are the only
193 # param passed -- I'll work on this soon -- JF
195 #my $uri = $cgi->url(-base => 1);
196 #my $relative_url = $cgi->url(-relative=>1);
197 #$uri.="/".$relative_url."?";
199 #my @cgi_params_list = $cgi->param();
200 #my $url_params = $cgi->Vars;
202 #for my $each_param_set (@cgi_params_list) {
203 # $uri.= join "", map "\&$each_param_set=".$_, split("\0",$url_params->{$each_param_set}) if $url_params->{$each_param_set};
205 #warn "New URI:$uri";
206 # Only re-write a URI if there are params or if it already hasn't been re-written
207 #unless (($cgi->param('r')) || (!$cgi->param()) ) {
208 # print $cgi->redirect( -uri=>$uri."&r=1",
209 # -cookie => $cookie);
214 my $branches = GetBranches();
216 #push @branch_loop, {value => "", branchname => "All Branches", };
217 for my $branch_hash (sort keys %$branches) {
218 push @branch_loop, {value => "branch:$branch_hash" , branchname => $branches->{$branch_hash}->{'branchname'}, };
221 my $categories = GetBranchCategories(undef,'searchdomain');
223 $template->param(branchloop => \@branch_loop, searchdomainloop => $categories);
225 # load the itemtypes (Called Collection Codes in the template -- used for circ rules )
226 my $itemtypes = GetItemTypes;
230 my $imgdir = getitemtypeimagesrc();
231 foreach my $thisitemtype ( sort {$itemtypes->{$a}->{'description'} cmp $itemtypes->{$b}->{'description'} } keys %$itemtypes ) {
232 my %row =( number=>$cnt++,
233 imageurl=> $itemtypes->{$thisitemtype}->{'imageurl'}?($imgdir."/".$itemtypes->{$thisitemtype}->{'imageurl'}):"",
234 code => $thisitemtype,
235 selected => $selected,
236 description => $itemtypes->{$thisitemtype}->{'description'},
239 $selected = 0 if ($selected) ;
240 push @itemtypesloop, \%row;
242 $template->param(itemtypeloop => \@itemtypesloop);
244 # # load the itypes (Called item types in the template -- just authorized values for searching)
245 # my ($itypecount,@itype_loop) = GetCcodes();
246 # $template->param(itypeloop=>\@itype_loop,);
248 # load the languages ( for switching from one template to another )
249 # my @languages_options = displayLanguages($cgi);
250 # my $languages_count = @languages_options;
251 # if($languages_count > 1){
252 # $template->param(languages => \@languages_options);
255 # The following should only be loaded if we're bringing up the advanced search template
256 if ( $template_name eq "catalogue/advsearch.tmpl" ) {
257 # load the servers (used for searching -- to do federated searching, etc.)
258 my $primary_servers_loop;# = displayPrimaryServers();
259 $template->param(outer_servers_loop => $primary_servers_loop,);
261 my $secondary_servers_loop;# = displaySecondaryServers();
262 $template->param(outer_sup_servers_loop => $secondary_servers_loop,);
264 # determine what to display next to the search boxes (ie, boolean option
265 # shouldn't appear on the first one, scan indexes should, adding a new
266 # box should only appear on the last, etc.
267 # FIXME: this stuff should be cleaned up a bit and the html should be turned
268 # into flags for the template -- I'll work on that soon -- JF
269 my @search_boxes_array;
270 my $search_boxes_count = 1; # should be a syspref
271 for (my $i=0;$i<=$search_boxes_count;$i++) {
273 push @search_boxes_array,
275 search_boxes_label => 1,
280 elsif ($i==$search_boxes_count) {
281 push @search_boxes_array,
287 $template->param(uc(C4::Context->preference("marcflavour")) => 1,
288 search_boxes_loop => \@search_boxes_array);
290 # load the language limits (for search)
291 my $languages_limit_loop = getAllLanguages();
292 $template->param(search_languages_loop => $languages_limit_loop,);
294 my $expanded_options;
295 if (not defined $cgi->param('expanded_options')){
296 $expanded_options = C4::Context->preference("expandedSearchOption");
299 $expanded_options = $cgi->param('expanded_options');
301 $template->param(expanded_options => $expanded_options);
303 # load the sort_by options for the template
304 my $sort_by = $cgi->param('sort_by');
306 output_html_with_http_headers $cgi, $cookie, $template->output;
310 ### OK, if we're this far, we're performing an actual search
312 # Fetch the paramater list as a hash in scalar context:
313 # * returns paramater list as tied hash ref
314 # * we can edit the values by changing the key
315 # * multivalued CGI paramaters are returned as a packaged string separated by "\0" (null)
316 my $params = $cgi->Vars;
318 # Params that can have more than one value
319 # sort by is used to sort the query
321 @sort_by = split("\0",$params->{'sort_by'}) if $params->{'sort_by'};
323 # Use the servers defined, or just search our local catalog(default)
325 @servers = split("\0",$params->{'server'}) if $params->{'server'};
327 #FIXME: this should be handled using Context.pm
328 @servers = ("biblioserver");
329 # @servers = C4::Context->config("biblioserver");
332 # operators include boolean and proximity operators and are used
333 # to evaluate multiple operands
335 @operators = split("\0",$params->{'op'}) if $params->{'op'};
337 # indexes are query qualifiers, like 'title', 'author', etc. They
338 # can be simple or complex
340 @indexes = split("\0",$params->{'idx'}) if $params->{'idx'};
342 # an operand can be a single term, a phrase, or a complete ccl query
344 @operands = split("\0",$params->{'q'}) if $params->{'q'};
346 # limits are use to limit to results to a pre-defined category such as branch or language
348 @limits = split("\0",$params->{'limit'}) if $params->{'limit'};
350 if($params->{'multibranchlimit'}) {
351 push @limits, join(" or ", map { "branch: $_ "} @{GetBranchesInCategory($params->{'multibranchlimit'})}) ;
355 foreach my $limit(@limits) {
356 if ($limit =~/available/) {
360 $template->param(available => $available);
361 push @limits, map "yr:".$_, split("\0",$params->{'limit-yr'}) if $params->{'limit-yr'};
363 # Params that can only have one value
364 my $query = $params->{'q'};
365 my $scan = $params->{'scan'};
366 my $results_per_page = $params->{'count'} || 20;
367 my $page = $cgi->param('page') || 1;
368 my $offset = ($page-1)*$results_per_page;
370 my $expanded_facet = $params->{'expand'};
372 # Define some global variables
373 my $error; # used for error handling
374 my $search_desc; # the query expressed in terms that humans understand
375 my $koha_query; # the query expressed in terms that zoom understands with field weighting and stemming
377 my $query_type; # usually not needed, but can be used to trigger ccl, cql, or pqf queries if set
379 ## I. BUILD THE QUERY
380 ($error,$search_desc,$koha_query,$federated_query,$query_type) = buildQuery($query,\@operators,\@operands,\@indexes,\@limits);
382 ## II. DO THE SEARCH AND GET THE RESULTS
383 my $total; # the total results for the whole set
384 my $facets; # this object stores the faceted results that display on the left-hand of the results page
388 if (C4::Context->preference('NoZebra')) {
390 ($error, $results_hashref, $facets) = NZgetRecords($koha_query,$federated_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
394 ($error, $results_hashref, $facets) = getRecords($koha_query,$federated_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
398 $template->param(query_error => $error.$@);
400 output_html_with_http_headers $cgi, $cookie, $template->output;
403 my $op=$cgi->param("operation");
404 if ($op eq "bulkedit"){
405 my ($countchanged,$listunchanged)=
406 ModBiblios($results_hashref->{'biblioserver'}->{"RECORDS"},
407 $params->{"tagsubfield"},
408 $params->{"inputvalue"},
409 $params->{"targetvalue"},
412 $template->param(bulkeditresults=>1,
413 tagsubfield=>$params->{"tagsubfield"},
414 inputvalue=>$params->{"inputvalue"},
415 targetvalue=>$params->{"targetvalue"},
416 countchanged=>$countchanged,
417 countunchanged=>scalar(@$listunchanged),
418 listunchanged=>$listunchanged);
421 if (C4::Context->userenv->{'flags'}==1 ||(C4::Context->userenv->{'flags'} & ( 2**9 ) )){
422 #Edit Catalogue Permissions
423 $template->param(bulkedit => 1);
424 $template->param(tagsubfields=>GetManagedTagSubfields());
426 # At this point, each server has given us a result set
427 # now we build that set for template display
428 my @sup_results_array;
429 for (my $i=0;$i<=@servers;$i++) {
430 my $server = $servers[$i];
431 if ($server =~/biblioserver/) { # this is the local bibliographic server
432 $hits = $results_hashref->{$server}->{"hits"};
433 my $page = $cgi->param('page') || 0;
434 my @newresults = searchResults( $search_desc,$hits,$results_per_page,$offset,@{$results_hashref->{$server}->{"RECORDS"}});
435 $total = $total + $results_hashref->{$server}->{"hits"};
437 $template->param(total => $hits);
438 $template->param(searchdesc => ($query_type?"$query_type=":"")."$search_desc" );
439 $template->param(results_per_page => $results_per_page);
440 $template->param(SEARCH_RESULTS => \@newresults);
443 # my $pages = ceil($hits / $results_per_page);
444 # my $previous_page_offset = $offset - $results_per_page unless ($offset - $results_per_page <0);
445 # my $next_page_offset = $offset + $results_per_page;
446 # for (my $j=1; $j<=$pages;$j++) {
447 # my $this_offset = (($j*$results_per_page)-$results_per_page);
448 # my $this_page_number = $j;
449 # my $highlight = 1 if ($this_page_number == $current_page_number);
450 # if ($this_page_number <= $pages) {
451 # push @page_numbers, { offset => $this_offset, pg => $this_page_number, highlight => $highlight, sort_by => join " ",@sort_by };
454 # $template->param(PAGE_NUMBERS => \@page_numbers,
455 # previous_page_offset => $previous_page_offset,
456 # next_page_offset => $next_page_offset) unless $pages < 2;
457 my $link="/cgi-bin/koha/catalogue/search.pl?q=$search_desc&";
458 foreach my $sort (@sort_by){
459 $link.="&sort_by=".$sort."&";
462 pagination_bar => pagination_bar(
464 getnbpages($hits, $results_per_page),
470 } # end of the if local
472 # check if it's a z3950 or opensearch source
473 my $zed3950 = 0; # FIXME :: Hardcoded value.
475 my @inner_sup_results_array;
476 for my $sup_record ( @{$results_hashref->{$server}->{"RECORDS"}} ) {
477 my $marc_record_object = MARC::Record->new_from_usmarc($sup_record);
478 my $control_number = $marc_record_object->field('010')->subfield('a') if $marc_record_object->field('010');
479 $control_number =~ s/^ //g;
480 my $link = "http://catalog.loc.gov/cgi-bin/Pwebrecon.cgi?SAB1=".$control_number."&BOOL1=all+of+these&FLD1=LC+Control+Number+LCCN+%28K010%29+%28K010%29&GRP1=AND+with+next+set&SAB2=&BOOL2=all+of+these&FLD2=Keyword+Anywhere+%28GKEY%29+%28GKEY%29&PID=6211&SEQ=20060816121838&CNT=25&HIST=1";
481 my $title = $marc_record_object->title();
482 push @inner_sup_results_array, {
487 my $servername = $server;
488 push @sup_results_array, { servername => $servername, inner_sup_results_loop => \@inner_sup_results_array};
489 $template->param(outer_sup_results_loop => \@sup_results_array);
493 } #/end of the for loop
494 #$template->param(FEDERATED_RESULTS => \@results_array);
498 #classlist => $classlist,
500 searchdesc => ($query_type?"$query_type=":"")."$search_desc",
502 facets_loop => $facets,
504 search_error => $error,
506 ## Now let's find out if we have any supplemental data to show the user
507 # and in the meantime, save the current query for statistical purposes, etc.
508 my $koha_spsuggest; # a flag to tell if we've got suggestions coming from Koha
509 my @koha_spsuggest; # place we store the suggestions to be returned to the template as LOOP
510 my $phrases = $search_desc;
513 if ( C4::Context->preference("kohaspsuggest") ) {
514 my ($suggest_host, $suggest_dbname, $suggest_user, $suggest_pwd) = split(':', C4::Context->preference("kohaspsuggest"));
516 my $koha_spsuggest_dbh;
517 # FIXME: this needs to be moved to Context.pm
519 $koha_spsuggest_dbh=DBI->connect("DBI:mysql:$suggest_dbname:$suggest_host","$suggest_user","$suggest_pwd");
522 warn "can't connect to spsuggest db";
525 my $koha_spsuggest_insert = "INSERT INTO phrase_log(phr_phrase,phr_resultcount,phr_ip) VALUES(?,?,?)";
526 my $koha_spsuggest_query = "SELECT display FROM distincts WHERE strcmp(soundex(suggestion), soundex(?)) = 0 order by soundex(suggestion) limit 0,5";
527 my $koha_spsuggest_sth = $koha_spsuggest_dbh->prepare($koha_spsuggest_query);
528 $koha_spsuggest_sth->execute($phrases);
529 while (my $spsuggestion = $koha_spsuggest_sth->fetchrow_array) {
530 $spsuggestion =~ s/(:|\/)//g;
532 $line{spsuggestion} = $spsuggestion;
533 push @koha_spsuggest,\%line;
537 # Now save the current query
538 $koha_spsuggest_sth=$koha_spsuggest_dbh->prepare($koha_spsuggest_insert);
539 #$koha_spsuggest_sth->execute($phrases,$results_per_page,$ipaddress);
540 $koha_spsuggest_sth->finish;
542 $template->param( koha_spsuggest => $koha_spsuggest ) unless $hits;
543 $template->param( SPELL_SUGGEST => \@koha_spsuggest,
548 warn "Kohaspsuggest failure:".$@;
552 # VI. BUILD THE TEMPLATE
553 output_html_with_http_headers $cgi, $cookie, $template->output;