Algorithm-VSM
view release on metacpan or search on metacpan
examples/calculate_precision_and_recall_from_file_based_relevancies_for_LSA.pl view on Meta::CPAN
# Uncomment the following statement if you want to see the corpus
# vocabulary:
#$lsa->display_corpus_vocab();
# Uncomment the following statement if you want to see the individual
# document vectors:
#$lsa->display_doc_vectors();
$lsa->construct_lsa_model();
$lsa->upload_document_relevancies_from_file(); # The format of the relevancy
# file must be as shown in
# relevance.txt
# Uncomment the following statement if you wish to see the list of all
# the documents relevant to each of the queries:
#$lsa->display_doc_relevancies();
$lsa->precision_and_recall_calculator('lsa');
$lsa->display_precision_vs_recall_for_queries();
examples/calculate_precision_and_recall_from_file_based_relevancies_for_VSM.pl view on Meta::CPAN
# Uncomment the following statement if you want to see the corpus
# vocabulary:
#$vsm->display_corpus_vocab();
# Uncomment the following statement if you want to see the individual
# document vectors:
#$vsm->display_doc_vectors();
#$vsm->construct_lsa_model();
$vsm->upload_document_relevancies_from_file(); # The format of the relevancy
# file must be as shown in
# relevance.txt
# Uncomment the following statement if you wish to see the list of all
# the documents relevant to each of the queries:
#$vsm->display_doc_relevancies();
# Use only one of the following statements. If you wish to carry out
# precision vs. recall analysis for LSA, comment out the first and
# uncomment the second.
examples/retrieve_with_disk_based_LSA.pl view on Meta::CPAN
my $doc_vectors_db = "doc_vectors_db";
my $normalized_doc_vecs_db = "normalized_doc_vecs_db";
my $lsa = Algorithm::VSM->new(
corpus_vocab_db => $corpus_vocab_db,
doc_vectors_db => $doc_vectors_db,
normalized_doc_vecs_db => $normalized_doc_vecs_db,
max_number_retrievals => 10,
);
$lsa->upload_normalized_vsm_model_from_disk();
# Uncomment the following if you would like to see the corpus vocabulary:
#$lsa->display_corpus_vocab();
# Uncomment the following if you would like to see the doc vectors for
# each of the documents in the corpus:
#$lsa->display_doc_vectors();
$lsa->construct_lsa_model();
examples/retrieve_with_disk_based_VSM.pl view on Meta::CPAN
my $vsm = Algorithm::VSM->new(
corpus_vocab_db => $corpus_vocab_db,
doc_vectors_db => $doc_vectors_db,
normalized_doc_vecs_db => $normalized_doc_vecs_db,
max_number_retrievals => 10,
);
# Use the following call ONLY if you are setting the use_idf_filter option to
# 0 in the above constructor.
#$vsm->upload_vsm_model_from_disk();
$vsm->upload_normalized_vsm_model_from_disk();
# Uncomment the following statement if you would like to see the corpus
# vocabulary:
#$vsm->display_corpus_vocab();
# Use the following call ONLY if you are setting the use_idf_filter option to
# 0 in the above constructor.
#$vsm->display_doc_vectors();
# Uncomment the following statement if you would like to the individual
examples/significance_testing.pl view on Meta::CPAN
min_word_length => 4,
query_file => $query_file,
relevancy_file => $relevancy_file,
stop_words_file => $stop_words_file,
want_stemming => 1, # default: 0
);
$lsa1->get_corpus_vocabulary_and_word_counts();
$lsa1->generate_document_vectors();
$lsa1->construct_lsa_model();
$lsa1->upload_document_relevancies_from_file();
$lsa1->precision_and_recall_calculator('lsa');
my $avg_precisions_1 = $lsa1->get_query_sorted_average_precision_for_queries();
my $MAP_Algo_1 = 0;
map {$MAP_Algo_1 += $_} @$avg_precisions_1;
$MAP_Algo_1 /= @$avg_precisions_1;
print "MAP value for LSA-1: $MAP_Algo_1\n";
print "Avg precisions for LSA-1: @$avg_precisions_1\n"
if $debug_signi;
examples/significance_testing.pl view on Meta::CPAN
min_word_length => 4,
query_file => $query_file,
relevancy_file => $relevancy_file,
stop_words_file => $stop_words_file,
want_stemming => 1, # default: 0
);
$lsa2->get_corpus_vocabulary_and_word_counts();
$lsa2->generate_document_vectors();
$lsa2->construct_lsa_model();
$lsa2->upload_document_relevancies_from_file();
$lsa2->precision_and_recall_calculator('lsa');
my $avg_precisions_2 = $lsa2->get_query_sorted_average_precision_for_queries();
my $MAP_Algo_2 = 0;
map {$MAP_Algo_2 += $_} @$avg_precisions_2;
$MAP_Algo_2 /= @$avg_precisions_2;
print "MAP value for LSA-2: $MAP_Algo_2\n";
print "Average precisions for LSA-2: @$avg_precisions_2\n"
if $debug_signi;
# This is the observed value for the test statistic that will be subject to
lib/Algorithm/VSM.pm view on Meta::CPAN
print "\n\nShowing the VSM retrievals and the similarity scores:\n\n";
foreach (sort {$retrievals{$b} <=> $retrievals{$a}} keys %retrievals) {
print "$_ => $retrievals{$_}\n";
}
}
return \%retrievals;
}
######################### Upload a Previously Constructed Model #########################
sub upload_vsm_model_from_disk {
my $self = shift;
die "\nCannot find the database files for the VSM model"
unless -s "$self->{_corpus_vocab_db}.pag"
&& -s $self->{_doc_vectors_db};
$self->{_corpus_doc_vectors} = retrieve($self->{_doc_vectors_db});
tie %{$self->{_vocab_hist_on_disk}}, 'SDBM_File',
$self->{_corpus_vocab_db}, O_RDONLY, 0640
or die "Can't open DBM file: $!";
if ($self->{_debug}) {
foreach ( sort keys %{$self->{_vocab_hist_on_disk}} ) {
lib/Algorithm/VSM.pm view on Meta::CPAN
$self->{_vocab_hist}->{$_} = $self->{_vocab_hist_on_disk}->{$_};
}
$self->{_corpus_vocab_done} = 1;
$self->{_vocab_size} = scalar( keys %{$self->{_vocab_hist}} );
print "\n\nVocabulary size: $self->{_vocab_size}\n\n"
if $self->{_debug};
$self->{_corpus_doc_vectors} = retrieve($self->{_doc_vectors_db});
untie %{$self->{_vocab_hist_on_disk}};
}
sub upload_normalized_vsm_model_from_disk {
my $self = shift;
die "\nCannot find the database files for the VSM model"
unless -s "$self->{_corpus_vocab_db}.pag"
&& -s $self->{_normalized_doc_vecs_db};
$self->{_normalized_doc_vecs} = retrieve($self->{_normalized_doc_vecs_db});
tie %{$self->{_vocab_hist_on_disk}}, 'SDBM_File',
$self->{_corpus_vocab_db}, O_RDONLY, 0640
or die "Can't open DBM file: $!";
if ($self->{_debug}) {
foreach ( sort keys %{$self->{_vocab_hist_on_disk}} ) {
lib/Algorithm/VSM.pm view on Meta::CPAN
keys %{$self->{_relevancy_estimates}}) {
@relevancy_list_for_query =
keys %{$self->{_relevancy_estimates}->{$_}};
print OUT "$_ => @relevancy_list_for_query\n\n";
print "Number of relevant docs for query $_: " .
scalar(@relevancy_list_for_query) . "\n";
}
}
# If there are available human-supplied relevancy judgments in a disk
# file, use this script to upload that information. One of the scripts
# in the 'examples' directory carries out the precision-recall analysis
# by using this approach. IMPORTANT: The human-supplied relevancy
# judgments must be in a format that is shown in the sample file
# relevancy.txt in the 'examples' directory.
sub upload_document_relevancies_from_file {
my $self = shift;
chdir $self->{_working_directory};
open( IN, $self->{_relevancy_file} )
or die "unable to open the relevancy file $self->{_relevancy_file}: $!";
while (<IN>) {
next if /^#/;
next if /^[ ]*\r?\n?$/;
$_ =~ s/\r?\n?$//;
die "Format of query file is not correct" unless /^[ ]*q[0-9]+[ ]*=>/;
/^[ ]*(q[0-9]+)[ ]*=>[ ]*(.*)/;
lib/Algorithm/VSM.pm view on Meta::CPAN
corpus_directory => $corpus_dir,
file_types => ['.txt', '.java'],
min_word_length => 4,
query_file => $query_file,
relevancy_file => $relevancy_file,
stop_words_file => $stop_words_file,
want_stemming => 1,
);
$vsm->get_corpus_vocabulary_and_word_counts();
$vsm->generate_document_vectors();
$vsm->upload_document_relevancies_from_file();
$vsm->display_doc_relevancies();
$vsm->precision_and_recall_calculator('vsm');
$vsm->display_precision_vs_recall_for_queries();
$vsm->display_average_precision_for_queries_and_map();
Now the filename supplied through the constructor parameter 'relevancy_file' must
contain relevance judgments for the queries that are named in the file supplied
through the parameter 'query_file'. The format of these two files must be
according to what is shown in the sample files 'test_queries.txt' and
'relevancy.txt' in the 'examples' directory.
lib/Algorithm/VSM.pm view on Meta::CPAN
file_types => ['.txt', '.java'],
lsa_svd_threshold => 0.01,
min_word_length => 4,
query_file => $query_file,
relevancy_file => $relevancy_file,
stop_words_file => $stop_words_file,
want_stemming => 1,
);
$lsa->get_corpus_vocabulary_and_word_counts();
$lsa->generate_document_vectors();
$lsa->upload_document_relevancies_from_file();
$lsa->display_doc_relevancies();
$lsa->precision_and_recall_calculator('vsm');
$lsa->display_precision_vs_recall_for_queries();
$lsa->display_average_precision_for_queries_and_map();
As mentioned for the previous code block, the filename supplied through the
constructor parameter 'relevancy_file' must contain relevance judgments for the
queries that are named in the file supplied through the parameter 'query_file'.
The format of this file must be according to what is shown in the sample file
'relevancy.txt' in the 'examples' directory. We have already explained the roles
lib/Algorithm/VSM.pm view on Meta::CPAN
my $retrievals = $vsm->retrieve_with_vsm( \@query );
The argument, C<@query>, is simply a list of words that you wish to use for
retrieval. The method returns a hash whose keys are the document names and whose
values the similarity distance between the document and the query. As is commonly
the case with VSM, this module uses the cosine similarity distance when comparing a
document vector with the query vector.
=item B<upload_document_relevancies_from_file():>
When human-supplied relevancies are available, you can upload them into the program
by calling
$vsm->upload_document_relevancies_from_file();
These relevance judgments will be read from a file that is named with the
C<relevancy_file> constructor parameter.
=item B<upload_normalized_vsm_model_from_disk():>
When you invoke the methods C<get_corpus_vocabulary_and_word_counts()> and
C<generate_document_vectors()>, that automatically deposits the VSM model in the
database files named with the constructor parameters C<corpus_vocab_db>,
C<doc_vectors_db> and C<normalized_doc_vecs_db>. Subsequently, you can carry out
retrieval by directly using this disk-based VSM model for speedier performance. In
order to do so, you must upload the disk-based model by
$vsm->upload_normalized_vsm_model_from_disk();
Subsequently you call
my $retrievals = $vsm->retrieve_with_vsm( \@query );
$vsm->display_retrievals( $retrievals );
for retrieval and for displaying the results.
=item B<write_corpus_vocab_to_file():>
( run in 3.104 seconds using v1.01-cache-2.11-cpan-b16cb0d3907 )