Searching files with Xapian and Knowledgebase plugins

Added by Joshua Villagomez about 11 years ago

Does anyone have both these plugins enabled? If you have searching hacked for Knowledgebase, are you able to do full-text searching in files attached to Knowledgebase articles? - josh

Replies (10)

RE: Searching files with Xapian and Knowledgebase plugins - Added by T Leung about 11 years ago

There are a few file i've looked into to see if we can enable this:

/vendor/plugins/redmine_xapian/lib/acts_as_searchable.rb

contains some blocks of code that searches through the different attachments. Perhaps this holds the key to doing searching on attached knowledgebase articles?


        #Attahcment on documents
        results_doc= []
        results_count_doc=0
        find_options_tmp=Hash.new
        find_options_tmp=find_options_tmp.merge(find_options)
        find_options_tmp[:conditions] = merge_conditions (find_options_tmp[:conditions], :container_type=>"Document" )
        find_options_tmp [:joins] =  "INNER JOIN documents ON documents.id=container_id " +  "LEFT JOIN #{Project.table_name} ON #{Document.table_name}.project_id = #{Project.table_name}.id" 
        with_scope(:find => {:conditions => project_conditions.join(' AND ')}) do
              with_scope(:find => find_options_tmp) do
                    results_count_doc = count(:all)
                    results_doc = find(:all, limit_options)
                  end
        end
        results +=results_doc
                results_count += results_count_doc

        #Attachemnts on WikiPage
        find_options_tmp=Hash.new
        find_options_tmp=find_options_tmp.merge(find_options)
               results_wiki= []
                results_count_wiki=0
        find_options_tmp[:conditions] =  merge_conditions (find_options_tmp[:conditions], :container_type=>"WikiPage" )
        find_options_tmp [:joins] =  "INNER JOIN wiki_pages ON wiki_pages.id=container_id " +  
                    "INNER JOIN wikis ON wikis.id=wiki_pages.wiki_id " +
                    "INNER JOIN #{Project.table_name} ON #{Wiki.table_name}.project_id = #{Project.table_name}.id" 
        with_scope(:find => {:conditions => project_conditions.join(' AND ')}) do
              with_scope(:find => find_options_tmp) do
                   results_count_wiki = count(:all)
                   results_wiki = find(:all, limit_options)
                  end
        end

RE: Searching files with Xapian and Knowledgebase plugins - Added by Luis Serrano Aranda about 11 years ago

Ithnk yes but knowledgebase don't have project number

RE: Searching files with Xapian and Knowledgebase plugins - Added by Luis Serrano Aranda about 11 years ago

I try this:

#Attachments on Knowledgebase
find_options_tmp =Hash.new
find_options_tmp=find_options_tmp.merge(find_options)
results_knowledgebase= []
results_count_knowledgebase=0
find_options_tmp[:conditions] = merge_conditions (find_options[:conditions], :container_type=>"Article" )
find_options_tmp [:joins] = "INNER JOIN kb_articles ON kb_articles.id=container_id "
with_scope(:find => {:conditions => project_conditions.join(' AND ')}) do
with_scope(:find => find_options_tmp) do
results_count_knowledgebase = count(:all)
results_knowledgebase = find(:all, limit_options)
end
end
results+=results_knowledgebase
results_count+= results_count_knowledgebase

But don't work

RE: Searching files with Xapian and Knowledgebase plugins - Added by Luis Serrano Aranda about 11 years ago

Luis Serrano Aranda wrote:

I think yes but knowledgebase don't have project number

RE: Searching files with Xapian and Knowledgebase plugins - Added by T Leung about 11 years ago

Finally got it to search the actual file contents. I did a hack - but it works.

I'm sure there's a better way to do this, but for now this is what i'm doing:

In /redmine_xapian/lib/xapian_search.rb

if allowed and project_included(docattach.container.project.id, projects_to_search)
    docattach[:description ...

the allowed was always false, which was evaluated with

allowed = User.current.allowed_to?("view_documents".to_sym, docattach.container.project)

I added this

if docattach.container.type#class == "Article" 
    allowed = true
end

That allows all knowledge base articles to pass.

In acts_as_searchable.rb in the xapian area, I extended the block of code to include knowledgebase articles

find_options_tmp=Hash.new
        find_options_tmp=find_options_tmp.merge(find_options)
        results_article = []
        results_count_article=0
        find_options_tmp[:conditions] = merge_conditions (find_options_tmp[:conditions], :container_type=>"Article" )

        find_options_tmp [:joins] =  "INNER JOIN kb_articles ON kb_articles.id=container_id " +  "LEFT JOIN #{Project.table_name} ON #{Article.table_name}.project_id = #{Project.table_name}.id" 
        with_scope(:find => {:conditions => project_conditions.join(' AND ')}) do
              with_scope(:find => find_options_tmp) do
                   results_count_article = count(:all)
                    results_article = find(:all, limit_options)
              end
        end
        results +=results_article
        results_count += results_count_article

That portion joins the sql tables together with the attachments table

In the attachment_patch.rb, extended the container_url, container_name, and container_type methods to include Article

which is used in the view index.rhtml, in which was added

<%    elsif e.container_type == "Article" %>
<%=        link_to truncate( e.container_type+e.container_name, :length => 255), e.container_url %>
<%    end

there is supposed to be some labels we can use for the other ones they have label_document, label_message, label_wiki, label_issue, but don't know if there is a label_article ...

For now, I used the container_type, which is Article.

RE: Searching files with Xapian and Knowledgebase plugins - Added by Xabier Elkano about 11 years ago

T Leung wrote:

Finally got it to search the actual file contents. I did a hack - but it works.

I'm sure there's a better way to do this, but for now this is what i'm doing:

In /redmine_xapian/lib/xapian_search.rb

[...]

the allowed was always false, which was evaluated with

[...]

I added this

[...]

That allows all knowledge base articles to pass.

In acts_as_searchable.rb in the xapian area, I extended the block of code to include knowledgebase articles
[...]

That portion joins the sql tables together with the attachments table

In the attachment_patch.rb, extended the container_url, container_name, and container_type methods to include Article

which is used in the view index.rhtml, in which was added

[...]

there is supposed to be some labels we can use for the other ones they have label_document, label_message, label_wiki, label_issue, but don't know if there is a label_article ...

For now, I used the container_type, which is Article.

I've added support for article container type on 1.1.3 redmine_xapian version ( github ).
Now, it's possible to do searches by name and by content on articles containers too.
Thanks you for your help.

Knowledgebase plugin is a great plugin :-)

RE: Searching files with Xapian and Knowledgebase plugins - Added by Joshua Villagomez about 11 years ago

Thanks everyone, and Xelkano for including it in the plugin. We tried the code changes applied here and were able to extend searches into kb articles and attachments! With these two plugins, Redmine has become a powerful document management system for us.

RE: Searching files with Xapian and Knowledgebase plugins - Added by T Leung about 11 years ago

Though the kb_articles table has a project_id field, perhaps something in one of the models is not linking it together with a project?

In our articles_controller.rb, we hard-coded the project_id in the create function to the project id number we wanted.

I found that if I selected a project, knowledgebase articles' contents will not be searchable. After some troubleshooting, I found that it was being filtered - not on purpose, but because of something that is missing with linking the articles to a project.

Therefore, I had to add this code to the xapian_search.rb, so that the right project_id was being put into project_included function

project_id = docattach.container_type == "Article" ? docattach.container.project_id : docattach.container.project.id

if allowed and project_included(project_id, projects_to_search)
...

Any ideas of a better solution? It will really make the xapian search more robust. Thanks!

(1-10/10)