Carrot2

Reusing Solr's lexical resources in Carrot2 Solr plugin

Details

  • Type: Improvement Improvement
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 3.6.0
  • Component/s: None
  • Labels:
    None

Description

As suggested here, it would be good if Carrot2 Solr plugin reused some of Solr's lexical resources, such as stopwords. Ideally, there should still be a way to provide Carrot2-specific resources as the characteristic of IR vs. clustering stopwords sometimes need to be different.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Grant Ingersoll added a comment -

What I had in mind is that I could setup a separate field type for C2 to use and then I can handle them through copy fields.

Show
Grant Ingersoll added a comment - What I had in mind is that I could setup a separate field type for C2 to use and then I can handle them through copy fields.
Hide
Permalink
Grant Ingersoll added a comment -

I'd also add, it may not necessarily be Solr specific. One thought is simply to see if it is possible to feed C2 a Document of Lucene TokenStreams, for instance. I think Solr could easily work with this, but so would Lucene and other tools, too. I suppose the downside for C2 is more dependence on Lucene, but I say why re-invent the wheel when it comes to tokenization, etc.

Show
Grant Ingersoll added a comment - I'd also add, it may not necessarily be Solr specific. One thought is simply to see if it is possible to feed C2 a Document of Lucene TokenStreams, for instance. I think Solr could easily work with this, but so would Lucene and other tools, too. I suppose the downside for C2 is more dependence on Lucene, but I say why re-invent the wheel when it comes to tokenization, etc.
Hide
Permalink
Stanisław Osiński added a comment -

The problem with feeding Carrot2 with TokenStreams is that they'll most likely contain stemmed words and miss stop words. This may not be much of a problem for conventional clustering algorithms that don't put much emphasis on cluster description, but for Carrot2 it means illegible cluster labels (with word stems and missing prepositions). Unless there is some way for TokenStreams to mark certain words (stopwords) instead of removing them?

Show
Stanisław Osiński added a comment - The problem with feeding Carrot2 with TokenStreams is that they'll most likely contain stemmed words and miss stop words. This may not be much of a problem for conventional clustering algorithms that don't put much emphasis on cluster description, but for Carrot2 it means illegible cluster labels (with word stems and missing prepositions). Unless there is some way for TokenStreams to mark certain words (stopwords) instead of removing them?
Hide
Permalink
Stanisław Osiński added a comment -

The 3.4.0 release will concentrate on a native C# API.

Show
Stanisław Osiński added a comment - The 3.4.0 release will concentrate on a native C# API.
Hide
Permalink
Stanisław Osiński added a comment -

Some simple lexical resource sharing will be possible when this issue gets resolved: https://issues.apache.org/jira/browse/SOLR-2450. I'm not closing this issue yet because I'm still tempted to write a counterpart of Carrot2's preprocessing pipeline using Lucene's in-memory index to compare the performance and flexibility.

Show
Stanisław Osiński added a comment - Some simple lexical resource sharing will be possible when this issue gets resolved: https://issues.apache.org/jira/browse/SOLR-2450 . I'm not closing this issue yet because I'm still tempted to write a counterpart of Carrot2's preprocessing pipeline using Lucene's in-memory index to compare the performance and flexibility.
Hide
Permalink
Stanisław Osiński added a comment -

Already done in Solr. Next logical step would be https://issues.apache.org/jira/browse/SOLR-2917.

Show
Stanisław Osiński added a comment - Already done in Solr. Next logical step would be https://issues.apache.org/jira/browse/SOLR-2917 .

People

  • Assignee:
    Stanisław Osiński
    Reporter:
    Stanisław Osiński
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: