Carrot2

tf matrix values may be incorrectly computed in TermDocumentMatrixBuilder

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 3.4.4, 3.5.0, 3.5.0.1, 3.5.1, 3.5.2, 3.5.3
  • Fix Version/s: 3.6.0
  • Component/s: Clustering Algorithms
  • Labels:
    None

Description

This loop is on the dark side of the force:

            for (int documentIndex = 0; documentIndex < documentCount; documentIndex++)
            {
                if (tfByDocumentIndex * 2 < tfByDocument.length
                    && tfByDocument[tfByDocumentIndex * 2] == documentIndex)
                {
                    double weight = termWeighting.calculateTermWeight(
                        tfByDocument[tfByDocumentIndex * 2 + 1], df, documentCount);

                    weight *= getWeightBoost(titleFieldIndex, fieldIndices);
                    tfByDocumentIndex++;

                    tdMatrix.set(i, documentIndex, weight);
                }
            }

Originally reported by Taojian Lu.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Stanisław Osiński added a comment -

The issue seems to be relevant only when TermDocumentMatrixBuilder is used separately from Carrot2 preprocessing pipeline. Otherwise, indices in tfByDocumentIndex should be increasing (assuming that IndirectSorter.mergesort() is stable) and the bug should not affect the results.

Aside from that, the loop code didn't make any sense, it's fixed in master.

Show
Stanisław Osiński added a comment - The issue seems to be relevant only when TermDocumentMatrixBuilder is used separately from Carrot2 preprocessing pipeline. Otherwise, indices in tfByDocumentIndex should be increasing (assuming that IndirectSorter.mergesort() is stable) and the bug should not affect the results. Aside from that, the loop code didn't make any sense, it's fixed in master.
Hide
Permalink
Stanisław Osiński added a comment -

In fact, the issue document indices will be increasing only if all words are 1:1 with stems. If there are multiple original words per stem, indices will not be increasing and therefore the term-document matrix will be incorrect.

The bug fix has been pushed to master on GitHub.

Show
Stanisław Osiński added a comment - In fact, the issue document indices will be increasing only if all words are 1:1 with stems. If there are multiple original words per stem, indices will not be increasing and therefore the term-document matrix will be incorrect. The bug fix has been pushed to master on GitHub.
Hide
Permalink
Dawid Weiss added a comment -

This patch changed the output results in some cases – that's why solr integration tests fail now.

Show
Dawid Weiss added a comment - This patch changed the output results in some cases – that's why solr integration tests fail now.

People

  • Assignee:
    Stanisław Osiński
    Reporter:
    Dawid Weiss
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: