piskvorky/gensim

Ambiguous `docvecs` indexing, documentation missing

Open

#2.097 aberto em 20 de jun. de 2018

Ver no GitHub
 (4 comments) (0 reactions) (0 assignees)Python (4.349 forks)batch import
Hacktoberfestdifficulty easydocumentation

Métricas do repositório

Stars
 (15.144 stars)
Métricas de merge de PR
 (Nenhuma PRs mesclada em 30d)

Description

Description

The tagging system for training corpora appears to result in ambiguous indexing on model.docvecs

Steps/Code/Corpus to Reproduce

This example is a lightly modified version of code snippets from the main demo.

import os
import smart_open
import gensim

# Set file names for train and test data
test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])
lee_train_file = test_data_dir + os.sep + 'lee_background.cor'

def read_corpus(fname, offset = 0):
    with smart_open.smart_open(fname, encoding="iso-8859-1") as f:
        for i, line in enumerate(f):
            yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), [i + offset])

def build_model(corpus):
    model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
    model.build_vocab(corpus)
    model.train(corpus, total_examples=model.corpus_count, epochs=model.epochs)
    return model

corpus = list(read_corpus(lee_train_file))
offset_corpus = list(read_corpus(lee_train_file, offset = 100))

model = build_model(corpus)
offset_model = build_model(offset_corpus)

Expected Results

The only difference between the two models is that the training corpora use different sets of tags.

  1. The number of training documents is the same for both, so I would expect len(model.docvecs) == len(offset_model.docvecs)

  2. offset_corpus.docvecs[0] should throw an error since 0 is not among its tags.

  3. OR ... if not (2), then offset_corpus.docvecs[0] should correspond to the first document, in which case offset_corpus.docvecs[0] == offset_corpus.docvecs[100]. Specifically, note that offset_corpus.docvecs[399] appears to return a valid result.

Actual Results

None of (1), (2), or (3) hold.

Versions

>>> import platform; print(platform.platform())
Darwin-16.7.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.6.1 |Anaconda 4.4.0 (x86_64)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.14.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.0.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.4.0
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 0

Guia do colaborador