All Versions
82
Latest Version
Avg Release Cycle
82 days
Latest Release
779 days ago

Changelog History
Page 3

  • v3.4.0 Changes

    March 01, 2018

    3.4.0, 2018-03-01

    🍱 🌟 New features:

    Massive optimizations of gensim.models.LdaModel: much faster training, using Cython. (@arlenk, #1767)

    🍱 Training benchmark 💥

    ⚡️ | dataset | old LDA [sec] | optimized LDA [sec] | speed up | | --- | --- | --- | --- | | nytimes | 3473 | 1975 | 1.76x | | enron | 774 | 437 | 1.77x |

    This change affects all models that depend on LdaModel , such as LdaMulticore, LdaSeqModel, AuthorTopicModel.

    Huge speed-ups to corpus I/O with MmCorpus (Cython) (@arlenk, #1825)

    File reading benchmark

    ⚡️ | dataset | file compressed? | old MmReader [sec] | optimized MmReader [sec] | speed up | | --- | --- | --- | --- | --- | | enron | no | 22.3 | 2.6 | 8.7x | | | yes | 37.3 | 14.4 | 2.6x | | nytimes | no | 419.3 | 49.2 | 8.5x | | | yes | 686.2 | 275.1 | 2.5x | | text8 | no | 25.4 | 2.5 | 10.1x | | | yes | 41.9 | 17.0 | 2.5x |

    Overall, a 2.5x speedup for compressed .mm.gz input and 8.5x 🔥🔥🔥 for uncompressed plaintext .mm.

    Performance and memory optimization to gensim.models.FastText 🚀 (@jbaiter, #1916)

    Benchmark (first 500,000 articles from English Wikipedia)

    ⚡️ | Metric | old FastText | optimized FastText | improvement | | --- | --- | --- | --- | | Training time (1 epoch) | 4823.4s (80.38 minutes) | 1873.6s (31.22 minutes) | 2.57x | | Training time (full) | 1h 26min 13s | 36min 43s | 2.35x | | Training words/sec | 72,781 | 187,366 | 2.57x | | Training peak memory | 5.2 GB | 3.7 GB | 1.4x |

    Overall, a 2.5x speedup & memory usage reduced by 30%.

    Implemented Soft Cosine Measure (@Witiko, #1827)

    🆕 New method for assessing document similarity, a nice faster alternative to WMD, Word Mover's Distance

    Benchmark

    Technique MAP score Duration
    softcossim 45.99 1.24 sec
    wmd-relax 44.48 12.22 sec
    cossim 44.22 4.39 sec
    wmd-gensim 44.08 98.29 sec

    Soft Cosine notebook with detailed description, examples & benchmarks

    Related papers:

    - [Soft Similarity and Soft Cosine Measure: Similarity of Features in Vector Space Model](http://www.scielo.org.mx/pdf/cys/v18n3/v18n3a7.pdf)
- [SimBow at SemEval-2017 Task 3: Soft-Cosine Semantic Similarity between Questions for Community Question Answering](http://www.aclweb.org/anthology/S17-2051)
- [Vector Space Representations in IR](https://github.com/witiko-masters-thesis/thesis/blob/master/main.pdf)

    🍱 👍 Improvements:

    • New method to show the Gensim installation parameters: python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903)
    • Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
    • Added license field to setup.py, allowing the use of tools like pip-licenses (@nils-werner, #1909)

    🍱 🔴 Bug fixes:

    🍱 📚 Tutorial and doc improvements:

    🚀 ⚠️ Deprecations (will be removed in the next major release)

    ✂ Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)

    - "deprecated" functions and attributes

    🚚 Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

  • v3.3.0 Changes

    February 02, 2018

    3.3.0, 2018-02-02

    🍱 🌟 New features:

    Re-designed all "*2vec" implementations (@manneshiva, #1777)

    • Modular organization of Word2Vec, Doc2Vec, FastText, etc ..., making it easier to add new models in the future and re-use code
    • Fully backward compatible (even with loading models stored by a previous Gensim version)

    - Detailed documentation for the *2vec refactoring project

    👌 Improve gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712
    (@steremma, PR #1839)

    Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article:

    {
    "interlinks": {
        "article title 1": "interlink text 1",
        "article title 2": "interlink text 2",
        ...
    }
}

    Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:

    python -m gensim.scripts.segment\_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz

- Read this field from the `segment_wiki` output:

import jsonfrom smart\_open import smart\_openwith smart\_open("enwiki-latest.jsonl.gz") as infile: for doc in infile: doc = json.loads(doc) src\_node = doc['title'] dst\_nodes = doc['interlinks'].keys() print(u"Source node: {}".format(src\_node)) print(u"Destination nodes: {}".format(u", ".join(dst\_nodes))) break"""OUTPUT:Source node: AnarchismDestination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics

    Add support for SMART notation for TfidfModel. Fix #1785 (@markroxor, #1791)

    • Natural extension of TfidfModel to allow different weighting and normalization schemes

      from gensim.corpora import Dictionaryfrom gensim.models import TfidfModelimport gensim.downloader as api data = api.load("text8") dct = Dictionary(data) corpus = [dct.doc2bow(line) for line in data]# Train Tfidf model using the SMART notation, smartirs="ntc" where# 'n' - natural term frequency# 't' - idf document frequency# 'c' - cosine normalization## More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.htmlmodel = TfidfModel(corpus, id2word=dct, smartirs="ntc") vectorized_corpus = list(model[corpus])

    - SMART Information Retrieval System (wiki)

    Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)

    • An easy way to preview the rendered documentation (especially, if don't use Linux)
      • Go to "Details" link of CircleCI in your PR, click on the "Artifacts" tab, choose the HTML file that you want to view; a new tab will open with the rendered HTML page
    • Integration with Github, to see the documentation directly from the pull request page

    🍱 🔴 Bug fixes:

    🍱 📚 Tutorial and doc improvements:

    🍱 👍 Improvements:

    🚀 ⚠️ Deprecations (will be removed in the next major release)

    ✂ Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)

    - "deprecated" functions and attributes

    🚚 Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

  • v3.2.0 Changes

    December 09, 2017

    3.2.0, 2017-12-09

    🍱 🌟 New features:

    🆕 New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)

    • Download large NLP datasets in one line of Python, then use with memory-efficient data streaming:

      import gensim.downloader as apifor article in api.load("wiki-english-20171001"): print(article)

    • Don’t waste time searching for good word embeddings, use the curated ones:

      import gensim.downloader as api model = api.load("glove-twitter-25") model.most_similar("engineer")# [('specialist', 0.957542896270752),# ('developer', 0.9548177123069763),# ('administrator', 0.9432312846183777),# ('consultant', 0.93915855884552),# ('technician', 0.9368376135826111),# ('analyst', 0.9342101216316223),# ('architect', 0.9257484674453735),# ('engineering', 0.9159940481185913),# ('systems', 0.9123805165290833),# ('consulting', 0.9112802147865295)]

    • Blog post introducing the API and design decisions.

    - Jupyter notebook with examples

    🆕 New model: Poincaré embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)

    • Embed a graph (taxonomy) in the same way as word2vec embeds words:

      from gensim.models.poincare import PoincareRelations, PoincareModelfrom gensim.test.utils import datapath data = PoincareRelations(datapath('poincare_hypernyms.tsv')) model = PoincareModel(data) model.kv.most_similar("cat.n.01")# [('kangaroo.n.01', 0.010581353439700418),# ('gib.n.02', 0.011171531439892076),# ('striped_skunk.n.01', 0.012025106076442395),# ('metatherian.n.01', 0.01246679759214648),# ('mammal.n.01', 0.013281303506525968),# ('marsupial.n.01', 0.013941330203709653)]

    • Tutorial on Poincaré embeddings (Jupyter notebook).

    • Model introduction and the journey of its implementation (blog post).

    - Original paper on arXiv.

    ⚡️ Optimized FastText (@manneshiva, #1742)

    • New fast multithreaded implementation of FastText , natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation.

      import gensim.downloader as apifrom gensim.models import FastText model = FastText(api.load("text8")) model.most_similar("cat")# [('catnip', 0.8538144826889038),# ('catwalk', 0.8136177062988281),# ('catchy', 0.7828493118286133),# ('caf', 0.7826495170593262),# ('bobcat', 0.7745151519775391),# ('tomcat', 0.7732658386230469),# ('moat', 0.7728310823440552),# ('caye', 0.7666271328926086),# ('catv', 0.7651021480560303),# ('caveat', 0.7643581628799438)]

    🐧 Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)

    • Users no longer need to have a C compiler for using the fast (Cythonized) version of word2vec, doc2vec, fasttext etc.

    - Faster Gensim pip installation

    ➕ Added DeprecationWarnings to deprecated methods and parameters, with a clear schedule for removal.

    🍱 👍 Improvements:

    🍱 🔴 Bug fixes:

    🍱 📚 Tutorial and doc improvements:

    🚀 ⚠️ Deprecations (will be removed in the next major release)

    ✂ Remove

    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug

    - gensim.scripts.make_wiki

    🚚 Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

  • v3.1.0 Changes

    November 06, 2017

    :star2: New features:

    • Massive optimizations to LSI model training (@isamaru, #1620 & #1622)

      • LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster.
      • LSI model can now also accept CSC matrix as input, for further memory and speed boost.
      • Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory! ```python # just an example; the corpus stream is up to you streaming_corpus = gensim.corpora.MmCorpus("my_tfidf_corpus.mm.gz")

      convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM)

      in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32)

      then pass the CSC to LsiModel directly

      model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)

      - Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now):
```python
model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)

    • Add common terms to Phrases. Fix #1258 (@alexgarel, #1568)

      • Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like car_with_driver and car_without_driver, you can either remove stop words before processing, but you will only find car_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter. ```python phr_old = Phrases(corpus) phr_new = Phrases(corpus, common_terms=stopwords.words('en'))

      print(phr_old[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with", "driver"] print(phr_new[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with_driver"]

    • New segment_wiki.py script (@menshikh-iv, #1483 & #1694)

      • CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by WikiMedia) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line: bash python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).

      The output format is one article per line, serialized into JSON:

            for line in smart_open('enwiki-latest-pages-articles.json.gz'):  # read the file we just created
          article = json.loads(line)
          print("Article title: %s" % article['title'])
          for section_title, section_text in zip(article['section_titles'], article['section_texts']):
              print("Section title: %s" % section_title)
              print("Section text: %s" % section_text)
    ```

    :+1: Improvements:

    🛠 :red_circle: Bug fixes:

    :books: Tutorial and doc improvements:

    🚀 :warning: Deprecation part (will come into force in the next major release)

    • ✂ Remove

      • gensim.examples
      • gensim.nosy
      • gensim.scripts.word2vec_standalone
      • gensim.scripts.make_wiki_lemma
      • gensim.scripts.make_wiki_online
      • gensim.scripts.make_wiki_online_lemma
      • gensim.scripts.make_wiki_online_nodebug
      • gensim.scripts.make_wiki

    • 🚚 Move

      • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
      • gensim.summarization ➡ gensim.models.summarization
      • gensim.topic_coherence ➡ gensim.models._coherence
      • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
      • gensim.parsing.* ➡ gensim.utils.text_utils

    🚀 Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.

  • v3.0.1 Changes

    October 12, 2017

    🛠 :red_circle: Bug fixes:

    • 🛠 Fix Keras import, speedup importing time. Fix #1614 (@menshikh-v, #1615)
    • 🛠 Fix Sphinx warnings and retreive all missing .rst (@anotherbugmaster and @menshikh-iv, #1612)
    • 🛠 Fix logger message in lsi_dispatcher (@lorosanu, #1603)

    :books: Tutorial and doc improvements:

    • 🛠 Fix spelling (@jberkel, #1625)

    🚀 :warning: Deprecation part (will come into force in the next release)

    • ✂ Remove

      • gensim.examples
      • gensim.nosy
      • gensim.scripts.word2vec_standalone
      • gensim.scripts.make_wiki_lemma
      • gensim.scripts.make_wiki_online
      • gensim.scripts.make_wiki_online_lemma
      • gensim.scripts.make_wiki_online_nodebug
      • gensim.scripts.make_wiki

    • 🚚 Move

      • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
      • gensim.summarization ➡ gensim.models.summarization
      • gensim.topic_coherence ➡ gensim.models._coherence
      • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
      • gensim.parsing.* ➡ gensim.utils.text_utils

    🚀 Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next release.

  • v3.0.0 Changes

    September 27, 2017

    :star2: New features:

    • ➕ Add unsupervised FastText to Gensim (@chinmayapancholi13, #1525)
    • ➕ Add sklearn API for gensim models (@chinmayapancholi13, #1462)
    • ➕ Add callback metrics for LdaModel and integration with Visdom (@parulsethi, #1399)
    • ➕ Add TranslationMatrix model (@robotcator, #1434)
    • ➕ Add word2vec-based coherence. Fix #1380 (@macks22, #1530)

    :+1: Improvements:

    • ➕ Add 'diagonal' parameter for LdaModel.diff (@parulsethi, #1448)
    • ➕ Add 'score' function for SklLdaModel (@chinmayapancholi13, #1445)
    • ⚡️ Update sklearn API for gensim models (@chinmayapancholi13, #1473) [:warning: breaks backward compatibility]
    • ➕ Add CoherenceModel to LdaModel.top_topics. Fix #1128 (@macks22, #1427)
    • ➕ Add dendrogram viz for topics and JS metric (@parulsethi, #1484)
    • ➕ Add topic network viz (@parulsethi, #1536)
    • Replace viewitems to iteritems. Fix #1495 (@HodorTheCoder, #1508)
    • 🛠 Fix Travis config and add style-checking for Ipython Notebooks. Fix #1518, #1520 (@menshikh-iv, #1522)
    • ✂ Remove mutable args from definitions. Fix #1561 (@zsef123, #1562)
    • ➕ Add Appveyour for all PRs. Fix #1565 (@menshikh-iv, #1565)
    • 🔨 Refactor code by PEP8. Partially fix #1521 (@zsef123, #1550)
    • 🔨 Refactor code by PEP8 with additional limitations. Fix #1521 (@menshikh-iv, #1569)
    • Update FastTextKeyedVectors.__contains__ (@ELind77, #1499)
    • ⚡️ Update WikiCorpus tokenization. Fix #1534 (@roopalgarg, #1537)

    🛠 :red_circle: Bug fixes:

    • ✂ Remove round in LdaSeqModel.print_topic. Fix #1480 (@menshikh-iv, #1547)
    • 🛠 Fix TextCorpus.samle_text (@menshikh-iv, #1548)
    • 🛠 Fix Mallet wrapper and tests for HDPTransform (@menshikh-iv, #1555)
    • 🛠 Fix incorrect initialization ShardedCorpus with a generator. Fix #1511 (@karkkainenk1, #1512)
    • ➕ Add verification when summarize_corpus returns null. Fix #1531 (@fbarrios, #1570)
    • 🛠 Fix doctag unicode problem. Fix 1543 (@englhardt, #1544)
    • 🛠 Fix Translation Matrix (@robotcator, #1594)
    • Add trainable flag to KeyedVectors.get_embedding_layer. Fix #1557 (@zsef123, #1558)

    :books: Tutorial and doc improvements:

    • ⚡️ Update exception text in TextCorpus.samle_text. Partial fix #308 (@vlejd, #1444)
    • ✂ Remove extra filter_token from tutorial (@VorontsovIE, #1502)
    • ⚡️ Update Doc2Vec-IMDB notebook (@pahdo, #1476)
    • ➕ Add Google Tag Manager for site (@yardos, #1556)
    • ⚡️ Update docstring explaining lack of multistream support in WikiCopus. Fix #1496 (@polm and @menshikh-iv, #1515)
    • 🛠 Fix PathLineSentences docstring (@gojomo)
    • 🛠 Fix typos from Translation Matrix notebook (@robotcator, #1598)

  • v2.3.0 Changes

    July 25, 2017

    :star2: New features:

    • ➕ Add Dockerfile for gensim with external wrappers (@parulsethi, #1368)
    • ➕ Add sklearn wrapper for Word2Vec (@chinmayapancholi13, #1437)
    • ➕ Add loss function for Word2Vec. Fix #999 (@chinmayapancholi13, #1201)
    • ➕ Add sklearn wrapper for AuthorTopic model (@chinmayapancholi13, #1403)

    :+1: Improvements:

    • ✂ Remove unittest2 (@souravsingh, #1490)
    • ➕ Add multiple scoring methods for Phrases. Partial fix #1363 (@michaelwsherman, #1464)
    • ➕ Add WordRank wrapper to Dockerfile (@parulsethi, #1460)
    • ➕ Add PathLineSentences. Fix #1364 (@michaelwsherman, #1423)
    • ➕ Add TextDirectoryCorpus and refactor TextCorpus. Fix #1387 (@macks22, #1459)
    • ➕ Add sparse input support with topn parameter in any2sparse. Fix #1294 (@manneshiva, #1321)
    • ➕ Add seed and length for sample_text. Partial fix #308 (@vlejd, #1422)
    • ➕ Add word_ngram parameter to FastText (@fsonntag, #1432)

    🛠 :red_circle: Bug fixes:

    • 🛠 Fix fastText loading from .bin file. Fix #1236 (@prakhar2b, #1341)
    • 🛠 Fix paths in WordRank and running gensim version in Dockerfile (@parulsethi, #1503)
    • 🛠 Fix commit version for gensim in Dockerfile (@parulsethi, #1491)
    • 🛠 Fix encoding problems with tests on windows. Fix #1441 (@menshikh-iv, #1469)
    • Fix parameters in score_cbow_pair (@jmhessel, #1468)
    • Fix parameters in score_sentence_cbow (@jmhessel, #1467)
    • 🛠 Fix TextDirectoryCorpus on windows (@macks22, #1463)
    • 🛠 Fix gensim version in Dockerfile (@parulsethi, #1456)
    • 🛠 Fix WordOccurenceAccumulator on windows. Fix #1441 (@macks22, #1449)
    • 🛠 Fix scipy/numpy requirements (downgrade). Fix #1450 (@menshikh-iv, #1450)

    :books: Tutorial and doc improvements:

    • 🛠 Fix links and spaces in quick start guide (@iamsanten, #1500)
    • 🛠 Fix error of ConcatedDoc2Vec in doc2vec-imdb notebook (@robocator, #1377)
    • 🛠 Fix Sphinx warnings. Fix #1192 (@prerna135, #1442)
    • 🛠 Fix typo in LdaModel.diff method (@parulsethi, #1461)
    • ➕ Add Tensorboard visualization for LDA (@parulsethi, #1396)
    • ⚡️ Update old and add new notebook with CoherenceModel (@macks22, #1431)

  • v2.2.0 Changes

    June 21, 2017

    :star2: New features:

    • ➕ Add sklearn wrapper for RpModel (@chinmayapancholi13, #1395)
    • ➕ Add sklearn wrappers for LdaModel and LsiModel (@chinmayapancholi13, #1398)
    • ➕ Add sklearn wrapper for LdaSeq (@chinmayapancholi13, #1405)
    • ➕ Add keras wrapper for Word2Vec model (@chinmayapancholi13, #1248)
    • ➕ Add LdaModel.diff method (@menshikh-iv, #1334)
    • 👍 Allow use of truncated Dictionary for coherence measures. Fix #1342 (@macks22, #1349)

    :+1: Improvements:

    • Fix save_as_text/load_as_text for Dictionary (@vlejd, #1402)
    • ➕ Add sampling support for corpus. Fix #308 (@vlejd, #1408)
    • ➕ Add napoleon extension to sphinx (@rasto2211, #1411)
    • ➕ Add KeyedVectors support to AnnoyIndexer (@quole, #1318)
    • ➕ Add BaseSklearnWrapper (@chinmayapancholi13, #1383)
    • Replace num_words to topn in model for unification. Fix #1198 (@prakhar2b, #1200)
    • Rename out_path to out_name & add logging for WordRank model. Fix #1310 (@parulsethi, #1332)
    • Remove multiple iterations of corpus in p_boolean_document (@danielchamberlain, #1325)
    • 🛠 Fix codestyle in TfIdf (@piskvorky, #1313)
    • 🛠 Fix warnings from Sphinx. Partial fix #1192 (@souravsingh, #1330)
    • ➕ Add test_env to setup.py (@menshikh-iv, #1336)

    🛠 :red_circle: Bug fixes:

    • ➕ Add cleanup in annoy test (@prakhar2b, #1420)
    • ➕ Add cleanup in lda backprop test (@prakhar2b, #1417)
    • 🛠 Fix out-of-vocab in FastText (@jayantj, #1409)
    • ➕ Add cleanup in WordRank test (@parulsethi, #1410)
    • 🛠 Fix rest requirements in Travis. Partial fix #1393 (@ibrahimsharaf, @menshikh-iv, #1400)
    • 🛠 Fix morfessor exception. Partial fix #1324 (@souravsingh, #1406)
    • 🛠 Fix test for FastText (@prakhar2b, #1371)
    • 🛠 Fix WikiCorpus (@alekol, #1333)
    • 🛠 Fix backward incompatibility for LdaModel (@chinmayapancholi13, #1327)
    • 🛠 Fix support for old and new FastText model format. Fix #1301 (@prakhar2b, #1319)
    • 🛠 Fix wrapper tests. Fix #1323 (@shubhamjain74, #1359)
    • ⚡️ Update export_phrases method. Fix #794 (@toumorokoshi, #1362)
    • 🛠 Fix sklearn exception in test (@souravsingh, #1350)

    :books: Tutorial and doc improvements:

    • 🛠 Fix incorrect link in tutorials (@aneesh-joshi, #1426)
    • ➕ Add notebook with sklearn wrapper examples (@chinmayapancholi13, #1428)
    • Replace absolute pathes to relative in notebooks (@vochicong, #1414)
    • 🛠 Fix code-style in keras notebook (@chinmayapancholi13, #1394)
    • Replace absolute pathes to relative in notebooks (@vochicong, #1407)
    • 🛠 Fix typo in quickstart guide (@vochicong, #1404)
    • ⚡️ Update docstring for WordRank. Fix #1384 (@parulsethi, #1378)
    • ⚡️ Update docstring for SkLdaModel (@chinmayapancholi13, #1382)
    • ⚡️ Update logic for updatetype in LdaModel (@chinmayapancholi13, #1389)
    • ⚡️ Update docstring for Doc2Vec (@jstol, #1379)
    • 🛠 Fix docstring for KL-distance (@viciousstar, #1373)
    • Update Corpora_and_Vector_Spaces tutorial (@charliejharrison, #1308)
    • ➕ Add visualization for difference between LdaModel (@menshikh-iv, #1374)
    • 🛠 Fix punctuation & typo in changelog (@piskvorky, @menshikh-iv, #1366)
    • 🛠 Fix PEP8 & typo in several PRs (@menshikh-iv, #1369)
    • ⚡️ Update docstrings connected with backward compability in for LdaModel (@chinmayapancholi13, #1365)
    • Update Corpora_and_Vector_Spaces tutorial (@schuyler1d, #1360)
    • 🛠 Fix typo in Doc2Vec doctsring (@fujiyuu75, #1356)
    • ⚡️ Update Annoy tutorial (@pmbaumgartner, #1355)
    • ⚡️ Update temp folder in tutorials (@yl2526, #1352)
    • Remove spaces after print in Topics_and_Transformation tutorial (@gsimore, #1354)
    • ⚡️ Update Dictionary docstring (@oonska, #1347)
    • ➕ Add section headings in word2vec notebook (@MikeTheReader, #1348)
    • 🛠 Fix broken urls in starter tutorials (@ka7eh, #1346)
    • ⚡️ Update quick start notebook (@yardsale8, #1345)
    • 🛠 Fix typo in quick start notebook (@MikeTheReader, #1344)
    • 🛠 Fix docstring in keyedvectors (@chinmayapancholi13, #1337)

  • v2.1.0 Changes

    May 12, 2017

    :star2: New features:

    • Add modified save_word2vec_format for Doc2Vec, to save document vectors. (@parulsethi, #1256)

    :+1: Improvements:

    • ➕ Add automatic code style check limited only to the code modified in PR (@tmylk, #1287)
    • ⚠ Replace logger.warn by logger.warning (@chinmayapancholi13, #1295)
    • 📄 Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274)
    • Stop passing 'sentences' as parameter to Doc2Vec. Fix #511 (@gogokaradjov, #1306)

    🛠 :red_circle: Bug fixes:

    • 👍 Allow indexing with np.int64 in doc2vec. Fix #1231 (@bogdanteleaga, #1254)
    • ⚡️ Update Doc2Vec docstring. Fix #1302 (@datapythonista, #1307)
    • Ignore rst and ipynb file in Travis flake8 validations (@datapythonista, #1309)

    :books: Tutorial and doc improvements:

    • ⚡️ Update Tensorboard Doc2Vec notebook (@parulsethi, #1286)
    • ⚡️ Update Doc2Vec IMDB Notebook, replace codesc to smart_open (@robotcator, #1278)
    • ➕ Add explanation of size to Word2Vec Notebook (@jbcoe, #1305)
    • ➕ Add extra param to WordRank notebook. Fix #1276 (@parulsethi, #1300)
    • ⚡️ Update warning message in WordRank (@parulsethi, #1299)

  • v2.0.0 Changes

    April 10, 2017

    💥 Breaking changes:

    Any direct calls to method train() of Word2Vec/Doc2Vec now require an explicit epochs parameter and explicit estimate of corpus size. The most usual way to call train is vec_model.train(sentences, total_examples=self.corpus_count, epochs=self.iter) 📚 See the method documentation for more information.

    • Explicit epochs and corpus size in word2vec train(). (@gojomo, @robotcator, #1139, #1237)

    🆕 New features:

    • ➕ Add output word prediction in word2vec. Only for negative sampling scheme. See ipynb (@chinmayapancholi13, #1209)
    • scikit_learn wrapper for LSI Model in Gensim (@chinmayapancholi13, #1244)
    • Add the 'keep_tokens' parameter to 'filter_extremes'. (@toliwa, #1210)
    • Load FastText models with specified encoding (@jayantj, #1210)

    👌 Improvements:

    • 🛠 Fix loading large FastText models on Mac. (@jaksmid, #1196)
    • Sklearn LDA wrapper now works in sklearn pipeline (@kris-singh, #1213)
    • 🔨 glove2word2vec conversion script refactoring (@parulsethi, #1247)
    • ⚡️ Word2vec error message when update called before train . Fix #1162 (@hemavakade, #1205)
    • Allow training if model is not modified by "_minimize_model". Add deprecation warning. (@chinmayapancholi13, #1207)
    • ⚡️ Update the warning text when building vocab on a trained w2v model (@prakhar2b, #1190)

    🐛 Bug fixes:

    • Fix word2vec reset_from bug in v1.0.1 Fix #1230. (@Kreiswolke, #1234)

    • 📄 Distributed LDA: checking the length of docs instead of the boolean value, plus int index conversion (@saparina, #1191)

    • syn0_lockf initialised with zero in intersect_word2vec_format() (@KiddoZhu, #1267)

    • Fix wordrank max_iter_dump calculation. Fix #1216 (@ajkl, #1217)

    • ✅ Make SgNegative test use skip-gram (@shubhvachher, #1252)

    • 💅 pep8/pycodestyle fixes for hanging indents in Summarization module (@SamriddhiJain, #1202)

    • 🏁 WordRank and Mallet wrappers single vs double quote issue in windows. (@prakhar2b, #1208)

    • 🛠 Fix #824 : no corpus in init, but trim_rule in init (@prakhar2b, #1186)

    • Hardcode version number. Fix #1138. (@tmylk, #1138)

    Tutorial and doc improvements:

    • ⚡️ Color dictionary according to topic notebook update (@bhargavvader, #1164)

    • 🛠 Fix hdp show_topic/s docstring (@parulsethi, #1264)

    • ➕ Add docstrings for word2vec.py forwarding functions (@shubhvachher, #1251)

    • ⚡️ updated description for worker_loop function used in score function (@chinmayapancholi13, #1206)