pytext v0.3.0 Release Notes

Release Date: 2019-11-28 // over 4 years ago
  • ๐Ÿ†• New Features

    RoBERTa and XLM-R

    • โ†” Integrate XLM-R into PyText (#1120)
    • Consolidate BERT, XLM and RobERTa Tensorizers (#1119)
    • โž• Add XLM-R for joint model (#1135)
    • Open source Roberta (#1032)
    • Simple Transformer module components for RoBERTa (#1043)
    • RoBERTa models for document classification (#933)
    • Enable MLM training for RobertaEncoder (#1126)
    • Standardize RoBERTa Tensorizer Vocab Creation (#1113)
    • ๐Ÿ‘‰ Make RoBERTa usable in more tasks including QA (#1017)
    • RoBERTa-QA JIT (#1088)
    • Unify GPT2BPE Tokenizer (#1110)
    • โž• Adding Google SentencePiece as a Tokenizer (#1106)

    ๐Ÿ‘ TorchScript support

    • General torchscript module (#1134)
    • ๐Ÿ‘Œ Support torchscriptify XLM-R (#1138)
    • โž• Add support for torchscriptification of XLM intent slot models (#1167)
    • Script xlm tensorizer (#1118)
    • ๐Ÿ”จ Refactor ScriptTensorizer with general tensorize API (#1117)
    • ScriptXLMTensorizer (#1123)
    • โž• Add support for Torchscript export of IntentSlotOutputLayer and CRF (#1146)
    • ๐Ÿ”จ Refactor ScriptTensorizor to support both text and tokens input (#1096)
    • โž• Add torchscriptify API in tokenizer and tensorizer (#1055)
    • โž• Add more stats in torchscript latency script (#1044)
    • Exported Roberta torchscript model include both traced_model and pre-processing logic (#1013)
    • Native Torchscript Wordpiece Tokenizer Op for BERTSquadQA, Torchscriptify BertSQUADQAModel (#879)
    • TorchScript-ify BERT training (#887)
    • Modify Return Signature of TorchScript BERT (#1058)
    • Implement BertTensorizer and RoBERTaTensorizer in TorchScript (#1053)

    Others

    • FairseqModelEnsemble class (#1116)
    • โฑ Inverse Sqrt Scheduler (#1150)
    • Lazy modules (#1039)
    • โšก๏ธ Adopt Fairseq MemoryEfficientFP16Optimizer in PyText (#910)
    • โž• Add RAdam (#952)
    • โž• Add AdamW (#945)
    • Unify FP16&FP32 API (#1006)
    • โž• Add precision at recall metric (#1079)
    • โž• Added PandasDataSource (#1098)
    • ๐Ÿ‘Œ Support testing Caffe2 model (#1097)
    • โž• Add contextual feature support to export for Seq2Seq models
    • Convert matmuls to quantizable nn.Linear modules (#1304)
    • PyTorch eager mode implementation (#1072)
    • Implement Blockwise Sparsification (#1050)
    • ๐Ÿ‘Œ Support Fairseq FP16Optimizer (#1008)
    • โšก๏ธ Make FP16OptimizerApex wrapper on Apex/amp (#1007)
    • โœ‚ Remove vocab from cuda (#955)
    • โž• Add dense input to XLMModel (#997)
    • Replace tensorboardX with torch.utils.tensorboard (#1003)
    • โž• Add mentioning of mixed precision training support (#643)
    • Sparsification for CRF transition matrix (#982)
    • โž• Add dense feature normalization to Char-LSTM TorchScript model. (#986)
    • ๐Ÿ‘ Cosine similarity support for BERT pairwise model training (#967)
    • Combine training data from multiple sources (#953)
    • ๐Ÿ‘Œ Support visualization of word embeddings in Tensorboard (#969)
    • Decouple decoder and output layer creation in BasePairwiseModel (#973)
    • โฌ‡๏ธ Drop rows with insufficient columns in TSV data source (#954)
    • Add use_config_from_snapshot option(load config from snapshot or current task) (#970)
    • โž• Add predict function for NewTask (#936)
    • ๐Ÿ‘‰ Use create_module to create CharacterEmbedding (#920)
    • โž• Add XLM based joint model
    • โž• Add ConsistentXLMModel (#913)
    • โšก๏ธ Optimize Gelu module for caffe2 export (#918)
    • ๐Ÿ’พ Save best model's sub-modules when enabled (#912)

    ๐Ÿ“š Documentation / Usability

    • XLM-R tutorial in notebook (#1159)
    • โšก๏ธ Update XLM-R OSS tutorial and add Google Colab link (#1168)
    • โšก๏ธ Update "raw_text" to "text" in tutorial (#1010)
    • ๐Ÿ‘ฏ Make tutorial more trivial (add git clone) (#1037)
    • ๐Ÿ”„ Changes to make tutorial code simpler (#1002)
    • ๐Ÿ›  Fix datasource tutorial example (#998)
    • ๐Ÿ– Handle long documents in squad qa datasource and models (#975)
    • ๐Ÿ›  Fix pytext tutorial syntax (#971)
    • ๐Ÿ‘‰ Use torch.equal() instead of "==" in Custom Tensorizer tutorial (#939)
    • โœ‚ Remove and mock doc dependencies because readthedocs is OOM (#983)
    • ๐Ÿ›  Fix Circle CI build_docs error (#959)
    • โž• Add OSS integration tests: DocNN (#1021)
    • ๐Ÿ–จ Print model into the output log (#1127)
    • Migrate pytext/utils/torch.py logic into pytext/torchscript/ for long term maintainability (#1082)
    • Demo datasource fix + cleanup (#994)
    • ๐Ÿ“š Documentation on the config files and config-related commands (#984)
    • Config adapter old data handler helper (#943)
    • Nicer gen_config_impl (#944)

    ๐Ÿ—„ Deprecated Features

    • โœ‚ Remove DocModel_Deprecated (#916)
    • ๐Ÿ—„ Remove RNNGParser_Deprecated, SemanticParsingTask_Deprecated, SemanticParsingCppTask_Deprecate, RnngJitTask,
    • โœ‚ Remove QueryDocumentTask_Deprecated(#926)
    • ๐Ÿ—„ Remove LMTask_Deprecated and LMLSTM_Deprecated (#882)
    • ๐Ÿ—„ CompositionDataHandler to fb/deprecated (#963)
    • โœ‚ Delete deprecated Word Tagging tasks, models and data handlers (#910)

    ๐Ÿ› Bug Fixes

    • ๐Ÿ›  Fix caffe2 predict (#1103)
    • ๐Ÿ›  Fix bug when tensorizer is not defined (#1169)
    • ๐Ÿ›  Fix multitask metric reporter for lr logging (#1164)
    • ๐Ÿ›  Fix broken gradients logging and add lr logging to tensorboard (#1158)
    • Minor fix in blockwise sparsifier (#1130)
    • Fix clip_grad_norm API (#1143)
    • ๐Ÿ›  Fix for roberta squad tensorizer (#1137)
    • ๐Ÿ›  Fix multilabel metric reporter (#1115)
    • ๐Ÿ›  Fixed prepare_input in tensorizer (#1102)
    • ๐Ÿ›  Fix unk bug in exported model (#1076)
    • ๐Ÿ›  Fp16 fixes for byte-lstm and distillation (#1059)
    • Fix clip_grad_norm_ if grad_norm > max_norm > 0: TypeError: '>' not supported between instances of 'float' and 'NoneType' (#1054)
    • ๐Ÿ›  Fix context in multitask (#1040)
    • ๐Ÿ›  Fix regression in ensemble trainer caused by recent fp16 change (#1033)
    • ReadTheDocs OOM fix with CPU Torch (#1027)
    • Dimension mismatch after setting max sequence length (#1154)
    • ๐Ÿ‘ Allow null learning rate (#1156)
    • Don't fail on 0 input (#1104)
    • โœ‚ Remove side effect during pickling PickleableGPT2BPEEncoder
    • ๐Ÿ— Set onnx==1.5.0 to fix CircleCI build temporarily (#1014)
    • Complete training loop gracefully even if no timing is reported (#1128)
    • Propagate min_freq for vocab correctly (#907)
    • ๐Ÿ›  Fix gen-default-config with Model param (#917)
    • ๐Ÿ›  Fix torchscript export for PyText modules (#1125)
    • ๐Ÿ›  Fix label_weights in DocModel (#1081)
    • ๐Ÿ›  Fix label_weights in bert models (#1100)
    • ๐Ÿ›  Fix config issues with Python 3.7 (#1066)
    • Temporary fix for Fairseq dependency (#1026)
    • ๐Ÿ›  Fix MultipleData by making tensorizers able to initialize from multiple data sources (#972)
    • ๐Ÿ›  Fix bug in copy_unk (#964)
    • Division by Zero bug in MLM Metric Reporter (#968)