pytext v0.3.0 Release Notes
Release Date: 2019-11-28 // over 4 years ago-
๐ New Features
RoBERTa and XLM-R
- โ Integrate XLM-R into PyText (#1120)
- Consolidate BERT, XLM and RobERTa Tensorizers (#1119)
- โ Add XLM-R for joint model (#1135)
- Open source Roberta (#1032)
- Simple Transformer module components for RoBERTa (#1043)
- RoBERTa models for document classification (#933)
- Enable MLM training for RobertaEncoder (#1126)
- Standardize RoBERTa Tensorizer Vocab Creation (#1113)
- ๐ Make RoBERTa usable in more tasks including QA (#1017)
- RoBERTa-QA JIT (#1088)
- Unify GPT2BPE Tokenizer (#1110)
- โ Adding Google SentencePiece as a Tokenizer (#1106)
๐ TorchScript support
- General torchscript module (#1134)
- ๐ Support torchscriptify XLM-R (#1138)
- โ Add support for torchscriptification of XLM intent slot models (#1167)
- Script xlm tensorizer (#1118)
- ๐จ Refactor ScriptTensorizer with general tensorize API (#1117)
- ScriptXLMTensorizer (#1123)
- โ Add support for Torchscript export of IntentSlotOutputLayer and CRF (#1146)
- ๐จ Refactor ScriptTensorizor to support both text and tokens input (#1096)
- โ Add torchscriptify API in tokenizer and tensorizer (#1055)
- โ Add more stats in torchscript latency script (#1044)
- Exported Roberta torchscript model include both traced_model and pre-processing logic (#1013)
- Native Torchscript Wordpiece Tokenizer Op for BERTSquadQA, Torchscriptify BertSQUADQAModel (#879)
- TorchScript-ify BERT training (#887)
- Modify Return Signature of TorchScript BERT (#1058)
- Implement BertTensorizer and RoBERTaTensorizer in TorchScript (#1053)
Others
- FairseqModelEnsemble class (#1116)
- โฑ Inverse Sqrt Scheduler (#1150)
- Lazy modules (#1039)
- โก๏ธ Adopt Fairseq MemoryEfficientFP16Optimizer in PyText (#910)
- โ Add RAdam (#952)
- โ Add AdamW (#945)
- Unify FP16&FP32 API (#1006)
- โ Add precision at recall metric (#1079)
- โ Added PandasDataSource (#1098)
- ๐ Support testing Caffe2 model (#1097)
- โ Add contextual feature support to export for Seq2Seq models
- Convert matmuls to quantizable nn.Linear modules (#1304)
- PyTorch eager mode implementation (#1072)
- Implement Blockwise Sparsification (#1050)
- ๐ Support Fairseq FP16Optimizer (#1008)
- โก๏ธ Make FP16OptimizerApex wrapper on Apex/amp (#1007)
- โ Remove vocab from cuda (#955)
- โ Add dense input to XLMModel (#997)
- Replace tensorboardX with torch.utils.tensorboard (#1003)
- โ Add mentioning of mixed precision training support (#643)
- Sparsification for CRF transition matrix (#982)
- โ Add dense feature normalization to Char-LSTM TorchScript model. (#986)
- ๐ Cosine similarity support for BERT pairwise model training (#967)
- Combine training data from multiple sources (#953)
- ๐ Support visualization of word embeddings in Tensorboard (#969)
- Decouple decoder and output layer creation in BasePairwiseModel (#973)
- โฌ๏ธ Drop rows with insufficient columns in TSV data source (#954)
- Add use_config_from_snapshot option(load config from snapshot or current task) (#970)
- โ Add predict function for NewTask (#936)
- ๐ Use
create_module
to create CharacterEmbedding (#920) - โ Add XLM based joint model
- โ Add
ConsistentXLMModel
(#913) - โก๏ธ Optimize Gelu module for caffe2 export (#918)
- ๐พ Save best model's sub-modules when enabled (#912)
๐ Documentation / Usability
- XLM-R tutorial in notebook (#1159)
- โก๏ธ Update XLM-R OSS tutorial and add Google Colab link (#1168)
- โก๏ธ Update "raw_text" to "text" in tutorial (#1010)
- ๐ฏ Make tutorial more trivial (add git clone) (#1037)
- ๐ Changes to make tutorial code simpler (#1002)
- ๐ Fix datasource tutorial example (#998)
- ๐ Handle long documents in squad qa datasource and models (#975)
- ๐ Fix pytext tutorial syntax (#971)
- ๐ Use torch.equal() instead of "==" in Custom Tensorizer tutorial (#939)
- โ Remove and mock doc dependencies because readthedocs is OOM (#983)
- ๐ Fix Circle CI build_docs error (#959)
- โ Add OSS integration tests: DocNN (#1021)
- ๐จ Print model into the output log (#1127)
- Migrate pytext/utils/torch.py logic into pytext/torchscript/ for long term maintainability (#1082)
- Demo datasource fix + cleanup (#994)
- ๐ Documentation on the config files and config-related commands (#984)
- Config adapter old data handler helper (#943)
- Nicer gen_config_impl (#944)
๐ Deprecated Features
- โ Remove DocModel_Deprecated (#916)
- ๐ Remove RNNGParser_Deprecated, SemanticParsingTask_Deprecated, SemanticParsingCppTask_Deprecate, RnngJitTask,
- โ Remove QueryDocumentTask_Deprecated(#926)
- ๐ Remove LMTask_Deprecated and LMLSTM_Deprecated (#882)
- ๐ CompositionDataHandler to fb/deprecated (#963)
- โ Delete deprecated Word Tagging tasks, models and data handlers (#910)
๐ Bug Fixes
- ๐ Fix caffe2 predict (#1103)
- ๐ Fix bug when tensorizer is not defined (#1169)
- ๐ Fix multitask metric reporter for lr logging (#1164)
- ๐ Fix broken gradients logging and add lr logging to tensorboard (#1158)
- Minor fix in blockwise sparsifier (#1130)
- Fix clip_grad_norm API (#1143)
- ๐ Fix for roberta squad tensorizer (#1137)
- ๐ Fix multilabel metric reporter (#1115)
- ๐ Fixed prepare_input in tensorizer (#1102)
- ๐ Fix unk bug in exported model (#1076)
- ๐ Fp16 fixes for byte-lstm and distillation (#1059)
- Fix clip_grad_norm_ if grad_norm > max_norm > 0: TypeError: '>' not supported between instances of 'float' and 'NoneType' (#1054)
- ๐ Fix context in multitask (#1040)
- ๐ Fix regression in ensemble trainer caused by recent fp16 change (#1033)
- ReadTheDocs OOM fix with CPU Torch (#1027)
- Dimension mismatch after setting max sequence length (#1154)
- ๐ Allow null learning rate (#1156)
- Don't fail on 0 input (#1104)
- โ Remove side effect during pickling PickleableGPT2BPEEncoder
- ๐ Set onnx==1.5.0 to fix CircleCI build temporarily (#1014)
- Complete training loop gracefully even if no timing is reported (#1128)
- Propagate min_freq for vocab correctly (#907)
- ๐ Fix gen-default-config with Model param (#917)
- ๐ Fix torchscript export for PyText modules (#1125)
- ๐ Fix label_weights in DocModel (#1081)
- ๐ Fix label_weights in bert models (#1100)
- ๐ Fix config issues with Python 3.7 (#1066)
- Temporary fix for Fairseq dependency (#1026)
- ๐ Fix MultipleData by making tensorizers able to initialize from multiple data sources (#972)
- ๐ Fix bug in copy_unk (#964)
- Division by Zero bug in MLM Metric Reporter (#968)