Comparison of ON-LSTM and DIORA

ON-LSTM

Insprition under the hood: How to introduce grammar tree structure into LSTM in an unsupervised apporach.

Introduction: Ordered Neurons(ON)

1. The neurons inside ON-LSTM are specifically ordered to express richer information: Change the order of update frequency.
2. The specific order of neurons is to integrate the hierarchical structure (tree structure) into the LSTM, allowing the LSTM to automatically learn the hierarchical structure.
3. High/Low level information: Should keep longer/shorter in corresponding coding interval.
4. cumax(): A special function to internate special $F1$ and $F2$gate.

The nuts and bolts in Mathematic

Reference

1. ON-LSTM：用有序神经元表达层次结构