ON-LSTM
Insprition under the hood: How to introduce grammar tree structure into LSTM in an unsupervised apporach.
Introduction: Ordered Neurons(ON)
-
The neurons inside ON-LSTM are specifically
orderedtoexpress richer information: Change the order of update frequency. -
The specific order of neurons is to integrate the hierarchical structure (tree structure) into the LSTM, allowing the LSTM to
automatically learn the hierarchical structure. -
High/Low level information: Should keep longer/shorter in corresponding coding interval. -
cumax(): A special function to internate special $F1$ and $F2$gate.
