Effect of fixed training sequence lengths on generalization and state abstraction
Ascertain whether training Kinaema and related recurrent sequence models with constant sequence lengths causes the models to conflate temporal state updates with abstraction layering, thereby hindering generalization to longer sequences, and determine the mechanisms responsible for this behavior.
References
We found that training with constant lengths hindered generalization to longer sequences. We conjecture that it leads to models confusing the notion of 'state' (in a control theory sense) with 'layer of abstraction', i.e using recurrent updates not only to push representations forward in time, but also to make changes in abstraction levels as a neural network would do between layers.
As a result, similarity computed from segment-level representations may not fully reflect similarity at the full-series level. More generally, the segmentation strategy itself is fixed rather than optimized, and we do not yet study how to choose segment boundaries or segment lengths so as to minimize retrieval distortion.