RNNs made everything look like a sequence, so it became easy to forget that many problems — sorting numbers, modeling a set of variables, encoding an unordered collection — have no natural order at all. This paper's quietly subversive claim is that the moment you force such data into a seq2seq model, you smuggle in an ordering, and that arbitrary choice measurably changes what the model can learn. Order was never neutral; it was a hidden hyperparameter nobody was tuning.
Key Findings
- Order is a learnable variable, not a given. The same set fed in different orders yields different model quality, on real tasks like language modeling and parsing as well as toy ones like sorting — so "the data is unordered" does not let you off the hook.
- Read-Process-Write handles sets directly. Instead of pretending a set is a sequence, an attention-based encoder reads all elements, processes them with a permutation-invariant step, then writes the output — an early, principled answer to "how do you input a set?"
- Train by searching over output orders. When outputs are a set, the proposed loss searches over possible orderings during training rather than committing to one, addressing the structural mismatch at its source.
- A lineage, not a one-off. The framing seeded later permutation-invariant work (Deep Sets, Set Transformer) and connects to Pointer Networks and the attention mechanisms that the same group pushed forward.
Great Fit / When to Skip
Great fit if you want the origin of treating ordering as a first-class modeling decision, or to understand why set-structured inputs and outputs need more than a vanilla seq2seq. Look elsewhere if you need a ready-made permutation-invariant layer to drop into code — that engineering matured in the follow-up architectures; this is the conceptual groundwork, with experiments that are illustrative rather than large-scale.
