58 points by zerop 7 hours ago
Comments are read-only. Visit Hacker News to participate in the discussion.
kooi 1 hour ago | 0 points
Cool, I like state space models because they "feel" like they're doing proper homage to the power of control and modeling for linear dynamical systems. (Mamba is a bunch of time varying linear systems chained by non-linear activations).<p>I'm reading one of the OG papers now and here are some interesting snip-bits. I wonder how much of inference in data centers is done with traditional transformers vs SSM, because the efficiency gains are quite dramatic.<p>> We propose a new class of selective state space models, that improves on prior work on several axes to achieve the modeling power of Transformers while scaling linearly in sequence length.<p>> Our Mamba language model has 5x generation throughput compared to Transformers of similar size, and Mamba-3B’s quality matches that of Transformers twice its size.<p>Then, interestingly for training:<p>> Commonly, the model[SSM] uses the convolutional mode (3) for efficient parallelizable training (where the whole input sequence is seen ahead of time), and switched into recurrent mode (2) for efficient autoregressive inference (where the inputs are seen one timestep at a time).<p><a href="https://arxiv.org/html/2312.00752v2" rel="nofollow">https://arxiv.org/html/2312.00752v2</a>
buildbot 3 hours ago | 0 points
In my opinion, a better & more comprehensive look at a similar idea for all state space attention types is this paper from AWS: <a href="https://arxiv.org/abs/2605.08301" rel="nofollow">https://arxiv.org/abs/2605.08301</a>
thisisauserid 4 hours ago | 0 points
I totally get why people want to reinvent LLMs so they're better/cheaper/faster and this hybrid approach with State Space Models is likely going to bear fruit.<p>But I'm WAY more interested in what's going to happen with Mamba-3, et al. when it's relatively easy to predict, ya know, <i>state</i>.