Hacker News Clone

random3Jan 25, 2026, 6:40 AM

David Patterson is such a legend! From RAID to RISC and one of the best books in computer architecture, he's on my personal hall of fame.

Several years ago I was at one of the Berkley AMP Lab retreats at Asilomar, and as I was hanging out, I couldn't figure how I know this person in front of me, until an hour later when I saw his name during a panel :)).

It was always the network. And David Patterson, after RISC, started working on iRAM, that was tackling a related problem.

NVIDIA bought Mellanox/Infiniband, but Google has historically excelled at networking, and the TPU seems to be designed to scale out in the best possible way.

zozbot234Jan 25, 2026, 12:38 PM

Weird to see no mention in this paper of persistent memory technologies beyond NAND flash. Some of them, like ReRAM, also enable compute-in-memory which the authors regard as quite important.

HPsquaredJan 25, 2026, 5:00 PM

Why not, instead of passing the entire model through a processor and running it on every bit of data, pass the data (which is much smaller) through the model? As in, have compute and memory together in the silicon. Then you only need to shuffle the data itself around (perhaps by broadcast) rather than the entire model. That seems like it would use a LOT less energy.

Or is it not possible to make the algorithms parallel to this degree?

Edit: apparently this is called "compute-in-memory"

jmalickiJan 25, 2026, 6:08 PM

This is done that way at the GPU layer of abstraction - generally (with some exceptions!) the model lives in GPU vram, and you stream the data batch by batch through the model.

The problem is that for larger models the model barely fits in VRAM, so it definitely doesn't fit in cache.

Dataflow processors like cerebras do stream the data through the model (for smaller models at least, or if they can have smaller portions of models) - each little core has local memory and you move the data to where it needs to go. To achieve this though, Cerebras has 96GB of what is basically L1 cache among its cores, which is... a lot of SRAM.

pavpanchekhaJan 25, 2026, 5:23 PM

Frontier models are now much bigger than an individual query, hence batching, MoE, etc. So this idea, while very plausible, has economic constraints, you'd need vast amounts of memory.

westurnerJan 25, 2026, 10:51 PM

Designing a concept sustainable RAM product and in working around multiplexing scaling challenges I somewhat accidentally developed a potential solution for hosting already-trained LLMs with very low energy and hardware in carbon and lignin;

> You have effectively designed a Diffractive Deep Neural Network (D^2NN) that doubles as a storage device.

Mode Division Multiplexing (MDM) via OAM Solitons potentially with gratings designed with Inverse Design of a Transition Map to be lasered possibly with a Galvo Laser. This would be a very low power way to run LLMs; on a lasered substrate

westurnerJan 25, 2026, 10:50 PM

In-memory processing: https://en.wikipedia.org/wiki/In-memory_processing

Computational RAM: https://en.wikipedia.org/wiki/Computational_RAM

bluehat974Jan 25, 2026, 6:35 AM

random_duckJan 25, 2026, 3:39 PM

Yup, reads like the executive summary (in a good way).

jauntywundrkindJan 25, 2026, 5:41 AM

> To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication.

High Bandwidth Flash (HBF) got submitted 6 hours ago! It's a great article, fantastic coverage of a wide section of the rapidly moving industry. https://news.ycombinator.com/item?id=46700384 https://blocksandfiles.com/2026/01/19/a-window-into-hbf-prog...

HBF is about having many dozens or hundreds of channels of flash memory. The idea of having Processing Near HBF, spread out, perhaps in mixed 3d design, would be not at all surprising to me. One of the main challenges for HBF is building improved vias, improved stacking, and if that tech advanced the idea of more mixed NAND and compute layers rather than just NAND stacks perhaps opens up too.

This is all really exciting possible next steps.

ameliusJan 25, 2026, 12:00 PM

Why is persistence such a big thing here? Non-flash memory just needs a tiny bit of power to keep its data. I don't see the revolutionary usecase.

GracanaJan 25, 2026, 1:54 PM

Density is the key here, not persistence.

ameliusJan 25, 2026, 2:03 PM

Thanks! This explains it.

Now I'm wondering how you deal with the limited number of write cycles of Flash memory. Or maybe that is not an issue in some applications?

mrobJan 25, 2026, 3:25 PM

During inference, most of the memory is read only.

ameliusJan 25, 2026, 3:44 PM

Sounds fair. That's not the kind of machine I'd want as a development system though. And usually development systems are beefier than production systems. So curious how they'd solve that.

GracanaJan 25, 2026, 4:06 PM

Yeah, it is quite specialized for inference. It's unlikely that you'd see this stuff outside of hardware specifically for that.

Development systems for AI inference tend to be smaller by necessity. A DGX Spark, Station, a single B300 node... you'd work on something like that before deploying to a larger cluster. There's just nothing bigger than what you'd actually deploy to.

transputeJan 25, 2026, 4:07 PM

HBF, like expensive HBM, is targeted at AI data centers.

  The KAIST professor discussed an HBF unit having a capacity of 512 GB and a 1.638 TBps bandwidth.

PCIe x8 GPU bandwidth is about 32GBbps, so HBF could be 50x PCIe bandwidth.

suggeststrongidJan 25, 2026, 10:58 AM

Can’t we credit the first author in the title too? Come on.

transputeJan 25, 2026, 3:08 PM

The current title uses 79 characters of 80 character budget:

  75% = title written by first author
  22% = name of second author, endorsing work of first author

HN mods can revert the title to the original headline, without any author.

random_duckJan 25, 2026, 2:34 PM

No we can't, that would be a crime against royalty :)

ameliusJan 25, 2026, 11:51 AM

That appendix of memory prices looks interesting, but misses the recent trend.

Challenges and Research Directions for Large Language Model Inference Hardware

Comments