Yeah, this thing is unusual. It resembles a consumer GPU but using normal DRAM with more channels, so then you get more RAM but less memory bandwidth than GDDR/HBM, which is the same thing Apple does. And the TDP is pretty attractive for what it is.
But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?
The Intel D270, DGX Spark, and (somewhat) Strix Halo use the same approach as well so it seems to be getting pretty common.
SRAM wise it doesn't seem too insane. About the same ratio of memory:SRAM as the 5090 (IIRC) but still more than I would have expected. I assume they found it helps process the current layer faster or something.
Ok so it’s significantly worse than Apple M6 and we don’t know if it’s cheaper or not
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Aren't these specs already outdated? Seems meh. I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.
Give me 1TB RAM and 5TB/s or something that exciting.
For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.
It may be a semantic thing but I don't think bandwidth pooling works like that? Unless your blocks are distributed perfectly between cards there's going to be data passed between each card and from VRAM to compute. Correct me if I'm wrong though
I think the way it works is that the weights have to get streamed through the compute and the weights can be split very easily. With multiple cards you do have a coordination/communication phase after each layer of weights but that's usually much less significant than the compute phase.
But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?
SRAM wise it doesn't seem too insane. About the same ratio of memory:SRAM as the 5090 (IIRC) but still more than I would have expected. I assume they found it helps process the current layer faster or something.
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.
Give me 1TB RAM and 5TB/s or something that exciting.
Plenty of businesses want in local inference and are priced out.