Rentlar@lemmy.ca 1 day ago
It should be noted, how much will that affect the lifespan of those GPUs running double-dutyx8?
AI’s still replaceable but it will emulate human-like burnout.
Rentlar@lemmy.ca 1 day ago
It should be noted, how much will that affect the lifespan of those GPUs running double-dutyx8?
AI’s still replaceable but it will emulate human-like burnout.
BCOVertigo@lemmy.world 1 day ago
From their linked study:
“Filling this utilization gap requires us to better saturate each GPU by enabling it to serve requests from multiple models. As such, we aim to conduct effective GPU pooling. By sharing a GPU between as many models as possible with- out violating the service-level objective (SLO), GPU pooling promises great reductions in operational expenses (OPEX) for concurrent LLM serving.”
The “saturate each gpu” part seems to support your idea.
Rentlar@lemmy.ca 1 day ago
I do expect operational savings from this optimization, but my guesstimate would be a 2-5x savings rather than the reported 9x savings when looked at over a fixed time period.
BCOVertigo@lemmy.world 1 day ago
I mean if short term GPU supply is the bottleneck it will help that, but if it cooks the GPUs even faster and doesn’t result in less overall compute being done it just lets smaller players with less hardware compete with hyperscalers and may not actually be savings in the long term. The unit economics being negative are still the giant elephant in the room preparing to trample them.