Rendered at 12:39:36 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
viraptor 4 hours ago [-]
That's an interesting approach, but... what's the performance impact? It seems the check was done only against really trivial tasks. Why not run some known coding benchmark against the original and pruned one?
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
aetherspawn 3 hours ago [-]
Does it improve tps for Mac?
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
hxii 4 hours ago [-]
Can the same process be theoretically repeated with LFM2.5-24B-A2B ?
tamnv 5 hours ago [-]
I really like topics about optimization like this one. Great job, OP!
peter_d_sherman 7 hours ago [-]
Looks absolutely stellar!
Another step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
Another step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!