Repository navigation
Run big llm-models like Qwen3.8-Flash-Next in q3 or iq4 like on strata/poystrata on 48GB+16GBVRam #16460
Unanswered
RobAsticles
asked this question in
Q&A
Run big llm-models like Qwen3.8-Flash-Next in q3 or iq4 like on strata/poystrata on 48GB+16GBVRam
#16460
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
In strata or polystrata it is quite easy to run Swift-Qwen3.8-Flash-Next - IQ3_XXS, qwen3.8-flash-next-coder-iq1_m or Qwen3.8 Gyro-S (strata only) beween 25-31 t/s. Even Qwen3.8GSQ-RCO-abliterated-IQ2_XS runs in llama.cpp with 25.15 t/s.
Unsloth can run these models (except Gyro-S, missing codec) between 5-14 t/s.
I havent found any settingt to increase the speed.
Am i missing something? Is there a way to run these bigger models?
Would it be possible (suggestion) to add the magic of polystrata to the unsloth code?
All reactions