ServiceNow-AI/Apriel-H1-15b-Thinker-SFT
Text Generation
•
16B
•
Updated
•
624
•
27
Mike Ravkine PRO
mike-ravkine
AI & ML interests
LLM Research / Development / Evaluation
Recent Activity
liked
a model
about 15 hours ago
ServiceNow-AI/Apriel-H1-15b-Thinker-SFT
liked
a model
about 15 hours ago
ServiceNow-AI/Apriel-1.6-15b-Thinker
liked
a model
about 15 hours ago
arcee-ai/Trinity-Mini
Organizations
None yet
liked
4
models
about 15 hours ago
liked
a
model
8 days ago
liked
a
model
13 days ago
liked
a
model
14 days ago
posted
an
update
19 days ago
Post
1074
Applying Hazard and Entropy Analysis to LLMs
Here's an example of a model that behaves perfectly well up to 8k, smoothly increasing its entropy before going into a struggle zone, collapsing, seeing a region of recovery and finally falling down hard at the 16k wall.
Is your model implementation behaving badly like this?
Would you know if it was? 👀
Here's an example of a model that behaves perfectly well up to 8k, smoothly increasing its entropy before going into a struggle zone, collapsing, seeing a region of recovery and finally falling down hard at the 16k wall.
Is your model implementation behaving badly like this?
Would you know if it was? 👀
liked
a
model
23 days ago
liked
a
model
24 days ago
liked
a
model
30 days ago
posted
an
update
about 1 month ago
Post
285
sunday = lab day
goal: understand how GGUF compression works - what exactly is being lost?
approach: quantize/dequantize some images and look at error maps
spent 80% of the time tracing down what turns out to be a data distribution assumption: real LLM weights are symmetric and their mean is 0 so our test image MUST retain these properties or the results turn into a kind of nonsense soup where Q5_1 beats Q8
with that issue solved, we have some fun results! from left to right:
- test pattern image (mean value is around 0.01)
- q8 error (almost nothing - some light banding in the gradients)
- q5km error (starting to see the 'blocks' around the circles)
- q4_0 error (this is why q4_1 is 'preferred')
- q3k error. q3k is a really interesting set of trade-offs: it does not have a block-offset so it really leans into the 0-mean assumption HARD, if you violate it locally the results are BAD
- q2k error: q2k has a block-offset so for certain patterns the errors are actually less then q3k (a rather counter-intuitive result)
looking at mxfp4, i-quants and the other stuff that's possible inside gguf remains future work.. aiming to clean up this repo and push it this week, feel free to ping me if you want to play sooner.
goal: understand how GGUF compression works - what exactly is being lost?
approach: quantize/dequantize some images and look at error maps
spent 80% of the time tracing down what turns out to be a data distribution assumption: real LLM weights are symmetric and their mean is 0 so our test image MUST retain these properties or the results turn into a kind of nonsense soup where Q5_1 beats Q8
with that issue solved, we have some fun results! from left to right:
- test pattern image (mean value is around 0.01)
- q8 error (almost nothing - some light banding in the gradients)
- q5km error (starting to see the 'blocks' around the circles)
- q4_0 error (this is why q4_1 is 'preferred')
- q3k error. q3k is a really interesting set of trade-offs: it does not have a block-offset so it really leans into the 0-mean assumption HARD, if you violate it locally the results are BAD
- q2k error: q2k has a block-offset so for certain patterns the errors are actually less then q3k (a rather counter-intuitive result)
looking at mxfp4, i-quants and the other stuff that's possible inside gguf remains future work.. aiming to clean up this repo and push it this week, feel free to ping me if you want to play sooner.
reacted to
mitkox's
post with 🔥
about 1 month ago
Post
4134
I just threw Qwen3-0.6B in BF16 into an on device AI drag race on AMD Strix Halo with vLLM:
564 tokens/sec on short 100-token sprints
96 tokens/sec on 8K-token marathons
TL;DR You don't just run AI on AMD. You negotiate with it.
The hardware absolutely delivers. Spoiler alert; there is exactly ONE configuration where vLLM + ROCm + Triton + PyTorch + Drivers + Ubuntu Kernel to work at the same time. Finding it required the patience of a saint
Consumer AMD for AI inference is the ultimate "budget warrior" play, insane performance-per-euro, but you need hardcore technical skills that would make a senior sysadmin nod in quiet respect.
564 tokens/sec on short 100-token sprints
96 tokens/sec on 8K-token marathons
TL;DR You don't just run AI on AMD. You negotiate with it.
The hardware absolutely delivers. Spoiler alert; there is exactly ONE configuration where vLLM + ROCm + Triton + PyTorch + Drivers + Ubuntu Kernel to work at the same time. Finding it required the patience of a saint
Consumer AMD for AI inference is the ultimate "budget warrior" play, insane performance-per-euro, but you need hardcore technical skills that would make a senior sysadmin nod in quiet respect.
- 1 reply
reacted to
danielhanchen's
post with ❤️
about 1 month ago
Post
4224
You can now run Kimi K2 Thinking locally with our Dynamic 1-bit GGUFs:
unsloth/Kimi-K2-Thinking-GGUF
We shrank the 1T model to 245GB (-62%) & retained ~85% of accuracy on Aider Polyglot. Run on >247GB RAM for fast inference.
We also collaborated with the Moonshot AI Kimi team on a system prompt fix! 🥰
Guide + fix details: https://docs.unsloth.ai/models/kimi-k2-thinking-how-to-run-locally
We shrank the 1T model to 245GB (-62%) & retained ~85% of accuracy on Aider Polyglot. Run on >247GB RAM for fast inference.
We also collaborated with the Moonshot AI Kimi team on a system prompt fix! 🥰
Guide + fix details: https://docs.unsloth.ai/models/kimi-k2-thinking-how-to-run-locally
replied to
their
post
about 1 month ago
The way the coolers on these cards are designed is VERY unusual - this photo should come in the box! If you're having trouble in a closed case, see if you have space to add an intake on the bottom beside the PCIe edge - if all air is coming from the front like in this pic, the rear blower steals it all!
replied to
their
post
about 1 month ago
I am running 3x140mm 89CFM as intakes on the front and directly underneath the in-blowers of those two 3090FE is the main trick - a dedicated 140mm 140CFM blowing upwards to feed the intake blowers at the front/pcie edge of these coolers. The cross-blower air passes through both cards and then has space to vent their heat on the right.
At 280W load the 'outside' card is bored and sitting at 50C with the blower fans at 50% while the 'inside' card maintains 60-65C with it's blowers closer to 70-80%.
The main thing to be careful about with the FE dual-blower coolers is do not try to pump air into the 'front' (pcie side) of them! I see this configuration on many mining rigs and its fine for air cooled cards but the FEs actually have a blower venting out this pci-slot side so you HAVE to feed them from either the rear or underneath.
When I had the 4-slot bridge installed across them, the rear-feed alone was sufficient.
posted
an
update
about 1 month ago
Post
1442
measuring the information content of a reasoning trace seems like a straightforward reasoning LLM KPI, but how can we achieve this?
what if we keep it simple: gzip the resulting text and take the length of the compressed stream... "compressed bytes of information per output token" becomes the KPI
if we split across correct answers vs incorrect answers vs truncated answers and group by difficulty, a whole new world of analysis becomes not just possible but visually intuitive and almost trivial:
1) what is the model's overall reasoning efficiency? this is the slope of the scatterplot curve segments (there may be more then one..)
2) is the model able to apply more test-time compute towards more difficult variations of the task? the two on the left are not, the two on the right are.
3) when applying more test-time compute, is that compute useful? this is the curvature of the scatterplot trends - the two in the middle are 'losing their mojo' as answers get longer the information content falls down
4) is the model applying multiple approaches to the task? (right) do those approaches change with difficulty?
5) are truncations because we don't have enough context budget (left) or because the model has lost its mind and gone into a repeat loop (middle two) and does this happen across the board (middle left) or only when the problem is more difficult (middle right)
would love to hear your guys feedback on this kind of analysis, is anyone doing similar work?
this approach generates 12 plots per model (one for each task) so quite a bit of data and i've been hesitant to publish it so far, consider this post a toe tip.
what if we keep it simple: gzip the resulting text and take the length of the compressed stream... "compressed bytes of information per output token" becomes the KPI
if we split across correct answers vs incorrect answers vs truncated answers and group by difficulty, a whole new world of analysis becomes not just possible but visually intuitive and almost trivial:
1) what is the model's overall reasoning efficiency? this is the slope of the scatterplot curve segments (there may be more then one..)
2) is the model able to apply more test-time compute towards more difficult variations of the task? the two on the left are not, the two on the right are.
3) when applying more test-time compute, is that compute useful? this is the curvature of the scatterplot trends - the two in the middle are 'losing their mojo' as answers get longer the information content falls down
4) is the model applying multiple approaches to the task? (right) do those approaches change with difficulty?
5) are truncations because we don't have enough context budget (left) or because the model has lost its mind and gone into a repeat loop (middle two) and does this happen across the board (middle left) or only when the problem is more difficult (middle right)
would love to hear your guys feedback on this kind of analysis, is anyone doing similar work?
this approach generates 12 plots per model (one for each task) so quite a bit of data and i've been hesitant to publish it so far, consider this post a toe tip.
replied to
their
post
about 1 month ago
The top high density rack is successfully hitting 65C targets at 1.15kW TDP.
Really impressive coolers on the FEs, there is barely 10mm of space but the sandwiched one still keeps up to unobstructed air coolers.
We are back, baby 🏇
posted
an
update
about 1 month ago
Post
2768
There is no anxiety quite like powering up 2KW of basement compute after rewiring it all. Small bit of trouble with the horizontal 3090 because I misread my motherboard manual, but otherwise so far so good.. Next we see if I've built up enough cooling to hit my target TDP on those 3-slot nvlinked cards especially. The 4-slot bridges are much easier to work with but their prices went bananas and I couldn't acquire a second, so gotta get a little creative with intakes.
-
- 5 replies
liked
a
model
about 1 month ago