Hugging Face's logo
Dmitrii Kostakov's picture
On Vacation 🏝️
2 4 78

Dmitrii Kostakov PRO

kostakoff
smbl07's profile picture 1aihk's profile picture Fishtiks's profile picture
·

AI & ML interests

MLOps

Recent Activity

posted an update 4 days ago
Just for fun, let's run the Alibaba MNN benchmark on a DGX Spark! From time to time, I look for something new or unusual in the AI world, and recently I stumbled upon MNN — a direct competitor to llama.cpp. I found this project intriguing and set a small goal for myself: to run it on my DGX Spark. I was glad to see that MNN is open-source under the Apache 2.0 license, meaning I was free to fork and modify it. However, MNN had a few issues out of the box: - No support for CUDA 13.0 - No support for the Blackwell architecture sm_12 - No built-in support for CUDA benchmarking I tackled these issues one by one and successfully compiled MNN on the DGX Spark. The benchmark results are currently quite low, but at least it works! Patch file here https://github.com/alibaba/MNN/issues/4289#issuecomment-4093931887 Here is the step-by-step guide on how I built MNN: ```bash mkdir mnn && cd mnn # Get the code git clone https://github.com/alibaba/MNN.git cd MNN # Reset repo to a specific commit git reset --hard b1d06d68b3366183d157f0703d7b8a8b61ae55b3 # Apply patch for CUDA 13.0 git apply ../my_changes.patch mkdir build && cd build # Configure the project cmake .. \ -DMNN_CUDA=ON \ -DMNN_BUILD_LLM=ON \ -DMNN_SUPPORT_TRANSFORMER_FUSE=ON \ -DCMAKE_BUILD_TYPE=Release # Build libraries and executable binaries cmake --build . --config Release -j$(nproc) make -j$(nproc) ``` How to run the test: - Download the MNN model: https://huggingface.co/taobao-mnn/Qwen3-30B-A3B-MNN - Run the benchmark: ```bash ./MNN/build/llm_bench -m /path/to/qwen/config.json -a cuda -c 2 -p 512 -n 128 -kv true -rep 3 ``` It works! Hopefully, the MNN developers will add official CUDA 13 support. https://huggingface.co/llmlaba
liked a model 5 days ago
baidu/ERNIE-4.5-21B-A3B-Thinking
updated a collection 6 days ago
favorite
View all activity

Organizations

Blog-explorers's profile picture Home Large Language Models research laboratory's profile picture