---
base_model: NousResearch/Hermes-4-70B
base_model_relation: quantized
quantized_by: ArtusDev
language:
- en
library_name: transformers
license: llama3
pipeline_tag: text-generation
tags:
- Llama-3.1
- instruct
- finetune
- reasoning
- hybrid-mode
- chatml
- function calling
- tool use
- json mode
- structured outputs
- atropos
- dataforge
- long context
- roleplaying
- chat
- exl3
---
<style>
  .container-dark {
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
    line-height: 1.6;
    color: #d4d4d4;
  }
  a {
    color: #569cd6;
    text-decoration: none;
    font-weight: 600;
  }
  a:hover {
    text-decoration: underline;
  }
  .card-dark {
    background-color: #252526;
    border-radius: 12px;
    padding: 24px;
    margin-bottom: 20px;
    box-shadow: 0 4px 12px rgba(0,0,0,0.3);
    border: 1px solid #3c3c3c;
  }
  .card-dark h1 {
    font-size: 2.2em;
    color: #ffffff;
    text-align: center;
    margin-bottom: 10px;
  }
  .card-dark.card-dark-title h1 {
    font-size: 1.5em;
  }
  .card-dark .subtitle {
    text-align: center;
    font-size: 1.1em;
    color: #a0a0a0;
  }
  .card-dark h2 {
    font-size: 1.5em;
    margin-top: 0;
    padding-bottom: 10px;
    border-bottom: 1px solid #3c3c3c;
    color: #c586c0;
  }
  .card-dark h3 {
    font-size: 1.2em;
    color: #d4d4d4;
  }
  .styled-table {
    display: table;
    border: none;
    width: 100%;
    font-size: 0.95em;
    margin-bottom: 0px;
  }
  .styled-table thead th {
    background-color: #333333;
    color: #c586c0;
    text-align: left;
  }
  .styled-table th {
    padding: 12px 15px;
  }
  .styled-table td {
    padding: 0;
  }
  .styled-table table, .styled-table th, .styled-table td {
    border-left: none;
    border-right: none;
    border-bottom: none;
  }
  .styled-table td {
    border-bottom: 1px solid #3c3c3c;
  }
  .styled-table tbody tr {
    transition: background-color 0.1s ease;
  }
  .styled-table tbody tr:hover {
    background-color: #3a3a3a;
  }
  .styled-table tr:last-child td {
    border-bottom: none;
  }
  .styled-table td a {
    display: block;
    padding: 12px 15px; 
  }
  .styled-table td a.fake-link {
    text-decoration:none;
    color:inherit;
  }
  details {
    margin-top: 20px;
    border: 1px solid #3c3c3c;
    border-radius: 8px;
    overflow: hidden;
  }
  summary {
    cursor: pointer;
    padding: 12px 18px;
    background-color: #6A5ACD;
    font-weight: 600;
    display: flex;
    align-items: center;
    gap: 10px;
    justify-content: space-between;
    list-style: none;
  }
  summary::-webkit-details-marker {
    display: none;
  }
  summary:hover {
    filter: brightness(1.1);
  }
  summary::after {
    content: '';
    display: inline-block;
    width: 8px;
    height: 8px;
    border-bottom: 2px solid white;
    border-right: 2px solid white;
    transform: rotate(45deg);
    transition: transform 0.3s ease;
  }
  details[open] > summary::after {
    transform: rotate(225deg);
  }
  .details-content {
    padding: 18px;
  }
  .btn-purple {
    display: inline-block;
    background-color: #6A5ACD;
    color: white !important;
    padding: 12px 24px;
    border-radius: 8px;
    text-decoration: none;
    font-weight: 600;
    transition: background-color 0.3s ease, transform 0.2s ease;
    text-align: center;
  }
  .btn-purple:hover {
    background-color: #7B68EE;
    transform: translateY(-2px);
  }
</style>

<div class="container-dark">

  <div class="card-dark card-dark-title">
    <h1>ArtusDev/NousResearch_Hermes-4-70B-EXL3</h1>
    <p class="subtitle">
      EXL3 quants of <a href="https://huggingface.co/NousResearch/Hermes-4-70B" target="_blank">NousResearch/Hermes-4-70B</a> using <a href="https://github.com/turboderp-org/exllamav3/" target="_blank">exllamav3</a> for quantization.
    </p>
  </div>

  <div class="card-dark">
    <h2>Quants</h2>
    <table class="styled-table">
      <thead>
        <tr>
          <th>Quant</th>
          <th>BPW</th>
          <th>Head Bits</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/2.5bpw_H6" target="_blank">2.5_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/2.5bpw_H6" target="_blank">2.5</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/2.5bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/3.0bpw_H6" target="_blank">3.0_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/3.0bpw_H6" target="_blank">3.0</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/3.0bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/3.5bpw_H6" target="_blank">3.5_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/3.5bpw_H6" target="_blank">3.5</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/3.5bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/4.0bpw_H6" target="_blank">4.0_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/4.0bpw_H6" target="_blank">4.0</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/4.0bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/4.25bpw_H6" target="_blank">4.25_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/4.25bpw_H6" target="_blank">4.25</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/4.25bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/5.0bpw_H6" target="_blank">5.0_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/5.0bpw_H6" target="_blank">5.0</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/5.0bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/6.0bpw_H6" target="_blank">6.0_H6</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/6.0bpw_H6" target="_blank">6.0</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/6.0bpw_H6" target="_blank">6</a></td>
        </tr>
        <tr>
          <td><a href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/8.0bpw_H8" target="_blank">8.0_H8</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/8.0bpw_H8" target="_blank">8.0</a></td>
          <td><a class="fake-link" href="https://huggingface.co/ArtusDev/NousResearch_Hermes-4-70B-EXL3/tree/8.0bpw_H8" target="_blank">8</a></td>
        </tr>
      </tbody>
    </table>
  </div>

  <div class="card-dark">
    <h2>How to Download and Use Quants</h2>
    <p>You can download quants by targeting specific size using the Hugging Face CLI.</p>
    <details>
      <summary>Click for download commands</summary>
      <div class="details-content">
        <b>1. Install huggingface-cli:</b>
         <pre><code>pip install -U "huggingface_hub[cli]"</code></pre>
        <b>2. Download a specific quant:</b>
         <pre><code>huggingface-cli download ArtusDev/NousResearch_Hermes-4-70B-EXL3 --revision "5.0bpw_H6" --local-dir ./</code></pre>
      </div>
    </details>
    <p>EXL3 quants can be run with any inference client that supports EXL3, such as <a href="https://github.com/theroyallab/tabbyapi" target="_blank"><b>TabbyAPI</b></a>. Refer to <a href="https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started" target="_blank">documentation</a> for set up instructions.</p>
  </div>

  <div class="card-dark">
    <h2>Quant Requests</h2>
    <div style="text-align: center; margin-top: 25px;">
      <a href="https://huggingface.co/ArtusDev/requests-exl/discussions/new?title=[MODEL_NAME_HERE]&description=[MODEL_HF_LINK_HERE]" class="btn-purple" target="_blank">Request EXL3 Quants</a>
    </div>
    <p class="subtitle">
      See <a href="https://huggingface.co/ArtusDev/requests-exl" target="_blank">EXL community hub</a> for request guidelines.
    </p>
  </div>

</div>