| [InCoder](https://huggingface.co/facebook/incoder-6B) is a code generation model that also allows code editing via [infilling](https://arxiv.org/pdf/2204.05999.pdf). It was trained on **216 GB** of preprocessed data from GitHub and Stack Overflow from 28 programming languages. 52 GB is in Python, 107GB in other programming languages and 57GB is content from Stackoverflow that isn't code. |
|
|
| The GitHub data was cleaned with the following steps: |
| - Average line length < 100 tokens |
| - Maximum line length < 3000 MB |
| - Alphanumeric characters fraction > 0.4 |
| - Remove auto-generated files (keyword search) |
|
|
| The second component of the data consists of questions, answers, and comments from Stack Overflow. It includes: |
| - all questions that have at least one answer |
| - up to ten answers with a non-negative score (sorted by score) per question |
| - up to five comments per question/answer |
|
|
| Exact match deduplication was performed on code files. For more details please refer to this [paper](https://arxiv.org/pdf/2204.05999.pdf). |
