Uploaded model, tokenizer and readme.

README.md

@@ -1,3 +1,116 @@
 ---
-license: mit
+language:
+- pt
+tags:
+- GlórIA
+- European Portuguese
+- gptneo
+- decoder
+- foundation model
+- text-generation
+datasets:
+- europarl_bilingual
+- assin2
+- dlb/plue
+- oscar-corpus/OSCAR-2301
+- PORTULAN/glue-ptpt
+widget:
+- text: A culinária portuguesa é rica em aromas e
+- text: Os computadores hoje em dia são muito
+- text: A literatura Portuguesa é
+inference:
+  parameters:
+    temperature: 1
+    repetition_penalty: 2
+    max_new_tokens: 30
+    num_beams: 4
+    do_sample: true
+    top_k: 50
+library_name: transformers
 ---
+
+# GlórIA 1.3B
+
+## Model Description
+**GlórIA** is a large generative language model, with special **focus on European Portuguese**. 
+
+It is a 1.3B parameters model, based on [GPTNeo](https://huggingface.co/EleutherAI/gpt-neo-1.3B), which has 24 layers and a hidden size of 2048.
+
+## Training Data
+**GlórIA 1.3B** was trained on a large corpora, with approximately 35B tokens. This corpus was built by gathering multiple Portuguese sources:
+- [ArquivoPT News PT-PT Dataset](): A collection of 1.4M European Portuguese archived news and periodicals from [Arquivo.pt](https://arquivo.pt/).
+- [ClueWeb-Large PT-PT](https://lemurproject.org/clueweb22.php/): Multilingual Corpus, similar to OSCAR. Metadata was used to filter only PT-PT webpages.
+- [Europarl PT-PT](https://www.statmt.org/europarl/): A parallel corpus with documents such as transcripts from the European Parliament (we only used the PT-PT documents).
+- [OpenSubtitles PT-PT](https://opus.nlpl.eu/OpenSubtitles.php): A corpus containing PT-PT subtitles from [OpenSubtitles](http://www.opensubtitles.org/).
+- [OSCAR PT-PT](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201): Multilingual Corpus obtained from filtering the Common Crawl corpus. We used metadata to filter only PT-PT webpages.
+- [PT WIKI](): The Portuguese Wikipedia. 2022/06/20 Dump.
+
+<br>
+
+## Evaluation - CALAME-PT
+GlórIA 1.3B generative capabilities were evaluated on **CALAME-PT** - a new Portuguese benchmark with the goal of predicting the last word of a sentence, according to a given context.
+
+| Model and Size   | Exact-Match   |
+| ---------------- | ----------    | 
+| Gervasio-PTPT (1B)    | 44.01         | 
+| mGPT (1.3B)            | 47.14         |
+| GlórIA (1.3B)     | 52.79         | 
+
+
+<br>
+
+# How to use
+## Basic Inference Example
+```py
+>>> from transformers import pipeline
+>>> generator = pipeline('text-generation', model='NOVA-vision-language/GlorIA-1.3B-original')
+>>> generator("A culinária portuguesa é rica em aromas e", do_sample=True, min_length=50)
+[{'generated_text': 'A culinária portuguesa é rica em aromas e'}]
+```
+## Recommended Parameters and Usage (for more flexibility)
+```py
+from transformers import GenerationConfig, TextGenerationPipeline
+generation_config = GenerationConfig(
+        max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id,
+        no_repeat_ngram_size=0, num_beams=4, repetition_penalty=2.0, temperature=1.0,
+        output_scores=True, early_stopping=True
+)
+generator = TextGenerationPipeline(model=model, task="text-generation",
+                                    tokenizer=loaded_tokenizer, device=0)
+completion_prompts = ["Fernando Pessoa foi um dos poetas mais relevantes de"]
+out = generator(completion_prompts, generation_config=generation_config)
+[[{'generated_text': 'Fernando Pessoa foi um dos poetas mais relevantes de toda a literatura portuguesa, autor de uma obra que se estende por mais de quatro dezenas de livros, entre os quais "Mensagem", "O Guardador de Rebanhos", "Livro do desassossego", "Odes",'}]]
+```
+
+<br>
+
+
+# Citation
+
+
+Please use the following BibTeX to cite our paper:
+```
+@InProceedings{gloria_ptpt_propor2024,
+  author="Lopes, Ricardo
+          and Magalhães, João
+          and Semedo, David",
+  title="GlórIA: A Generative and Open Large Language Model for Portuguese",
+  booktitle="Computational Processing of the Portuguese Language (PROPOR 2024)",
+  year="2024",
+}
+```
+
+**License**: GlórIA's usage is restricted to research-only purposes, subject to the ClueWeb22 Dataset license, which can be freely obtained [here](https://www.lemurproject.org/clueweb22/obtain.php).
+
+
+# Acknowledgements
+
+We would like to thank Arquivo.pt's team for their content preservation efforts, and for all the help and guidance in accessing the archived web pages at scale.
+This work was partially funded by the FCT project NOVA LINCS Ref. UIDP/04516/2020, by CMU|Portugal project iFetch, Ref. CMUP LISBOA-01-0247-FEDER-045920, and by the FCT project Ref. Nº CPCA-IAC/AV/594875/2023.
+
+<br>
+
+
+
+
+

config.json

@@ -0,0 +1,78 @@
+{
+  "_name_or_path": "D:\\FCT\\TESE\\ThesisTrainer\\model_hub",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPTNeoForCausalLM"
+  ],
+  "attention_dropout": 0,
+  "attention_layers": [
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local",
+    "global",
+    "local"
+  ],
+  "attention_types": [
+    [
+      [
+        "global",
+        "local"
+      ],
+      12
+    ]
+  ],
+  "bos_token_id": 50256,
+  "classifier_dropout": 0.1,
+  "do_sample": true,
+  "embed_dropout": 0,
+  "eos_token_id": 50256,
+  "gradient_checkpointing": false,
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": null,
+  "layer_norm_epsilon": 1e-05,
+  "max_length": 50,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neo",
+  "num_heads": 16,
+  "num_layers": 24,
+  "resid_dropout": 0,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50,
+      "temperature": 0.9
+    }
+  },
+  "temperature": 0.9,
+  "tokenizer_class": "GPT2Tokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.29.1",
+  "use_cache": true,
+  "vocab_size": 50258,
+  "window_size": 256
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "do_sample": true,
+  "eos_token_id": 50256,
+  "max_length": 50,
+  "temperature": 0.9,
+  "transformers_version": "4.29.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ff171448458ddb881d3095cc771f6d935004d12f807c304e5905f46a19fe492
+size 5363015080

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d067b12f2fe852c21aab4e4cef51ef7ddee1ff47896385dc077c9b1c784c2a93
+size 5363096533

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}