| | --- |
| | tags: |
| | - mteb |
| | - Sentence Transformers |
| | - sentence-similarity |
| | - feature-extraction |
| | - sentence-transformers |
| | model-index: |
| | - name: multilingual-e5-large |
| | results: |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_counterfactual |
| | name: MTEB AmazonCounterfactualClassification (en) |
| | config: en |
| | split: test |
| | revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| | metrics: |
| | - type: accuracy |
| | value: 79.05970149253731 |
| | - type: ap |
| | value: 43.486574390835635 |
| | - type: f1 |
| | value: 73.32700092140148 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_counterfactual |
| | name: MTEB AmazonCounterfactualClassification (de) |
| | config: de |
| | split: test |
| | revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| | metrics: |
| | - type: accuracy |
| | value: 71.22055674518201 |
| | - type: ap |
| | value: 81.55756710830498 |
| | - type: f1 |
| | value: 69.28271787752661 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_counterfactual |
| | name: MTEB AmazonCounterfactualClassification (en-ext) |
| | config: en-ext |
| | split: test |
| | revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| | metrics: |
| | - type: accuracy |
| | value: 80.41979010494754 |
| | - type: ap |
| | value: 29.34879922376344 |
| | - type: f1 |
| | value: 67.62475449011278 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_counterfactual |
| | name: MTEB AmazonCounterfactualClassification (ja) |
| | config: ja |
| | split: test |
| | revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| | metrics: |
| | - type: accuracy |
| | value: 77.8372591006424 |
| | - type: ap |
| | value: 26.557560591210738 |
| | - type: f1 |
| | value: 64.96619417368707 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_polarity |
| | name: MTEB AmazonPolarityClassification |
| | config: default |
| | split: test |
| | revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
| | metrics: |
| | - type: accuracy |
| | value: 93.489875 |
| | - type: ap |
| | value: 90.98758636917603 |
| | - type: f1 |
| | value: 93.48554819717332 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (en) |
| | config: en |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 47.564 |
| | - type: f1 |
| | value: 46.75122173518047 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (de) |
| | config: de |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 45.400000000000006 |
| | - type: f1 |
| | value: 44.17195682400632 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (es) |
| | config: es |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 43.068 |
| | - type: f1 |
| | value: 42.38155696855596 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (fr) |
| | config: fr |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 41.89 |
| | - type: f1 |
| | value: 40.84407321682663 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (ja) |
| | config: ja |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 40.120000000000005 |
| | - type: f1 |
| | value: 39.522976223819114 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (zh) |
| | config: zh |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 38.832 |
| | - type: f1 |
| | value: 38.0392533394713 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: arguana |
| | name: MTEB ArguAna |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 30.725 |
| | - type: map_at_10 |
| | value: 46.055 |
| | - type: map_at_100 |
| | value: 46.900999999999996 |
| | - type: map_at_1000 |
| | value: 46.911 |
| | - type: map_at_3 |
| | value: 41.548 |
| | - type: map_at_5 |
| | value: 44.297 |
| | - type: mrr_at_1 |
| | value: 31.152 |
| | - type: mrr_at_10 |
| | value: 46.231 |
| | - type: mrr_at_100 |
| | value: 47.07 |
| | - type: mrr_at_1000 |
| | value: 47.08 |
| | - type: mrr_at_3 |
| | value: 41.738 |
| | - type: mrr_at_5 |
| | value: 44.468999999999994 |
| | - type: ndcg_at_1 |
| | value: 30.725 |
| | - type: ndcg_at_10 |
| | value: 54.379999999999995 |
| | - type: ndcg_at_100 |
| | value: 58.138 |
| | - type: ndcg_at_1000 |
| | value: 58.389 |
| | - type: ndcg_at_3 |
| | value: 45.156 |
| | - type: ndcg_at_5 |
| | value: 50.123 |
| | - type: precision_at_1 |
| | value: 30.725 |
| | - type: precision_at_10 |
| | value: 8.087 |
| | - type: precision_at_100 |
| | value: 0.9769999999999999 |
| | - type: precision_at_1000 |
| | value: 0.1 |
| | - type: precision_at_3 |
| | value: 18.54 |
| | - type: precision_at_5 |
| | value: 13.542000000000002 |
| | - type: recall_at_1 |
| | value: 30.725 |
| | - type: recall_at_10 |
| | value: 80.868 |
| | - type: recall_at_100 |
| | value: 97.653 |
| | - type: recall_at_1000 |
| | value: 99.57300000000001 |
| | - type: recall_at_3 |
| | value: 55.619 |
| | - type: recall_at_5 |
| | value: 67.71000000000001 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/arxiv-clustering-p2p |
| | name: MTEB ArxivClusteringP2P |
| | config: default |
| | split: test |
| | revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
| | metrics: |
| | - type: v_measure |
| | value: 44.30960650674069 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/arxiv-clustering-s2s |
| | name: MTEB ArxivClusteringS2S |
| | config: default |
| | split: test |
| | revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
| | metrics: |
| | - type: v_measure |
| | value: 38.427074197498996 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/askubuntudupquestions-reranking |
| | name: MTEB AskUbuntuDupQuestions |
| | config: default |
| | split: test |
| | revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
| | metrics: |
| | - type: map |
| | value: 60.28270056031872 |
| | - type: mrr |
| | value: 74.38332673789738 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/biosses-sts |
| | name: MTEB BIOSSES |
| | config: default |
| | split: test |
| | revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 84.05942144105269 |
| | - type: cos_sim_spearman |
| | value: 82.51212105850809 |
| | - type: euclidean_pearson |
| | value: 81.95639829909122 |
| | - type: euclidean_spearman |
| | value: 82.3717564144213 |
| | - type: manhattan_pearson |
| | value: 81.79273425468256 |
| | - type: manhattan_spearman |
| | value: 82.20066817871039 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/bucc-bitext-mining |
| | name: MTEB BUCC (de-en) |
| | config: de-en |
| | split: test |
| | revision: d51519689f32196a32af33b075a01d0e7c51e252 |
| | metrics: |
| | - type: accuracy |
| | value: 99.46764091858039 |
| | - type: f1 |
| | value: 99.37717466945023 |
| | - type: precision |
| | value: 99.33194154488518 |
| | - type: recall |
| | value: 99.46764091858039 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/bucc-bitext-mining |
| | name: MTEB BUCC (fr-en) |
| | config: fr-en |
| | split: test |
| | revision: d51519689f32196a32af33b075a01d0e7c51e252 |
| | metrics: |
| | - type: accuracy |
| | value: 98.29407880255337 |
| | - type: f1 |
| | value: 98.11248073959938 |
| | - type: precision |
| | value: 98.02443319392472 |
| | - type: recall |
| | value: 98.29407880255337 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/bucc-bitext-mining |
| | name: MTEB BUCC (ru-en) |
| | config: ru-en |
| | split: test |
| | revision: d51519689f32196a32af33b075a01d0e7c51e252 |
| | metrics: |
| | - type: accuracy |
| | value: 97.79009352268791 |
| | - type: f1 |
| | value: 97.5176076665512 |
| | - type: precision |
| | value: 97.38136473848286 |
| | - type: recall |
| | value: 97.79009352268791 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/bucc-bitext-mining |
| | name: MTEB BUCC (zh-en) |
| | config: zh-en |
| | split: test |
| | revision: d51519689f32196a32af33b075a01d0e7c51e252 |
| | metrics: |
| | - type: accuracy |
| | value: 99.26276987888363 |
| | - type: f1 |
| | value: 99.20133403545726 |
| | - type: precision |
| | value: 99.17500438827453 |
| | - type: recall |
| | value: 99.26276987888363 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/banking77 |
| | name: MTEB Banking77Classification |
| | config: default |
| | split: test |
| | revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
| | metrics: |
| | - type: accuracy |
| | value: 84.72727272727273 |
| | - type: f1 |
| | value: 84.67672206031433 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/biorxiv-clustering-p2p |
| | name: MTEB BiorxivClusteringP2P |
| | config: default |
| | split: test |
| | revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
| | metrics: |
| | - type: v_measure |
| | value: 35.34220182511161 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/biorxiv-clustering-s2s |
| | name: MTEB BiorxivClusteringS2S |
| | config: default |
| | split: test |
| | revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
| | metrics: |
| | - type: v_measure |
| | value: 33.4987096128766 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 25.558249999999997 |
| | - type: map_at_10 |
| | value: 34.44425000000001 |
| | - type: map_at_100 |
| | value: 35.59833333333333 |
| | - type: map_at_1000 |
| | value: 35.706916666666665 |
| | - type: map_at_3 |
| | value: 31.691749999999995 |
| | - type: map_at_5 |
| | value: 33.252916666666664 |
| | - type: mrr_at_1 |
| | value: 30.252666666666666 |
| | - type: mrr_at_10 |
| | value: 38.60675 |
| | - type: mrr_at_100 |
| | value: 39.42666666666666 |
| | - type: mrr_at_1000 |
| | value: 39.48408333333334 |
| | - type: mrr_at_3 |
| | value: 36.17441666666665 |
| | - type: mrr_at_5 |
| | value: 37.56275 |
| | - type: ndcg_at_1 |
| | value: 30.252666666666666 |
| | - type: ndcg_at_10 |
| | value: 39.683 |
| | - type: ndcg_at_100 |
| | value: 44.68541666666667 |
| | - type: ndcg_at_1000 |
| | value: 46.94316666666668 |
| | - type: ndcg_at_3 |
| | value: 34.961749999999995 |
| | - type: ndcg_at_5 |
| | value: 37.215666666666664 |
| | - type: precision_at_1 |
| | value: 30.252666666666666 |
| | - type: precision_at_10 |
| | value: 6.904166666666667 |
| | - type: precision_at_100 |
| | value: 1.0989999999999995 |
| | - type: precision_at_1000 |
| | value: 0.14733333333333334 |
| | - type: precision_at_3 |
| | value: 16.037666666666667 |
| | - type: precision_at_5 |
| | value: 11.413583333333333 |
| | - type: recall_at_1 |
| | value: 25.558249999999997 |
| | - type: recall_at_10 |
| | value: 51.13341666666666 |
| | - type: recall_at_100 |
| | value: 73.08366666666667 |
| | - type: recall_at_1000 |
| | value: 88.79483333333334 |
| | - type: recall_at_3 |
| | value: 37.989083333333326 |
| | - type: recall_at_5 |
| | value: 43.787833333333325 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: climate-fever |
| | name: MTEB ClimateFEVER |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 10.338 |
| | - type: map_at_10 |
| | value: 18.360000000000003 |
| | - type: map_at_100 |
| | value: 19.942 |
| | - type: map_at_1000 |
| | value: 20.134 |
| | - type: map_at_3 |
| | value: 15.174000000000001 |
| | - type: map_at_5 |
| | value: 16.830000000000002 |
| | - type: mrr_at_1 |
| | value: 23.257 |
| | - type: mrr_at_10 |
| | value: 33.768 |
| | - type: mrr_at_100 |
| | value: 34.707 |
| | - type: mrr_at_1000 |
| | value: 34.766000000000005 |
| | - type: mrr_at_3 |
| | value: 30.977 |
| | - type: mrr_at_5 |
| | value: 32.528 |
| | - type: ndcg_at_1 |
| | value: 23.257 |
| | - type: ndcg_at_10 |
| | value: 25.733 |
| | - type: ndcg_at_100 |
| | value: 32.288 |
| | - type: ndcg_at_1000 |
| | value: 35.992000000000004 |
| | - type: ndcg_at_3 |
| | value: 20.866 |
| | - type: ndcg_at_5 |
| | value: 22.612 |
| | - type: precision_at_1 |
| | value: 23.257 |
| | - type: precision_at_10 |
| | value: 8.124 |
| | - type: precision_at_100 |
| | value: 1.518 |
| | - type: precision_at_1000 |
| | value: 0.219 |
| | - type: precision_at_3 |
| | value: 15.679000000000002 |
| | - type: precision_at_5 |
| | value: 12.117 |
| | - type: recall_at_1 |
| | value: 10.338 |
| | - type: recall_at_10 |
| | value: 31.154 |
| | - type: recall_at_100 |
| | value: 54.161 |
| | - type: recall_at_1000 |
| | value: 75.21900000000001 |
| | - type: recall_at_3 |
| | value: 19.427 |
| | - type: recall_at_5 |
| | value: 24.214 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: dbpedia-entity |
| | name: MTEB DBPedia |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 8.498 |
| | - type: map_at_10 |
| | value: 19.103 |
| | - type: map_at_100 |
| | value: 27.375 |
| | - type: map_at_1000 |
| | value: 28.981 |
| | - type: map_at_3 |
| | value: 13.764999999999999 |
| | - type: map_at_5 |
| | value: 15.950000000000001 |
| | - type: mrr_at_1 |
| | value: 65.5 |
| | - type: mrr_at_10 |
| | value: 74.53800000000001 |
| | - type: mrr_at_100 |
| | value: 74.71799999999999 |
| | - type: mrr_at_1000 |
| | value: 74.725 |
| | - type: mrr_at_3 |
| | value: 72.792 |
| | - type: mrr_at_5 |
| | value: 73.554 |
| | - type: ndcg_at_1 |
| | value: 53.37499999999999 |
| | - type: ndcg_at_10 |
| | value: 41.286 |
| | - type: ndcg_at_100 |
| | value: 45.972 |
| | - type: ndcg_at_1000 |
| | value: 53.123 |
| | - type: ndcg_at_3 |
| | value: 46.172999999999995 |
| | - type: ndcg_at_5 |
| | value: 43.033 |
| | - type: precision_at_1 |
| | value: 65.5 |
| | - type: precision_at_10 |
| | value: 32.725 |
| | - type: precision_at_100 |
| | value: 10.683 |
| | - type: precision_at_1000 |
| | value: 1.978 |
| | - type: precision_at_3 |
| | value: 50 |
| | - type: precision_at_5 |
| | value: 41.349999999999994 |
| | - type: recall_at_1 |
| | value: 8.498 |
| | - type: recall_at_10 |
| | value: 25.070999999999998 |
| | - type: recall_at_100 |
| | value: 52.383 |
| | - type: recall_at_1000 |
| | value: 74.91499999999999 |
| | - type: recall_at_3 |
| | value: 15.207999999999998 |
| | - type: recall_at_5 |
| | value: 18.563 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/emotion |
| | name: MTEB EmotionClassification |
| | config: default |
| | split: test |
| | revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
| | metrics: |
| | - type: accuracy |
| | value: 46.5 |
| | - type: f1 |
| | value: 41.93833713984145 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: fever |
| | name: MTEB FEVER |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 67.914 |
| | - type: map_at_10 |
| | value: 78.10000000000001 |
| | - type: map_at_100 |
| | value: 78.333 |
| | - type: map_at_1000 |
| | value: 78.346 |
| | - type: map_at_3 |
| | value: 76.626 |
| | - type: map_at_5 |
| | value: 77.627 |
| | - type: mrr_at_1 |
| | value: 72.74199999999999 |
| | - type: mrr_at_10 |
| | value: 82.414 |
| | - type: mrr_at_100 |
| | value: 82.511 |
| | - type: mrr_at_1000 |
| | value: 82.513 |
| | - type: mrr_at_3 |
| | value: 81.231 |
| | - type: mrr_at_5 |
| | value: 82.065 |
| | - type: ndcg_at_1 |
| | value: 72.74199999999999 |
| | - type: ndcg_at_10 |
| | value: 82.806 |
| | - type: ndcg_at_100 |
| | value: 83.677 |
| | - type: ndcg_at_1000 |
| | value: 83.917 |
| | - type: ndcg_at_3 |
| | value: 80.305 |
| | - type: ndcg_at_5 |
| | value: 81.843 |
| | - type: precision_at_1 |
| | value: 72.74199999999999 |
| | - type: precision_at_10 |
| | value: 10.24 |
| | - type: precision_at_100 |
| | value: 1.089 |
| | - type: precision_at_1000 |
| | value: 0.11299999999999999 |
| | - type: precision_at_3 |
| | value: 31.268 |
| | - type: precision_at_5 |
| | value: 19.706000000000003 |
| | - type: recall_at_1 |
| | value: 67.914 |
| | - type: recall_at_10 |
| | value: 92.889 |
| | - type: recall_at_100 |
| | value: 96.42699999999999 |
| | - type: recall_at_1000 |
| | value: 97.92 |
| | - type: recall_at_3 |
| | value: 86.21 |
| | - type: recall_at_5 |
| | value: 90.036 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: fiqa |
| | name: MTEB FiQA2018 |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 22.166 |
| | - type: map_at_10 |
| | value: 35.57 |
| | - type: map_at_100 |
| | value: 37.405 |
| | - type: map_at_1000 |
| | value: 37.564 |
| | - type: map_at_3 |
| | value: 30.379 |
| | - type: map_at_5 |
| | value: 33.324 |
| | - type: mrr_at_1 |
| | value: 43.519000000000005 |
| | - type: mrr_at_10 |
| | value: 51.556000000000004 |
| | - type: mrr_at_100 |
| | value: 52.344 |
| | - type: mrr_at_1000 |
| | value: 52.373999999999995 |
| | - type: mrr_at_3 |
| | value: 48.868 |
| | - type: mrr_at_5 |
| | value: 50.319 |
| | - type: ndcg_at_1 |
| | value: 43.519000000000005 |
| | - type: ndcg_at_10 |
| | value: 43.803 |
| | - type: ndcg_at_100 |
| | value: 50.468999999999994 |
| | - type: ndcg_at_1000 |
| | value: 53.111 |
| | - type: ndcg_at_3 |
| | value: 38.893 |
| | - type: ndcg_at_5 |
| | value: 40.653 |
| | - type: precision_at_1 |
| | value: 43.519000000000005 |
| | - type: precision_at_10 |
| | value: 12.253 |
| | - type: precision_at_100 |
| | value: 1.931 |
| | - type: precision_at_1000 |
| | value: 0.242 |
| | - type: precision_at_3 |
| | value: 25.617 |
| | - type: precision_at_5 |
| | value: 19.383 |
| | - type: recall_at_1 |
| | value: 22.166 |
| | - type: recall_at_10 |
| | value: 51.6 |
| | - type: recall_at_100 |
| | value: 76.574 |
| | - type: recall_at_1000 |
| | value: 92.192 |
| | - type: recall_at_3 |
| | value: 34.477999999999994 |
| | - type: recall_at_5 |
| | value: 41.835 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: hotpotqa |
| | name: MTEB HotpotQA |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 39.041 |
| | - type: map_at_10 |
| | value: 62.961999999999996 |
| | - type: map_at_100 |
| | value: 63.79899999999999 |
| | - type: map_at_1000 |
| | value: 63.854 |
| | - type: map_at_3 |
| | value: 59.399 |
| | - type: map_at_5 |
| | value: 61.669 |
| | - type: mrr_at_1 |
| | value: 78.082 |
| | - type: mrr_at_10 |
| | value: 84.321 |
| | - type: mrr_at_100 |
| | value: 84.49600000000001 |
| | - type: mrr_at_1000 |
| | value: 84.502 |
| | - type: mrr_at_3 |
| | value: 83.421 |
| | - type: mrr_at_5 |
| | value: 83.977 |
| | - type: ndcg_at_1 |
| | value: 78.082 |
| | - type: ndcg_at_10 |
| | value: 71.229 |
| | - type: ndcg_at_100 |
| | value: 74.10900000000001 |
| | - type: ndcg_at_1000 |
| | value: 75.169 |
| | - type: ndcg_at_3 |
| | value: 66.28699999999999 |
| | - type: ndcg_at_5 |
| | value: 69.084 |
| | - type: precision_at_1 |
| | value: 78.082 |
| | - type: precision_at_10 |
| | value: 14.993 |
| | - type: precision_at_100 |
| | value: 1.7239999999999998 |
| | - type: precision_at_1000 |
| | value: 0.186 |
| | - type: precision_at_3 |
| | value: 42.737 |
| | - type: precision_at_5 |
| | value: 27.843 |
| | - type: recall_at_1 |
| | value: 39.041 |
| | - type: recall_at_10 |
| | value: 74.96300000000001 |
| | - type: recall_at_100 |
| | value: 86.199 |
| | - type: recall_at_1000 |
| | value: 93.228 |
| | - type: recall_at_3 |
| | value: 64.105 |
| | - type: recall_at_5 |
| | value: 69.608 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/imdb |
| | name: MTEB ImdbClassification |
| | config: default |
| | split: test |
| | revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
| | metrics: |
| | - type: accuracy |
| | value: 90.23160000000001 |
| | - type: ap |
| | value: 85.5674856808308 |
| | - type: f1 |
| | value: 90.18033354786317 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: msmarco |
| | name: MTEB MSMARCO |
| | config: default |
| | split: dev |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 24.091 |
| | - type: map_at_10 |
| | value: 36.753 |
| | - type: map_at_100 |
| | value: 37.913000000000004 |
| | - type: map_at_1000 |
| | value: 37.958999999999996 |
| | - type: map_at_3 |
| | value: 32.818999999999996 |
| | - type: map_at_5 |
| | value: 35.171 |
| | - type: mrr_at_1 |
| | value: 24.742 |
| | - type: mrr_at_10 |
| | value: 37.285000000000004 |
| | - type: mrr_at_100 |
| | value: 38.391999999999996 |
| | - type: mrr_at_1000 |
| | value: 38.431 |
| | - type: mrr_at_3 |
| | value: 33.440999999999995 |
| | - type: mrr_at_5 |
| | value: 35.75 |
| | - type: ndcg_at_1 |
| | value: 24.742 |
| | - type: ndcg_at_10 |
| | value: 43.698 |
| | - type: ndcg_at_100 |
| | value: 49.145 |
| | - type: ndcg_at_1000 |
| | value: 50.23800000000001 |
| | - type: ndcg_at_3 |
| | value: 35.769 |
| | - type: ndcg_at_5 |
| | value: 39.961999999999996 |
| | - type: precision_at_1 |
| | value: 24.742 |
| | - type: precision_at_10 |
| | value: 6.7989999999999995 |
| | - type: precision_at_100 |
| | value: 0.95 |
| | - type: precision_at_1000 |
| | value: 0.104 |
| | - type: precision_at_3 |
| | value: 15.096000000000002 |
| | - type: precision_at_5 |
| | value: 11.183 |
| | - type: recall_at_1 |
| | value: 24.091 |
| | - type: recall_at_10 |
| | value: 65.068 |
| | - type: recall_at_100 |
| | value: 89.899 |
| | - type: recall_at_1000 |
| | value: 98.16 |
| | - type: recall_at_3 |
| | value: 43.68 |
| | - type: recall_at_5 |
| | value: 53.754999999999995 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (en) |
| | config: en |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 93.66621067031465 |
| | - type: f1 |
| | value: 93.49622853272142 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (de) |
| | config: de |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 91.94702733164272 |
| | - type: f1 |
| | value: 91.17043441745282 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (es) |
| | config: es |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 92.20146764509674 |
| | - type: f1 |
| | value: 91.98359080555608 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (fr) |
| | config: fr |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 88.99780770435328 |
| | - type: f1 |
| | value: 89.19746342724068 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (hi) |
| | config: hi |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 89.78486912871998 |
| | - type: f1 |
| | value: 89.24578823628642 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (th) |
| | config: th |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 88.74502712477394 |
| | - type: f1 |
| | value: 89.00297573881542 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (en) |
| | config: en |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 77.9046967624259 |
| | - type: f1 |
| | value: 59.36787125785957 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (de) |
| | config: de |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 74.5280360664976 |
| | - type: f1 |
| | value: 57.17723440888718 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (es) |
| | config: es |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 75.44029352901934 |
| | - type: f1 |
| | value: 54.052855531072964 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (fr) |
| | config: fr |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 70.5606013153774 |
| | - type: f1 |
| | value: 52.62215934386531 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (hi) |
| | config: hi |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 73.11581211903908 |
| | - type: f1 |
| | value: 52.341291845645465 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (th) |
| | config: th |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 74.28933092224233 |
| | - type: f1 |
| | value: 57.07918745504911 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (af) |
| | config: af |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 62.38063214525892 |
| | - type: f1 |
| | value: 59.46463723443009 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (am) |
| | config: am |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 56.06926698049766 |
| | - type: f1 |
| | value: 52.49084283283562 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ar) |
| | config: ar |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 60.74983187626093 |
| | - type: f1 |
| | value: 56.960640620165904 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (az) |
| | config: az |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 64.86550100874243 |
| | - type: f1 |
| | value: 62.47370548140688 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (bn) |
| | config: bn |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 63.971082716879636 |
| | - type: f1 |
| | value: 61.03812421957381 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (cy) |
| | config: cy |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 54.98318762609282 |
| | - type: f1 |
| | value: 51.51207916008392 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (da) |
| | config: da |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.45527908540686 |
| | - type: f1 |
| | value: 66.16631905400318 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (de) |
| | config: de |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.32750504371216 |
| | - type: f1 |
| | value: 66.16755288646591 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (el) |
| | config: el |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.09213180901143 |
| | - type: f1 |
| | value: 66.95654394661507 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (en) |
| | config: en |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 73.75588433086752 |
| | - type: f1 |
| | value: 71.79973779656923 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (es) |
| | config: es |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 70.49428379287154 |
| | - type: f1 |
| | value: 68.37494379215734 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (fa) |
| | config: fa |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.90921318090115 |
| | - type: f1 |
| | value: 66.79517376481645 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (fi) |
| | config: fi |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 70.12104909213181 |
| | - type: f1 |
| | value: 67.29448842879584 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (fr) |
| | config: fr |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.34095494283793 |
| | - type: f1 |
| | value: 67.01134288992947 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (he) |
| | config: he |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 67.61264290517822 |
| | - type: f1 |
| | value: 64.68730512660757 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (hi) |
| | config: hi |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 67.79757901815738 |
| | - type: f1 |
| | value: 65.24938539425598 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (hu) |
| | config: hu |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.68728984532616 |
| | - type: f1 |
| | value: 67.0487169762553 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (hy) |
| | config: hy |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 62.07464694014795 |
| | - type: f1 |
| | value: 59.183532276789286 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (id) |
| | config: id |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 70.04707464694015 |
| | - type: f1 |
| | value: 67.66829629003848 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (is) |
| | config: is |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 62.42434431741762 |
| | - type: f1 |
| | value: 59.01617226544757 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (it) |
| | config: it |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 70.53127101546738 |
| | - type: f1 |
| | value: 68.10033760906255 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ja) |
| | config: ja |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 72.50504371217215 |
| | - type: f1 |
| | value: 69.74931103158923 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (jv) |
| | config: jv |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 57.91190316072628 |
| | - type: f1 |
| | value: 54.05551136648796 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ka) |
| | config: ka |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 51.78211163416275 |
| | - type: f1 |
| | value: 49.874888544058535 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (km) |
| | config: km |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 47.017484868863484 |
| | - type: f1 |
| | value: 44.53364263352014 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (kn) |
| | config: kn |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 62.16207128446537 |
| | - type: f1 |
| | value: 59.01185692320829 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ko) |
| | config: ko |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.42501681237391 |
| | - type: f1 |
| | value: 67.13169450166086 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (lv) |
| | config: lv |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 67.0780094149294 |
| | - type: f1 |
| | value: 64.41720167850707 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ml) |
| | config: ml |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 65.57162071284466 |
| | - type: f1 |
| | value: 62.414138683804424 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (mn) |
| | config: mn |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 61.71149966375252 |
| | - type: f1 |
| | value: 58.594805125087234 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ms) |
| | config: ms |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 66.03900470746471 |
| | - type: f1 |
| | value: 63.87937257883887 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (my) |
| | config: my |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 60.8776059179556 |
| | - type: f1 |
| | value: 57.48587618059131 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (nb) |
| | config: nb |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.87895090786819 |
| | - type: f1 |
| | value: 66.8141299430347 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (nl) |
| | config: nl |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 70.45057162071285 |
| | - type: f1 |
| | value: 67.46444039673516 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (pl) |
| | config: pl |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 71.546738399462 |
| | - type: f1 |
| | value: 68.63640876702655 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (pt) |
| | config: pt |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 70.72965702757229 |
| | - type: f1 |
| | value: 68.54119560379115 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ro) |
| | config: ro |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 68.35574983187625 |
| | - type: f1 |
| | value: 65.88844917691927 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ru) |
| | config: ru |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 71.70477471418964 |
| | - type: f1 |
| | value: 69.19665697061978 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (sl) |
| | config: sl |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 67.0880968392737 |
| | - type: f1 |
| | value: 64.76962317666086 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (sq) |
| | config: sq |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 65.18493611297916 |
| | - type: f1 |
| | value: 62.49984559035371 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (sv) |
| | config: sv |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 71.75857431069265 |
| | - type: f1 |
| | value: 69.20053687623418 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (sw) |
| | config: sw |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 58.500336247478145 |
| | - type: f1 |
| | value: 55.2972398687929 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ta) |
| | config: ta |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 62.68997982515132 |
| | - type: f1 |
| | value: 59.36848202755348 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (te) |
| | config: te |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 63.01950235373235 |
| | - type: f1 |
| | value: 60.09351954625423 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (th) |
| | config: th |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 68.29186281102892 |
| | - type: f1 |
| | value: 67.57860496703447 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (tl) |
| | config: tl |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 64.77471418964357 |
| | - type: f1 |
| | value: 61.913983147713836 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (tr) |
| | config: tr |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.87222595830532 |
| | - type: f1 |
| | value: 66.03679033708141 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (ur) |
| | config: ur |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 64.04505716207127 |
| | - type: f1 |
| | value: 61.28569169817908 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (vi) |
| | config: vi |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 69.38466711499663 |
| | - type: f1 |
| | value: 67.20532357036844 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (zh-CN) |
| | config: zh-CN |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 71.12306657700067 |
| | - type: f1 |
| | value: 68.91251226588182 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (zh-TW) |
| | config: zh-TW |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 66.20040349697378 |
| | - type: f1 |
| | value: 66.02657347714175 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (af) |
| | config: af |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 68.73907195696032 |
| | - type: f1 |
| | value: 66.98484521791418 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (am) |
| | config: am |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 60.58843308675185 |
| | - type: f1 |
| | value: 58.95591723092005 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ar) |
| | config: ar |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 66.22730329522528 |
| | - type: f1 |
| | value: 66.0894499712115 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (az) |
| | config: az |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 66.48285137861465 |
| | - type: f1 |
| | value: 65.21963176785157 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (bn) |
| | config: bn |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 67.74714189643578 |
| | - type: f1 |
| | value: 66.8212192745412 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (cy) |
| | config: cy |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 59.09213180901143 |
| | - type: f1 |
| | value: 56.70735546356339 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (da) |
| | config: da |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 75.05716207128448 |
| | - type: f1 |
| | value: 74.8413712365364 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (de) |
| | config: de |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.69737726967047 |
| | - type: f1 |
| | value: 74.7664341963 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (el) |
| | config: el |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.90383322125084 |
| | - type: f1 |
| | value: 73.59201554448323 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (en) |
| | config: en |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 77.51176866173503 |
| | - type: f1 |
| | value: 77.46104434577758 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (es) |
| | config: es |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.31069266980496 |
| | - type: f1 |
| | value: 74.61048660675635 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (fa) |
| | config: fa |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 72.95225285810356 |
| | - type: f1 |
| | value: 72.33160006574627 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (fi) |
| | config: fi |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.12373907195696 |
| | - type: f1 |
| | value: 73.20921012557481 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (fr) |
| | config: fr |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.86684599865501 |
| | - type: f1 |
| | value: 73.82348774610831 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (he) |
| | config: he |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 71.40215198386012 |
| | - type: f1 |
| | value: 71.11945183971858 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (hi) |
| | config: hi |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 72.12844653665098 |
| | - type: f1 |
| | value: 71.34450495911766 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (hu) |
| | config: hu |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.52252858103566 |
| | - type: f1 |
| | value: 73.98878711342999 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (hy) |
| | config: hy |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 64.93611297915265 |
| | - type: f1 |
| | value: 63.723200467653385 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (id) |
| | config: id |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.11903160726295 |
| | - type: f1 |
| | value: 73.82138439467096 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (is) |
| | config: is |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 67.15198386012105 |
| | - type: f1 |
| | value: 66.02172193802167 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (it) |
| | config: it |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.32414256893072 |
| | - type: f1 |
| | value: 74.30943421170574 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ja) |
| | config: ja |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 77.46805648957633 |
| | - type: f1 |
| | value: 77.62808409298209 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (jv) |
| | config: jv |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 63.318762609280434 |
| | - type: f1 |
| | value: 62.094284066075076 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ka) |
| | config: ka |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 58.34902488231338 |
| | - type: f1 |
| | value: 57.12893860987984 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (km) |
| | config: km |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 50.88433086751849 |
| | - type: f1 |
| | value: 48.2272350802058 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (kn) |
| | config: kn |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 66.4425016812374 |
| | - type: f1 |
| | value: 64.61463095996173 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ko) |
| | config: ko |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 75.04707464694015 |
| | - type: f1 |
| | value: 75.05099199098998 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (lv) |
| | config: lv |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 70.50437121721586 |
| | - type: f1 |
| | value: 69.83397721096314 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ml) |
| | config: ml |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 69.94283792871553 |
| | - type: f1 |
| | value: 68.8704663703913 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (mn) |
| | config: mn |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 64.79488903833222 |
| | - type: f1 |
| | value: 63.615424063345436 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ms) |
| | config: ms |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 69.88231338264963 |
| | - type: f1 |
| | value: 68.57892302593237 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (my) |
| | config: my |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 63.248150638870214 |
| | - type: f1 |
| | value: 61.06680605338809 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (nb) |
| | config: nb |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.84196368527236 |
| | - type: f1 |
| | value: 74.52566464968763 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (nl) |
| | config: nl |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.8285137861466 |
| | - type: f1 |
| | value: 74.8853197608802 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (pl) |
| | config: pl |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 74.13248150638869 |
| | - type: f1 |
| | value: 74.3982040999179 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (pt) |
| | config: pt |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.49024882313383 |
| | - type: f1 |
| | value: 73.82153848368573 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ro) |
| | config: ro |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 71.72158708809684 |
| | - type: f1 |
| | value: 71.85049433180541 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ru) |
| | config: ru |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 75.137861466039 |
| | - type: f1 |
| | value: 75.37628348188467 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (sl) |
| | config: sl |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 71.86953597848016 |
| | - type: f1 |
| | value: 71.87537624521661 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (sq) |
| | config: sq |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 70.27572293207801 |
| | - type: f1 |
| | value: 68.80017302344231 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (sv) |
| | config: sv |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 76.09952925353059 |
| | - type: f1 |
| | value: 76.07992707688408 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (sw) |
| | config: sw |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 63.140551445864155 |
| | - type: f1 |
| | value: 61.73855010331415 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ta) |
| | config: ta |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 66.27774041694687 |
| | - type: f1 |
| | value: 64.83664868894539 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (te) |
| | config: te |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 66.69468728984533 |
| | - type: f1 |
| | value: 64.76239666920868 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (th) |
| | config: th |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.44653665097512 |
| | - type: f1 |
| | value: 73.14646052013873 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (tl) |
| | config: tl |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 67.71351714862139 |
| | - type: f1 |
| | value: 66.67212180163382 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (tr) |
| | config: tr |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.9946200403497 |
| | - type: f1 |
| | value: 73.87348793725525 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (ur) |
| | config: ur |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 68.15400134498992 |
| | - type: f1 |
| | value: 67.09433241421094 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (vi) |
| | config: vi |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 73.11365164761264 |
| | - type: f1 |
| | value: 73.59502539433753 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (zh-CN) |
| | config: zh-CN |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 76.82582380632145 |
| | - type: f1 |
| | value: 76.89992945316313 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (zh-TW) |
| | config: zh-TW |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 71.81237390719569 |
| | - type: f1 |
| | value: 72.36499770986265 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/medrxiv-clustering-p2p |
| | name: MTEB MedrxivClusteringP2P |
| | config: default |
| | split: test |
| | revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
| | metrics: |
| | - type: v_measure |
| | value: 31.480506569594695 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/medrxiv-clustering-s2s |
| | name: MTEB MedrxivClusteringS2S |
| | config: default |
| | split: test |
| | revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
| | metrics: |
| | - type: v_measure |
| | value: 29.71252128004552 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/mind_small |
| | name: MTEB MindSmallReranking |
| | config: default |
| | split: test |
| | revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
| | metrics: |
| | - type: map |
| | value: 31.421396787056548 |
| | - type: mrr |
| | value: 32.48155274872267 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: nfcorpus |
| | name: MTEB NFCorpus |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 5.595 |
| | - type: map_at_10 |
| | value: 12.642000000000001 |
| | - type: map_at_100 |
| | value: 15.726 |
| | - type: map_at_1000 |
| | value: 17.061999999999998 |
| | - type: map_at_3 |
| | value: 9.125 |
| | - type: map_at_5 |
| | value: 10.866000000000001 |
| | - type: mrr_at_1 |
| | value: 43.344 |
| | - type: mrr_at_10 |
| | value: 52.227999999999994 |
| | - type: mrr_at_100 |
| | value: 52.898999999999994 |
| | - type: mrr_at_1000 |
| | value: 52.944 |
| | - type: mrr_at_3 |
| | value: 49.845 |
| | - type: mrr_at_5 |
| | value: 51.115 |
| | - type: ndcg_at_1 |
| | value: 41.949999999999996 |
| | - type: ndcg_at_10 |
| | value: 33.995 |
| | - type: ndcg_at_100 |
| | value: 30.869999999999997 |
| | - type: ndcg_at_1000 |
| | value: 39.487 |
| | - type: ndcg_at_3 |
| | value: 38.903999999999996 |
| | - type: ndcg_at_5 |
| | value: 37.236999999999995 |
| | - type: precision_at_1 |
| | value: 43.344 |
| | - type: precision_at_10 |
| | value: 25.480000000000004 |
| | - type: precision_at_100 |
| | value: 7.672 |
| | - type: precision_at_1000 |
| | value: 2.028 |
| | - type: precision_at_3 |
| | value: 36.636 |
| | - type: precision_at_5 |
| | value: 32.632 |
| | - type: recall_at_1 |
| | value: 5.595 |
| | - type: recall_at_10 |
| | value: 16.466 |
| | - type: recall_at_100 |
| | value: 31.226 |
| | - type: recall_at_1000 |
| | value: 62.778999999999996 |
| | - type: recall_at_3 |
| | value: 9.931 |
| | - type: recall_at_5 |
| | value: 12.884 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: nq |
| | name: MTEB NQ |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 40.414 |
| | - type: map_at_10 |
| | value: 56.754000000000005 |
| | - type: map_at_100 |
| | value: 57.457 |
| | - type: map_at_1000 |
| | value: 57.477999999999994 |
| | - type: map_at_3 |
| | value: 52.873999999999995 |
| | - type: map_at_5 |
| | value: 55.175 |
| | - type: mrr_at_1 |
| | value: 45.278 |
| | - type: mrr_at_10 |
| | value: 59.192 |
| | - type: mrr_at_100 |
| | value: 59.650000000000006 |
| | - type: mrr_at_1000 |
| | value: 59.665 |
| | - type: mrr_at_3 |
| | value: 56.141 |
| | - type: mrr_at_5 |
| | value: 57.998000000000005 |
| | - type: ndcg_at_1 |
| | value: 45.278 |
| | - type: ndcg_at_10 |
| | value: 64.056 |
| | - type: ndcg_at_100 |
| | value: 66.89 |
| | - type: ndcg_at_1000 |
| | value: 67.364 |
| | - type: ndcg_at_3 |
| | value: 56.97 |
| | - type: ndcg_at_5 |
| | value: 60.719 |
| | - type: precision_at_1 |
| | value: 45.278 |
| | - type: precision_at_10 |
| | value: 9.994 |
| | - type: precision_at_100 |
| | value: 1.165 |
| | - type: precision_at_1000 |
| | value: 0.121 |
| | - type: precision_at_3 |
| | value: 25.512 |
| | - type: precision_at_5 |
| | value: 17.509 |
| | - type: recall_at_1 |
| | value: 40.414 |
| | - type: recall_at_10 |
| | value: 83.596 |
| | - type: recall_at_100 |
| | value: 95.72 |
| | - type: recall_at_1000 |
| | value: 99.24 |
| | - type: recall_at_3 |
| | value: 65.472 |
| | - type: recall_at_5 |
| | value: 74.039 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: quora |
| | name: MTEB QuoraRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 70.352 |
| | - type: map_at_10 |
| | value: 84.369 |
| | - type: map_at_100 |
| | value: 85.02499999999999 |
| | - type: map_at_1000 |
| | value: 85.04 |
| | - type: map_at_3 |
| | value: 81.42399999999999 |
| | - type: map_at_5 |
| | value: 83.279 |
| | - type: mrr_at_1 |
| | value: 81.05 |
| | - type: mrr_at_10 |
| | value: 87.401 |
| | - type: mrr_at_100 |
| | value: 87.504 |
| | - type: mrr_at_1000 |
| | value: 87.505 |
| | - type: mrr_at_3 |
| | value: 86.443 |
| | - type: mrr_at_5 |
| | value: 87.10799999999999 |
| | - type: ndcg_at_1 |
| | value: 81.04 |
| | - type: ndcg_at_10 |
| | value: 88.181 |
| | - type: ndcg_at_100 |
| | value: 89.411 |
| | - type: ndcg_at_1000 |
| | value: 89.507 |
| | - type: ndcg_at_3 |
| | value: 85.28099999999999 |
| | - type: ndcg_at_5 |
| | value: 86.888 |
| | - type: precision_at_1 |
| | value: 81.04 |
| | - type: precision_at_10 |
| | value: 13.406 |
| | - type: precision_at_100 |
| | value: 1.5350000000000001 |
| | - type: precision_at_1000 |
| | value: 0.157 |
| | - type: precision_at_3 |
| | value: 37.31 |
| | - type: precision_at_5 |
| | value: 24.54 |
| | - type: recall_at_1 |
| | value: 70.352 |
| | - type: recall_at_10 |
| | value: 95.358 |
| | - type: recall_at_100 |
| | value: 99.541 |
| | - type: recall_at_1000 |
| | value: 99.984 |
| | - type: recall_at_3 |
| | value: 87.111 |
| | - type: recall_at_5 |
| | value: 91.643 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/reddit-clustering |
| | name: MTEB RedditClustering |
| | config: default |
| | split: test |
| | revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
| | metrics: |
| | - type: v_measure |
| | value: 46.54068723291946 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/reddit-clustering-p2p |
| | name: MTEB RedditClusteringP2P |
| | config: default |
| | split: test |
| | revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
| | metrics: |
| | - type: v_measure |
| | value: 63.216287629895994 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: scidocs |
| | name: MTEB SCIDOCS |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 4.023000000000001 |
| | - type: map_at_10 |
| | value: 10.071 |
| | - type: map_at_100 |
| | value: 11.892 |
| | - type: map_at_1000 |
| | value: 12.196 |
| | - type: map_at_3 |
| | value: 7.234 |
| | - type: map_at_5 |
| | value: 8.613999999999999 |
| | - type: mrr_at_1 |
| | value: 19.900000000000002 |
| | - type: mrr_at_10 |
| | value: 30.516 |
| | - type: mrr_at_100 |
| | value: 31.656000000000002 |
| | - type: mrr_at_1000 |
| | value: 31.723000000000003 |
| | - type: mrr_at_3 |
| | value: 27.400000000000002 |
| | - type: mrr_at_5 |
| | value: 29.270000000000003 |
| | - type: ndcg_at_1 |
| | value: 19.900000000000002 |
| | - type: ndcg_at_10 |
| | value: 17.474 |
| | - type: ndcg_at_100 |
| | value: 25.020999999999997 |
| | - type: ndcg_at_1000 |
| | value: 30.728 |
| | - type: ndcg_at_3 |
| | value: 16.588 |
| | - type: ndcg_at_5 |
| | value: 14.498 |
| | - type: precision_at_1 |
| | value: 19.900000000000002 |
| | - type: precision_at_10 |
| | value: 9.139999999999999 |
| | - type: precision_at_100 |
| | value: 2.011 |
| | - type: precision_at_1000 |
| | value: 0.33899999999999997 |
| | - type: precision_at_3 |
| | value: 15.667 |
| | - type: precision_at_5 |
| | value: 12.839999999999998 |
| | - type: recall_at_1 |
| | value: 4.023000000000001 |
| | - type: recall_at_10 |
| | value: 18.497 |
| | - type: recall_at_100 |
| | value: 40.8 |
| | - type: recall_at_1000 |
| | value: 68.812 |
| | - type: recall_at_3 |
| | value: 9.508 |
| | - type: recall_at_5 |
| | value: 12.983 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sickr-sts |
| | name: MTEB SICK-R |
| | config: default |
| | split: test |
| | revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 83.967008785134 |
| | - type: cos_sim_spearman |
| | value: 80.23142141101837 |
| | - type: euclidean_pearson |
| | value: 81.20166064704539 |
| | - type: euclidean_spearman |
| | value: 80.18961335654585 |
| | - type: manhattan_pearson |
| | value: 81.13925443187625 |
| | - type: manhattan_spearman |
| | value: 80.07948723044424 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts12-sts |
| | name: MTEB STS12 |
| | config: default |
| | split: test |
| | revision: a0d554a64d88156834ff5ae9920b964011b16384 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 86.94262461316023 |
| | - type: cos_sim_spearman |
| | value: 80.01596278563865 |
| | - type: euclidean_pearson |
| | value: 83.80799622922581 |
| | - type: euclidean_spearman |
| | value: 79.94984954947103 |
| | - type: manhattan_pearson |
| | value: 83.68473841756281 |
| | - type: manhattan_spearman |
| | value: 79.84990707951822 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts13-sts |
| | name: MTEB STS13 |
| | config: default |
| | split: test |
| | revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 80.57346443146068 |
| | - type: cos_sim_spearman |
| | value: 81.54689837570866 |
| | - type: euclidean_pearson |
| | value: 81.10909881516007 |
| | - type: euclidean_spearman |
| | value: 81.56746243261762 |
| | - type: manhattan_pearson |
| | value: 80.87076036186582 |
| | - type: manhattan_spearman |
| | value: 81.33074987964402 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts14-sts |
| | name: MTEB STS14 |
| | config: default |
| | split: test |
| | revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 79.54733787179849 |
| | - type: cos_sim_spearman |
| | value: 77.72202105610411 |
| | - type: euclidean_pearson |
| | value: 78.9043595478849 |
| | - type: euclidean_spearman |
| | value: 77.93422804309435 |
| | - type: manhattan_pearson |
| | value: 78.58115121621368 |
| | - type: manhattan_spearman |
| | value: 77.62508135122033 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts15-sts |
| | name: MTEB STS15 |
| | config: default |
| | split: test |
| | revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 88.59880017237558 |
| | - type: cos_sim_spearman |
| | value: 89.31088630824758 |
| | - type: euclidean_pearson |
| | value: 88.47069261564656 |
| | - type: euclidean_spearman |
| | value: 89.33581971465233 |
| | - type: manhattan_pearson |
| | value: 88.40774264100956 |
| | - type: manhattan_spearman |
| | value: 89.28657485627835 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts16-sts |
| | name: MTEB STS16 |
| | config: default |
| | split: test |
| | revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 84.08055117917084 |
| | - type: cos_sim_spearman |
| | value: 85.78491813080304 |
| | - type: euclidean_pearson |
| | value: 84.99329155500392 |
| | - type: euclidean_spearman |
| | value: 85.76728064677287 |
| | - type: manhattan_pearson |
| | value: 84.87947428989587 |
| | - type: manhattan_spearman |
| | value: 85.62429454917464 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (ko-ko) |
| | config: ko-ko |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 82.14190939287384 |
| | - type: cos_sim_spearman |
| | value: 82.27331573306041 |
| | - type: euclidean_pearson |
| | value: 81.891896953716 |
| | - type: euclidean_spearman |
| | value: 82.37695542955998 |
| | - type: manhattan_pearson |
| | value: 81.73123869460504 |
| | - type: manhattan_spearman |
| | value: 82.19989168441421 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (ar-ar) |
| | config: ar-ar |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 76.84695301843362 |
| | - type: cos_sim_spearman |
| | value: 77.87790986014461 |
| | - type: euclidean_pearson |
| | value: 76.91981583106315 |
| | - type: euclidean_spearman |
| | value: 77.88154772749589 |
| | - type: manhattan_pearson |
| | value: 76.94953277451093 |
| | - type: manhattan_spearman |
| | value: 77.80499230728604 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (en-ar) |
| | config: en-ar |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 75.44657840482016 |
| | - type: cos_sim_spearman |
| | value: 75.05531095119674 |
| | - type: euclidean_pearson |
| | value: 75.88161755829299 |
| | - type: euclidean_spearman |
| | value: 74.73176238219332 |
| | - type: manhattan_pearson |
| | value: 75.63984765635362 |
| | - type: manhattan_spearman |
| | value: 74.86476440770737 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (en-de) |
| | config: en-de |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 85.64700140524133 |
| | - type: cos_sim_spearman |
| | value: 86.16014210425672 |
| | - type: euclidean_pearson |
| | value: 86.49086860843221 |
| | - type: euclidean_spearman |
| | value: 86.09729326815614 |
| | - type: manhattan_pearson |
| | value: 86.43406265125513 |
| | - type: manhattan_spearman |
| | value: 86.17740150939994 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (en-en) |
| | config: en-en |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 87.91170098764921 |
| | - type: cos_sim_spearman |
| | value: 88.12437004058931 |
| | - type: euclidean_pearson |
| | value: 88.81828254494437 |
| | - type: euclidean_spearman |
| | value: 88.14831794572122 |
| | - type: manhattan_pearson |
| | value: 88.93442183448961 |
| | - type: manhattan_spearman |
| | value: 88.15254630778304 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (en-tr) |
| | config: en-tr |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 72.91390577997292 |
| | - type: cos_sim_spearman |
| | value: 71.22979457536074 |
| | - type: euclidean_pearson |
| | value: 74.40314008106749 |
| | - type: euclidean_spearman |
| | value: 72.54972136083246 |
| | - type: manhattan_pearson |
| | value: 73.85687539530218 |
| | - type: manhattan_spearman |
| | value: 72.09500771742637 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (es-en) |
| | config: es-en |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 80.9301067983089 |
| | - type: cos_sim_spearman |
| | value: 80.74989828346473 |
| | - type: euclidean_pearson |
| | value: 81.36781301814257 |
| | - type: euclidean_spearman |
| | value: 80.9448819964426 |
| | - type: manhattan_pearson |
| | value: 81.0351322685609 |
| | - type: manhattan_spearman |
| | value: 80.70192121844177 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (es-es) |
| | config: es-es |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 87.13820465980005 |
| | - type: cos_sim_spearman |
| | value: 86.73532498758757 |
| | - type: euclidean_pearson |
| | value: 87.21329451846637 |
| | - type: euclidean_spearman |
| | value: 86.57863198601002 |
| | - type: manhattan_pearson |
| | value: 87.06973713818554 |
| | - type: manhattan_spearman |
| | value: 86.47534918791499 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (fr-en) |
| | config: fr-en |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 85.48720108904415 |
| | - type: cos_sim_spearman |
| | value: 85.62221757068387 |
| | - type: euclidean_pearson |
| | value: 86.1010129512749 |
| | - type: euclidean_spearman |
| | value: 85.86580966509942 |
| | - type: manhattan_pearson |
| | value: 86.26800938808971 |
| | - type: manhattan_spearman |
| | value: 85.88902721678429 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (it-en) |
| | config: it-en |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 83.98021347333516 |
| | - type: cos_sim_spearman |
| | value: 84.53806553803501 |
| | - type: euclidean_pearson |
| | value: 84.61483347248364 |
| | - type: euclidean_spearman |
| | value: 85.14191408011702 |
| | - type: manhattan_pearson |
| | value: 84.75297588825967 |
| | - type: manhattan_spearman |
| | value: 85.33176753669242 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (nl-en) |
| | config: nl-en |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 84.51856644893233 |
| | - type: cos_sim_spearman |
| | value: 85.27510748506413 |
| | - type: euclidean_pearson |
| | value: 85.09886861540977 |
| | - type: euclidean_spearman |
| | value: 85.62579245860887 |
| | - type: manhattan_pearson |
| | value: 84.93017860464607 |
| | - type: manhattan_spearman |
| | value: 85.5063988898453 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (en) |
| | config: en |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 62.581573200584195 |
| | - type: cos_sim_spearman |
| | value: 63.05503590247928 |
| | - type: euclidean_pearson |
| | value: 63.652564812602094 |
| | - type: euclidean_spearman |
| | value: 62.64811520876156 |
| | - type: manhattan_pearson |
| | value: 63.506842893061076 |
| | - type: manhattan_spearman |
| | value: 62.51289573046917 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (de) |
| | config: de |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 48.2248801729127 |
| | - type: cos_sim_spearman |
| | value: 56.5936604678561 |
| | - type: euclidean_pearson |
| | value: 43.98149464089 |
| | - type: euclidean_spearman |
| | value: 56.108561882423615 |
| | - type: manhattan_pearson |
| | value: 43.86880305903564 |
| | - type: manhattan_spearman |
| | value: 56.04671150510166 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (es) |
| | config: es |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 55.17564527009831 |
| | - type: cos_sim_spearman |
| | value: 64.57978560979488 |
| | - type: euclidean_pearson |
| | value: 58.8818330154583 |
| | - type: euclidean_spearman |
| | value: 64.99214839071281 |
| | - type: manhattan_pearson |
| | value: 58.72671436121381 |
| | - type: manhattan_spearman |
| | value: 65.10713416616109 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (pl) |
| | config: pl |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 26.772131864023297 |
| | - type: cos_sim_spearman |
| | value: 34.68200792408681 |
| | - type: euclidean_pearson |
| | value: 16.68082419005441 |
| | - type: euclidean_spearman |
| | value: 34.83099932652166 |
| | - type: manhattan_pearson |
| | value: 16.52605949659529 |
| | - type: manhattan_spearman |
| | value: 34.82075801399475 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (tr) |
| | config: tr |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 54.42415189043831 |
| | - type: cos_sim_spearman |
| | value: 63.54594264576758 |
| | - type: euclidean_pearson |
| | value: 57.36577498297745 |
| | - type: euclidean_spearman |
| | value: 63.111466379158074 |
| | - type: manhattan_pearson |
| | value: 57.584543715873885 |
| | - type: manhattan_spearman |
| | value: 63.22361054139183 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (ar) |
| | config: ar |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 47.55216762405518 |
| | - type: cos_sim_spearman |
| | value: 56.98670142896412 |
| | - type: euclidean_pearson |
| | value: 50.15318757562699 |
| | - type: euclidean_spearman |
| | value: 56.524941926541906 |
| | - type: manhattan_pearson |
| | value: 49.955618528674904 |
| | - type: manhattan_spearman |
| | value: 56.37102209240117 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (ru) |
| | config: ru |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 49.20540980338571 |
| | - type: cos_sim_spearman |
| | value: 59.9009453504406 |
| | - type: euclidean_pearson |
| | value: 49.557749853620535 |
| | - type: euclidean_spearman |
| | value: 59.76631621172456 |
| | - type: manhattan_pearson |
| | value: 49.62340591181147 |
| | - type: manhattan_spearman |
| | value: 59.94224880322436 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (zh) |
| | config: zh |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 51.508169956576985 |
| | - type: cos_sim_spearman |
| | value: 66.82461565306046 |
| | - type: euclidean_pearson |
| | value: 56.2274426480083 |
| | - type: euclidean_spearman |
| | value: 66.6775323848333 |
| | - type: manhattan_pearson |
| | value: 55.98277796300661 |
| | - type: manhattan_spearman |
| | value: 66.63669848497175 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (fr) |
| | config: fr |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 72.86478788045507 |
| | - type: cos_sim_spearman |
| | value: 76.7946552053193 |
| | - type: euclidean_pearson |
| | value: 75.01598530490269 |
| | - type: euclidean_spearman |
| | value: 76.83618917858281 |
| | - type: manhattan_pearson |
| | value: 74.68337628304332 |
| | - type: manhattan_spearman |
| | value: 76.57480204017773 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (de-en) |
| | config: de-en |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 55.922619099401984 |
| | - type: cos_sim_spearman |
| | value: 56.599362477240774 |
| | - type: euclidean_pearson |
| | value: 56.68307052369783 |
| | - type: euclidean_spearman |
| | value: 54.28760436777401 |
| | - type: manhattan_pearson |
| | value: 56.67763566500681 |
| | - type: manhattan_spearman |
| | value: 53.94619541711359 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (es-en) |
| | config: es-en |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 66.74357206710913 |
| | - type: cos_sim_spearman |
| | value: 72.5208244925311 |
| | - type: euclidean_pearson |
| | value: 67.49254562186032 |
| | - type: euclidean_spearman |
| | value: 72.02469076238683 |
| | - type: manhattan_pearson |
| | value: 67.45251772238085 |
| | - type: manhattan_spearman |
| | value: 72.05538819984538 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (it) |
| | config: it |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 71.25734330033191 |
| | - type: cos_sim_spearman |
| | value: 76.98349083946823 |
| | - type: euclidean_pearson |
| | value: 73.71642838667736 |
| | - type: euclidean_spearman |
| | value: 77.01715504651384 |
| | - type: manhattan_pearson |
| | value: 73.61712711868105 |
| | - type: manhattan_spearman |
| | value: 77.01392571153896 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (pl-en) |
| | config: pl-en |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 63.18215462781212 |
| | - type: cos_sim_spearman |
| | value: 65.54373266117607 |
| | - type: euclidean_pearson |
| | value: 64.54126095439005 |
| | - type: euclidean_spearman |
| | value: 65.30410369102711 |
| | - type: manhattan_pearson |
| | value: 63.50332221148234 |
| | - type: manhattan_spearman |
| | value: 64.3455878104313 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (zh-en) |
| | config: zh-en |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 62.30509221440029 |
| | - type: cos_sim_spearman |
| | value: 65.99582704642478 |
| | - type: euclidean_pearson |
| | value: 63.43818859884195 |
| | - type: euclidean_spearman |
| | value: 66.83172582815764 |
| | - type: manhattan_pearson |
| | value: 63.055779168508764 |
| | - type: manhattan_spearman |
| | value: 65.49585020501449 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (es-it) |
| | config: es-it |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 59.587830825340404 |
| | - type: cos_sim_spearman |
| | value: 68.93467614588089 |
| | - type: euclidean_pearson |
| | value: 62.3073527367404 |
| | - type: euclidean_spearman |
| | value: 69.69758171553175 |
| | - type: manhattan_pearson |
| | value: 61.9074580815789 |
| | - type: manhattan_spearman |
| | value: 69.57696375597865 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (de-fr) |
| | config: de-fr |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 57.143220125577066 |
| | - type: cos_sim_spearman |
| | value: 67.78857859159226 |
| | - type: euclidean_pearson |
| | value: 55.58225107923733 |
| | - type: euclidean_spearman |
| | value: 67.80662907184563 |
| | - type: manhattan_pearson |
| | value: 56.24953502726514 |
| | - type: manhattan_spearman |
| | value: 67.98262125431616 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (de-pl) |
| | config: de-pl |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 21.826928900322066 |
| | - type: cos_sim_spearman |
| | value: 49.578506634400405 |
| | - type: euclidean_pearson |
| | value: 27.939890138843214 |
| | - type: euclidean_spearman |
| | value: 52.71950519136242 |
| | - type: manhattan_pearson |
| | value: 26.39878683847546 |
| | - type: manhattan_spearman |
| | value: 47.54609580342499 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (fr-pl) |
| | config: fr-pl |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 57.27603854632001 |
| | - type: cos_sim_spearman |
| | value: 50.709255283710995 |
| | - type: euclidean_pearson |
| | value: 59.5419024445929 |
| | - type: euclidean_spearman |
| | value: 50.709255283710995 |
| | - type: manhattan_pearson |
| | value: 59.03256832438492 |
| | - type: manhattan_spearman |
| | value: 61.97797868009122 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/stsbenchmark-sts |
| | name: MTEB STSBenchmark |
| | config: default |
| | split: test |
| | revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 85.00757054859712 |
| | - type: cos_sim_spearman |
| | value: 87.29283629622222 |
| | - type: euclidean_pearson |
| | value: 86.54824171775536 |
| | - type: euclidean_spearman |
| | value: 87.24364730491402 |
| | - type: manhattan_pearson |
| | value: 86.5062156915074 |
| | - type: manhattan_spearman |
| | value: 87.15052170378574 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/scidocs-reranking |
| | name: MTEB SciDocsRR |
| | config: default |
| | split: test |
| | revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
| | metrics: |
| | - type: map |
| | value: 82.03549357197389 |
| | - type: mrr |
| | value: 95.05437645143527 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: scifact |
| | name: MTEB SciFact |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 57.260999999999996 |
| | - type: map_at_10 |
| | value: 66.259 |
| | - type: map_at_100 |
| | value: 66.884 |
| | - type: map_at_1000 |
| | value: 66.912 |
| | - type: map_at_3 |
| | value: 63.685 |
| | - type: map_at_5 |
| | value: 65.35499999999999 |
| | - type: mrr_at_1 |
| | value: 60.333000000000006 |
| | - type: mrr_at_10 |
| | value: 67.5 |
| | - type: mrr_at_100 |
| | value: 68.013 |
| | - type: mrr_at_1000 |
| | value: 68.038 |
| | - type: mrr_at_3 |
| | value: 65.61099999999999 |
| | - type: mrr_at_5 |
| | value: 66.861 |
| | - type: ndcg_at_1 |
| | value: 60.333000000000006 |
| | - type: ndcg_at_10 |
| | value: 70.41 |
| | - type: ndcg_at_100 |
| | value: 73.10600000000001 |
| | - type: ndcg_at_1000 |
| | value: 73.846 |
| | - type: ndcg_at_3 |
| | value: 66.133 |
| | - type: ndcg_at_5 |
| | value: 68.499 |
| | - type: precision_at_1 |
| | value: 60.333000000000006 |
| | - type: precision_at_10 |
| | value: 9.232999999999999 |
| | - type: precision_at_100 |
| | value: 1.0630000000000002 |
| | - type: precision_at_1000 |
| | value: 0.11299999999999999 |
| | - type: precision_at_3 |
| | value: 25.667 |
| | - type: precision_at_5 |
| | value: 17.067 |
| | - type: recall_at_1 |
| | value: 57.260999999999996 |
| | - type: recall_at_10 |
| | value: 81.94399999999999 |
| | - type: recall_at_100 |
| | value: 93.867 |
| | - type: recall_at_1000 |
| | value: 99.667 |
| | - type: recall_at_3 |
| | value: 70.339 |
| | - type: recall_at_5 |
| | value: 76.25 |
| | - task: |
| | type: PairClassification |
| | dataset: |
| | type: mteb/sprintduplicatequestions-pairclassification |
| | name: MTEB SprintDuplicateQuestions |
| | config: default |
| | split: test |
| | revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
| | metrics: |
| | - type: cos_sim_accuracy |
| | value: 99.74356435643564 |
| | - type: cos_sim_ap |
| | value: 93.13411948212683 |
| | - type: cos_sim_f1 |
| | value: 86.80521991300147 |
| | - type: cos_sim_precision |
| | value: 84.00374181478017 |
| | - type: cos_sim_recall |
| | value: 89.8 |
| | - type: dot_accuracy |
| | value: 99.67920792079208 |
| | - type: dot_ap |
| | value: 89.27277565444479 |
| | - type: dot_f1 |
| | value: 83.9276990718124 |
| | - type: dot_precision |
| | value: 82.04393505253104 |
| | - type: dot_recall |
| | value: 85.9 |
| | - type: euclidean_accuracy |
| | value: 99.74257425742574 |
| | - type: euclidean_ap |
| | value: 93.17993008259062 |
| | - type: euclidean_f1 |
| | value: 86.69396110542476 |
| | - type: euclidean_precision |
| | value: 88.78406708595388 |
| | - type: euclidean_recall |
| | value: 84.7 |
| | - type: manhattan_accuracy |
| | value: 99.74257425742574 |
| | - type: manhattan_ap |
| | value: 93.14413755550099 |
| | - type: manhattan_f1 |
| | value: 86.82483594144371 |
| | - type: manhattan_precision |
| | value: 87.66564729867483 |
| | - type: manhattan_recall |
| | value: 86 |
| | - type: max_accuracy |
| | value: 99.74356435643564 |
| | - type: max_ap |
| | value: 93.17993008259062 |
| | - type: max_f1 |
| | value: 86.82483594144371 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/stackexchange-clustering |
| | name: MTEB StackExchangeClustering |
| | config: default |
| | split: test |
| | revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
| | metrics: |
| | - type: v_measure |
| | value: 57.525863806168566 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/stackexchange-clustering-p2p |
| | name: MTEB StackExchangeClusteringP2P |
| | config: default |
| | split: test |
| | revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
| | metrics: |
| | - type: v_measure |
| | value: 32.68850574423839 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/stackoverflowdupquestions-reranking |
| | name: MTEB StackOverflowDupQuestions |
| | config: default |
| | split: test |
| | revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
| | metrics: |
| | - type: map |
| | value: 49.71580650644033 |
| | - type: mrr |
| | value: 50.50971903913081 |
| | - task: |
| | type: Summarization |
| | dataset: |
| | type: mteb/summeval |
| | name: MTEB SummEval |
| | config: default |
| | split: test |
| | revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 29.152190498799484 |
| | - type: cos_sim_spearman |
| | value: 29.686180371952727 |
| | - type: dot_pearson |
| | value: 27.248664793816342 |
| | - type: dot_spearman |
| | value: 28.37748983721745 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: trec-covid |
| | name: MTEB TRECCOVID |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 0.20400000000000001 |
| | - type: map_at_10 |
| | value: 1.6209999999999998 |
| | - type: map_at_100 |
| | value: 9.690999999999999 |
| | - type: map_at_1000 |
| | value: 23.733 |
| | - type: map_at_3 |
| | value: 0.575 |
| | - type: map_at_5 |
| | value: 0.885 |
| | - type: mrr_at_1 |
| | value: 78 |
| | - type: mrr_at_10 |
| | value: 86.56700000000001 |
| | - type: mrr_at_100 |
| | value: 86.56700000000001 |
| | - type: mrr_at_1000 |
| | value: 86.56700000000001 |
| | - type: mrr_at_3 |
| | value: 85.667 |
| | - type: mrr_at_5 |
| | value: 86.56700000000001 |
| | - type: ndcg_at_1 |
| | value: 76 |
| | - type: ndcg_at_10 |
| | value: 71.326 |
| | - type: ndcg_at_100 |
| | value: 54.208999999999996 |
| | - type: ndcg_at_1000 |
| | value: 49.252 |
| | - type: ndcg_at_3 |
| | value: 74.235 |
| | - type: ndcg_at_5 |
| | value: 73.833 |
| | - type: precision_at_1 |
| | value: 78 |
| | - type: precision_at_10 |
| | value: 74.8 |
| | - type: precision_at_100 |
| | value: 55.50000000000001 |
| | - type: precision_at_1000 |
| | value: 21.836 |
| | - type: precision_at_3 |
| | value: 78 |
| | - type: precision_at_5 |
| | value: 78 |
| | - type: recall_at_1 |
| | value: 0.20400000000000001 |
| | - type: recall_at_10 |
| | value: 1.894 |
| | - type: recall_at_100 |
| | value: 13.245999999999999 |
| | - type: recall_at_1000 |
| | value: 46.373 |
| | - type: recall_at_3 |
| | value: 0.613 |
| | - type: recall_at_5 |
| | value: 0.991 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (sqi-eng) |
| | config: sqi-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 95.89999999999999 |
| | - type: f1 |
| | value: 94.69999999999999 |
| | - type: precision |
| | value: 94.11666666666667 |
| | - type: recall |
| | value: 95.89999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (fry-eng) |
| | config: fry-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 68.20809248554913 |
| | - type: f1 |
| | value: 63.431048720066066 |
| | - type: precision |
| | value: 61.69143958161298 |
| | - type: recall |
| | value: 68.20809248554913 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (kur-eng) |
| | config: kur-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 71.21951219512195 |
| | - type: f1 |
| | value: 66.82926829268293 |
| | - type: precision |
| | value: 65.1260162601626 |
| | - type: recall |
| | value: 71.21951219512195 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tur-eng) |
| | config: tur-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97.2 |
| | - type: f1 |
| | value: 96.26666666666667 |
| | - type: precision |
| | value: 95.8 |
| | - type: recall |
| | value: 97.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (deu-eng) |
| | config: deu-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 99.3 |
| | - type: f1 |
| | value: 99.06666666666666 |
| | - type: precision |
| | value: 98.95 |
| | - type: recall |
| | value: 99.3 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (nld-eng) |
| | config: nld-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97.39999999999999 |
| | - type: f1 |
| | value: 96.63333333333333 |
| | - type: precision |
| | value: 96.26666666666668 |
| | - type: recall |
| | value: 97.39999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ron-eng) |
| | config: ron-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96 |
| | - type: f1 |
| | value: 94.86666666666666 |
| | - type: precision |
| | value: 94.31666666666668 |
| | - type: recall |
| | value: 96 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ang-eng) |
| | config: ang-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 47.01492537313433 |
| | - type: f1 |
| | value: 40.178867566927266 |
| | - type: precision |
| | value: 38.179295828549556 |
| | - type: recall |
| | value: 47.01492537313433 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ido-eng) |
| | config: ido-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 86.5 |
| | - type: f1 |
| | value: 83.62537480063796 |
| | - type: precision |
| | value: 82.44555555555554 |
| | - type: recall |
| | value: 86.5 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (jav-eng) |
| | config: jav-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 80.48780487804879 |
| | - type: f1 |
| | value: 75.45644599303138 |
| | - type: precision |
| | value: 73.37398373983739 |
| | - type: recall |
| | value: 80.48780487804879 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (isl-eng) |
| | config: isl-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 93.7 |
| | - type: f1 |
| | value: 91.95666666666666 |
| | - type: precision |
| | value: 91.125 |
| | - type: recall |
| | value: 93.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (slv-eng) |
| | config: slv-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 91.73754556500607 |
| | - type: f1 |
| | value: 89.65168084244632 |
| | - type: precision |
| | value: 88.73025516403402 |
| | - type: recall |
| | value: 91.73754556500607 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (cym-eng) |
| | config: cym-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 81.04347826086956 |
| | - type: f1 |
| | value: 76.2128364389234 |
| | - type: precision |
| | value: 74.2 |
| | - type: recall |
| | value: 81.04347826086956 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (kaz-eng) |
| | config: kaz-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 83.65217391304348 |
| | - type: f1 |
| | value: 79.4376811594203 |
| | - type: precision |
| | value: 77.65797101449274 |
| | - type: recall |
| | value: 83.65217391304348 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (est-eng) |
| | config: est-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 87.5 |
| | - type: f1 |
| | value: 85.02690476190476 |
| | - type: precision |
| | value: 83.96261904761904 |
| | - type: recall |
| | value: 87.5 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (heb-eng) |
| | config: heb-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 89.3 |
| | - type: f1 |
| | value: 86.52333333333333 |
| | - type: precision |
| | value: 85.22833333333332 |
| | - type: recall |
| | value: 89.3 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (gla-eng) |
| | config: gla-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 65.01809408926418 |
| | - type: f1 |
| | value: 59.00594446432805 |
| | - type: precision |
| | value: 56.827215807915444 |
| | - type: recall |
| | value: 65.01809408926418 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (mar-eng) |
| | config: mar-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 91.2 |
| | - type: f1 |
| | value: 88.58 |
| | - type: precision |
| | value: 87.33333333333334 |
| | - type: recall |
| | value: 91.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (lat-eng) |
| | config: lat-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 59.199999999999996 |
| | - type: f1 |
| | value: 53.299166276284915 |
| | - type: precision |
| | value: 51.3383908045977 |
| | - type: recall |
| | value: 59.199999999999996 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (bel-eng) |
| | config: bel-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 93.2 |
| | - type: f1 |
| | value: 91.2 |
| | - type: precision |
| | value: 90.25 |
| | - type: recall |
| | value: 93.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (pms-eng) |
| | config: pms-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 64.76190476190476 |
| | - type: f1 |
| | value: 59.867110667110666 |
| | - type: precision |
| | value: 58.07390192653351 |
| | - type: recall |
| | value: 64.76190476190476 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (gle-eng) |
| | config: gle-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 76.2 |
| | - type: f1 |
| | value: 71.48147546897547 |
| | - type: precision |
| | value: 69.65409090909091 |
| | - type: recall |
| | value: 76.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (pes-eng) |
| | config: pes-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 93.8 |
| | - type: f1 |
| | value: 92.14 |
| | - type: precision |
| | value: 91.35833333333333 |
| | - type: recall |
| | value: 93.8 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (nob-eng) |
| | config: nob-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97.89999999999999 |
| | - type: f1 |
| | value: 97.2 |
| | - type: precision |
| | value: 96.85000000000001 |
| | - type: recall |
| | value: 97.89999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (bul-eng) |
| | config: bul-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.6 |
| | - type: f1 |
| | value: 92.93333333333334 |
| | - type: precision |
| | value: 92.13333333333333 |
| | - type: recall |
| | value: 94.6 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (cbk-eng) |
| | config: cbk-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 74.1 |
| | - type: f1 |
| | value: 69.14817460317461 |
| | - type: precision |
| | value: 67.2515873015873 |
| | - type: recall |
| | value: 74.1 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (hun-eng) |
| | config: hun-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 95.19999999999999 |
| | - type: f1 |
| | value: 94.01333333333335 |
| | - type: precision |
| | value: 93.46666666666667 |
| | - type: recall |
| | value: 95.19999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (uig-eng) |
| | config: uig-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 76.9 |
| | - type: f1 |
| | value: 72.07523809523809 |
| | - type: precision |
| | value: 70.19777777777779 |
| | - type: recall |
| | value: 76.9 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (rus-eng) |
| | config: rus-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.1 |
| | - type: f1 |
| | value: 92.31666666666666 |
| | - type: precision |
| | value: 91.43333333333332 |
| | - type: recall |
| | value: 94.1 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (spa-eng) |
| | config: spa-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97.8 |
| | - type: f1 |
| | value: 97.1 |
| | - type: precision |
| | value: 96.76666666666668 |
| | - type: recall |
| | value: 97.8 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (hye-eng) |
| | config: hye-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 92.85714285714286 |
| | - type: f1 |
| | value: 90.92093441150045 |
| | - type: precision |
| | value: 90.00449236298293 |
| | - type: recall |
| | value: 92.85714285714286 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tel-eng) |
| | config: tel-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 93.16239316239316 |
| | - type: f1 |
| | value: 91.33903133903132 |
| | - type: precision |
| | value: 90.56267806267806 |
| | - type: recall |
| | value: 93.16239316239316 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (afr-eng) |
| | config: afr-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 92.4 |
| | - type: f1 |
| | value: 90.25666666666666 |
| | - type: precision |
| | value: 89.25833333333334 |
| | - type: recall |
| | value: 92.4 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (mon-eng) |
| | config: mon-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 90.22727272727272 |
| | - type: f1 |
| | value: 87.53030303030303 |
| | - type: precision |
| | value: 86.37121212121211 |
| | - type: recall |
| | value: 90.22727272727272 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (arz-eng) |
| | config: arz-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 79.03563941299791 |
| | - type: f1 |
| | value: 74.7349505840072 |
| | - type: precision |
| | value: 72.9035639412998 |
| | - type: recall |
| | value: 79.03563941299791 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (hrv-eng) |
| | config: hrv-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97 |
| | - type: f1 |
| | value: 96.15 |
| | - type: precision |
| | value: 95.76666666666668 |
| | - type: recall |
| | value: 97 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (nov-eng) |
| | config: nov-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 76.26459143968872 |
| | - type: f1 |
| | value: 71.55642023346303 |
| | - type: precision |
| | value: 69.7544932369835 |
| | - type: recall |
| | value: 76.26459143968872 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (gsw-eng) |
| | config: gsw-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 58.119658119658126 |
| | - type: f1 |
| | value: 51.65242165242165 |
| | - type: precision |
| | value: 49.41768108434775 |
| | - type: recall |
| | value: 58.119658119658126 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (nds-eng) |
| | config: nds-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 74.3 |
| | - type: f1 |
| | value: 69.52055555555555 |
| | - type: precision |
| | value: 67.7574938949939 |
| | - type: recall |
| | value: 74.3 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ukr-eng) |
| | config: ukr-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.8 |
| | - type: f1 |
| | value: 93.31666666666666 |
| | - type: precision |
| | value: 92.60000000000001 |
| | - type: recall |
| | value: 94.8 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (uzb-eng) |
| | config: uzb-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 76.63551401869158 |
| | - type: f1 |
| | value: 72.35202492211837 |
| | - type: precision |
| | value: 70.60358255451713 |
| | - type: recall |
| | value: 76.63551401869158 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (lit-eng) |
| | config: lit-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 90.4 |
| | - type: f1 |
| | value: 88.4811111111111 |
| | - type: precision |
| | value: 87.7452380952381 |
| | - type: recall |
| | value: 90.4 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ina-eng) |
| | config: ina-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 95 |
| | - type: f1 |
| | value: 93.60666666666667 |
| | - type: precision |
| | value: 92.975 |
| | - type: recall |
| | value: 95 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (lfn-eng) |
| | config: lfn-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 67.2 |
| | - type: f1 |
| | value: 63.01595782872099 |
| | - type: precision |
| | value: 61.596587301587306 |
| | - type: recall |
| | value: 67.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (zsm-eng) |
| | config: zsm-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 95.7 |
| | - type: f1 |
| | value: 94.52999999999999 |
| | - type: precision |
| | value: 94 |
| | - type: recall |
| | value: 95.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ita-eng) |
| | config: ita-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.6 |
| | - type: f1 |
| | value: 93.28999999999999 |
| | - type: precision |
| | value: 92.675 |
| | - type: recall |
| | value: 94.6 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (cmn-eng) |
| | config: cmn-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.39999999999999 |
| | - type: f1 |
| | value: 95.28333333333333 |
| | - type: precision |
| | value: 94.75 |
| | - type: recall |
| | value: 96.39999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (lvs-eng) |
| | config: lvs-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 91.9 |
| | - type: f1 |
| | value: 89.83 |
| | - type: precision |
| | value: 88.92 |
| | - type: recall |
| | value: 91.9 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (glg-eng) |
| | config: glg-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.69999999999999 |
| | - type: f1 |
| | value: 93.34222222222223 |
| | - type: precision |
| | value: 92.75416666666668 |
| | - type: recall |
| | value: 94.69999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ceb-eng) |
| | config: ceb-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 60.333333333333336 |
| | - type: f1 |
| | value: 55.31203703703703 |
| | - type: precision |
| | value: 53.39971108326371 |
| | - type: recall |
| | value: 60.333333333333336 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (bre-eng) |
| | config: bre-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 12.9 |
| | - type: f1 |
| | value: 11.099861903031458 |
| | - type: precision |
| | value: 10.589187932631877 |
| | - type: recall |
| | value: 12.9 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ben-eng) |
| | config: ben-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 86.7 |
| | - type: f1 |
| | value: 83.0152380952381 |
| | - type: precision |
| | value: 81.37833333333333 |
| | - type: recall |
| | value: 86.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (swg-eng) |
| | config: swg-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 63.39285714285714 |
| | - type: f1 |
| | value: 56.832482993197274 |
| | - type: precision |
| | value: 54.56845238095237 |
| | - type: recall |
| | value: 63.39285714285714 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (arq-eng) |
| | config: arq-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 48.73765093304062 |
| | - type: f1 |
| | value: 41.555736920720456 |
| | - type: precision |
| | value: 39.06874531737319 |
| | - type: recall |
| | value: 48.73765093304062 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (kab-eng) |
| | config: kab-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 41.099999999999994 |
| | - type: f1 |
| | value: 36.540165945165946 |
| | - type: precision |
| | value: 35.05175685425686 |
| | - type: recall |
| | value: 41.099999999999994 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (fra-eng) |
| | config: fra-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.89999999999999 |
| | - type: f1 |
| | value: 93.42333333333333 |
| | - type: precision |
| | value: 92.75833333333333 |
| | - type: recall |
| | value: 94.89999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (por-eng) |
| | config: por-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.89999999999999 |
| | - type: f1 |
| | value: 93.63333333333334 |
| | - type: precision |
| | value: 93.01666666666665 |
| | - type: recall |
| | value: 94.89999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tat-eng) |
| | config: tat-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 77.9 |
| | - type: f1 |
| | value: 73.64833333333334 |
| | - type: precision |
| | value: 71.90282106782105 |
| | - type: recall |
| | value: 77.9 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (oci-eng) |
| | config: oci-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 59.4 |
| | - type: f1 |
| | value: 54.90521367521367 |
| | - type: precision |
| | value: 53.432840025471606 |
| | - type: recall |
| | value: 59.4 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (pol-eng) |
| | config: pol-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97.39999999999999 |
| | - type: f1 |
| | value: 96.6 |
| | - type: precision |
| | value: 96.2 |
| | - type: recall |
| | value: 97.39999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (war-eng) |
| | config: war-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 67.2 |
| | - type: f1 |
| | value: 62.25926129426129 |
| | - type: precision |
| | value: 60.408376623376626 |
| | - type: recall |
| | value: 67.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (aze-eng) |
| | config: aze-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 90.2 |
| | - type: f1 |
| | value: 87.60666666666667 |
| | - type: precision |
| | value: 86.45277777777778 |
| | - type: recall |
| | value: 90.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (vie-eng) |
| | config: vie-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 97.7 |
| | - type: f1 |
| | value: 97 |
| | - type: precision |
| | value: 96.65 |
| | - type: recall |
| | value: 97.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (nno-eng) |
| | config: nno-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 93.2 |
| | - type: f1 |
| | value: 91.39746031746031 |
| | - type: precision |
| | value: 90.6125 |
| | - type: recall |
| | value: 93.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (cha-eng) |
| | config: cha-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 32.11678832116788 |
| | - type: f1 |
| | value: 27.210415386260234 |
| | - type: precision |
| | value: 26.20408990846947 |
| | - type: recall |
| | value: 32.11678832116788 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (mhr-eng) |
| | config: mhr-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 8.5 |
| | - type: f1 |
| | value: 6.787319277832475 |
| | - type: precision |
| | value: 6.3452094433344435 |
| | - type: recall |
| | value: 8.5 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (dan-eng) |
| | config: dan-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.1 |
| | - type: f1 |
| | value: 95.08 |
| | - type: precision |
| | value: 94.61666666666667 |
| | - type: recall |
| | value: 96.1 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ell-eng) |
| | config: ell-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 95.3 |
| | - type: f1 |
| | value: 93.88333333333333 |
| | - type: precision |
| | value: 93.18333333333332 |
| | - type: recall |
| | value: 95.3 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (amh-eng) |
| | config: amh-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 85.11904761904762 |
| | - type: f1 |
| | value: 80.69444444444444 |
| | - type: precision |
| | value: 78.72023809523809 |
| | - type: recall |
| | value: 85.11904761904762 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (pam-eng) |
| | config: pam-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 11.1 |
| | - type: f1 |
| | value: 9.276381801735853 |
| | - type: precision |
| | value: 8.798174603174601 |
| | - type: recall |
| | value: 11.1 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (hsb-eng) |
| | config: hsb-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 63.56107660455487 |
| | - type: f1 |
| | value: 58.70433569191332 |
| | - type: precision |
| | value: 56.896926581464015 |
| | - type: recall |
| | value: 63.56107660455487 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (srp-eng) |
| | config: srp-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.69999999999999 |
| | - type: f1 |
| | value: 93.10000000000001 |
| | - type: precision |
| | value: 92.35 |
| | - type: recall |
| | value: 94.69999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (epo-eng) |
| | config: epo-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.8 |
| | - type: f1 |
| | value: 96.01222222222222 |
| | - type: precision |
| | value: 95.67083333333332 |
| | - type: recall |
| | value: 96.8 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (kzj-eng) |
| | config: kzj-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 9.2 |
| | - type: f1 |
| | value: 7.911555250305249 |
| | - type: precision |
| | value: 7.631246556216846 |
| | - type: recall |
| | value: 9.2 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (awa-eng) |
| | config: awa-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 77.48917748917748 |
| | - type: f1 |
| | value: 72.27375798804371 |
| | - type: precision |
| | value: 70.14430014430013 |
| | - type: recall |
| | value: 77.48917748917748 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (fao-eng) |
| | config: fao-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 77.09923664122137 |
| | - type: f1 |
| | value: 72.61541257724463 |
| | - type: precision |
| | value: 70.8998380754106 |
| | - type: recall |
| | value: 77.09923664122137 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (mal-eng) |
| | config: mal-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 98.2532751091703 |
| | - type: f1 |
| | value: 97.69529354682193 |
| | - type: precision |
| | value: 97.42843279961184 |
| | - type: recall |
| | value: 98.2532751091703 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ile-eng) |
| | config: ile-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 82.8 |
| | - type: f1 |
| | value: 79.14672619047619 |
| | - type: precision |
| | value: 77.59489247311828 |
| | - type: recall |
| | value: 82.8 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (bos-eng) |
| | config: bos-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.35028248587571 |
| | - type: f1 |
| | value: 92.86252354048965 |
| | - type: precision |
| | value: 92.2080979284369 |
| | - type: recall |
| | value: 94.35028248587571 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (cor-eng) |
| | config: cor-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 8.5 |
| | - type: f1 |
| | value: 6.282429263935621 |
| | - type: precision |
| | value: 5.783274240739785 |
| | - type: recall |
| | value: 8.5 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (cat-eng) |
| | config: cat-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 92.7 |
| | - type: f1 |
| | value: 91.025 |
| | - type: precision |
| | value: 90.30428571428571 |
| | - type: recall |
| | value: 92.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (eus-eng) |
| | config: eus-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 81 |
| | - type: f1 |
| | value: 77.8232380952381 |
| | - type: precision |
| | value: 76.60194444444444 |
| | - type: recall |
| | value: 81 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (yue-eng) |
| | config: yue-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 91 |
| | - type: f1 |
| | value: 88.70857142857142 |
| | - type: precision |
| | value: 87.7 |
| | - type: recall |
| | value: 91 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (swe-eng) |
| | config: swe-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.39999999999999 |
| | - type: f1 |
| | value: 95.3 |
| | - type: precision |
| | value: 94.76666666666667 |
| | - type: recall |
| | value: 96.39999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (dtp-eng) |
| | config: dtp-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 8.1 |
| | - type: f1 |
| | value: 7.001008218834307 |
| | - type: precision |
| | value: 6.708329562594269 |
| | - type: recall |
| | value: 8.1 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (kat-eng) |
| | config: kat-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 87.1313672922252 |
| | - type: f1 |
| | value: 84.09070598748882 |
| | - type: precision |
| | value: 82.79171454104429 |
| | - type: recall |
| | value: 87.1313672922252 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (jpn-eng) |
| | config: jpn-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.39999999999999 |
| | - type: f1 |
| | value: 95.28333333333333 |
| | - type: precision |
| | value: 94.73333333333332 |
| | - type: recall |
| | value: 96.39999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (csb-eng) |
| | config: csb-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 42.29249011857708 |
| | - type: f1 |
| | value: 36.981018542283365 |
| | - type: precision |
| | value: 35.415877813576024 |
| | - type: recall |
| | value: 42.29249011857708 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (xho-eng) |
| | config: xho-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 83.80281690140845 |
| | - type: f1 |
| | value: 80.86854460093896 |
| | - type: precision |
| | value: 79.60093896713614 |
| | - type: recall |
| | value: 83.80281690140845 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (orv-eng) |
| | config: orv-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 45.26946107784431 |
| | - type: f1 |
| | value: 39.80235464678088 |
| | - type: precision |
| | value: 38.14342660001342 |
| | - type: recall |
| | value: 45.26946107784431 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ind-eng) |
| | config: ind-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.3 |
| | - type: f1 |
| | value: 92.9 |
| | - type: precision |
| | value: 92.26666666666668 |
| | - type: recall |
| | value: 94.3 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tuk-eng) |
| | config: tuk-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 37.93103448275862 |
| | - type: f1 |
| | value: 33.15192743764172 |
| | - type: precision |
| | value: 31.57456528146183 |
| | - type: recall |
| | value: 37.93103448275862 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (max-eng) |
| | config: max-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 69.01408450704226 |
| | - type: f1 |
| | value: 63.41549295774648 |
| | - type: precision |
| | value: 61.342778895595806 |
| | - type: recall |
| | value: 69.01408450704226 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (swh-eng) |
| | config: swh-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 76.66666666666667 |
| | - type: f1 |
| | value: 71.60705960705961 |
| | - type: precision |
| | value: 69.60683760683762 |
| | - type: recall |
| | value: 76.66666666666667 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (hin-eng) |
| | config: hin-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 95.8 |
| | - type: f1 |
| | value: 94.48333333333333 |
| | - type: precision |
| | value: 93.83333333333333 |
| | - type: recall |
| | value: 95.8 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (dsb-eng) |
| | config: dsb-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 52.81837160751566 |
| | - type: f1 |
| | value: 48.435977731384824 |
| | - type: precision |
| | value: 47.11291973845539 |
| | - type: recall |
| | value: 52.81837160751566 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ber-eng) |
| | config: ber-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 44.9 |
| | - type: f1 |
| | value: 38.88962621607783 |
| | - type: precision |
| | value: 36.95936507936508 |
| | - type: recall |
| | value: 44.9 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tam-eng) |
| | config: tam-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 90.55374592833876 |
| | - type: f1 |
| | value: 88.22553125484721 |
| | - type: precision |
| | value: 87.26927252985884 |
| | - type: recall |
| | value: 90.55374592833876 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (slk-eng) |
| | config: slk-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 94.6 |
| | - type: f1 |
| | value: 93.13333333333333 |
| | - type: precision |
| | value: 92.45333333333333 |
| | - type: recall |
| | value: 94.6 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tgl-eng) |
| | config: tgl-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 93.7 |
| | - type: f1 |
| | value: 91.99666666666667 |
| | - type: precision |
| | value: 91.26666666666668 |
| | - type: recall |
| | value: 93.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ast-eng) |
| | config: ast-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 85.03937007874016 |
| | - type: f1 |
| | value: 81.75853018372703 |
| | - type: precision |
| | value: 80.34120734908137 |
| | - type: recall |
| | value: 85.03937007874016 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (mkd-eng) |
| | config: mkd-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 88.3 |
| | - type: f1 |
| | value: 85.5 |
| | - type: precision |
| | value: 84.25833333333334 |
| | - type: recall |
| | value: 88.3 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (khm-eng) |
| | config: khm-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 65.51246537396122 |
| | - type: f1 |
| | value: 60.02297410192148 |
| | - type: precision |
| | value: 58.133467727289236 |
| | - type: recall |
| | value: 65.51246537396122 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ces-eng) |
| | config: ces-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96 |
| | - type: f1 |
| | value: 94.89 |
| | - type: precision |
| | value: 94.39166666666667 |
| | - type: recall |
| | value: 96 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tzl-eng) |
| | config: tzl-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 57.692307692307686 |
| | - type: f1 |
| | value: 53.162393162393165 |
| | - type: precision |
| | value: 51.70673076923077 |
| | - type: recall |
| | value: 57.692307692307686 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (urd-eng) |
| | config: urd-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 91.60000000000001 |
| | - type: f1 |
| | value: 89.21190476190475 |
| | - type: precision |
| | value: 88.08666666666667 |
| | - type: recall |
| | value: 91.60000000000001 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (ara-eng) |
| | config: ara-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 88 |
| | - type: f1 |
| | value: 85.47 |
| | - type: precision |
| | value: 84.43266233766234 |
| | - type: recall |
| | value: 88 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (kor-eng) |
| | config: kor-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 92.7 |
| | - type: f1 |
| | value: 90.64999999999999 |
| | - type: precision |
| | value: 89.68333333333332 |
| | - type: recall |
| | value: 92.7 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (yid-eng) |
| | config: yid-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 80.30660377358491 |
| | - type: f1 |
| | value: 76.33044137466307 |
| | - type: precision |
| | value: 74.78970125786164 |
| | - type: recall |
| | value: 80.30660377358491 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (fin-eng) |
| | config: fin-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.39999999999999 |
| | - type: f1 |
| | value: 95.44 |
| | - type: precision |
| | value: 94.99166666666666 |
| | - type: recall |
| | value: 96.39999999999999 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (tha-eng) |
| | config: tha-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 96.53284671532847 |
| | - type: f1 |
| | value: 95.37712895377129 |
| | - type: precision |
| | value: 94.7992700729927 |
| | - type: recall |
| | value: 96.53284671532847 |
| | - task: |
| | type: BitextMining |
| | dataset: |
| | type: mteb/tatoeba-bitext-mining |
| | name: MTEB Tatoeba (wuu-eng) |
| | config: wuu-eng |
| | split: test |
| | revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 |
| | metrics: |
| | - type: accuracy |
| | value: 89 |
| | - type: f1 |
| | value: 86.23190476190476 |
| | - type: precision |
| | value: 85.035 |
| | - type: recall |
| | value: 89 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: webis-touche2020 |
| | name: MTEB Touche2020 |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 2.585 |
| | - type: map_at_10 |
| | value: 9.012 |
| | - type: map_at_100 |
| | value: 14.027000000000001 |
| | - type: map_at_1000 |
| | value: 15.565000000000001 |
| | - type: map_at_3 |
| | value: 5.032 |
| | - type: map_at_5 |
| | value: 6.657 |
| | - type: mrr_at_1 |
| | value: 28.571 |
| | - type: mrr_at_10 |
| | value: 45.377 |
| | - type: mrr_at_100 |
| | value: 46.119 |
| | - type: mrr_at_1000 |
| | value: 46.127 |
| | - type: mrr_at_3 |
| | value: 41.156 |
| | - type: mrr_at_5 |
| | value: 42.585 |
| | - type: ndcg_at_1 |
| | value: 27.551 |
| | - type: ndcg_at_10 |
| | value: 23.395 |
| | - type: ndcg_at_100 |
| | value: 33.342 |
| | - type: ndcg_at_1000 |
| | value: 45.523 |
| | - type: ndcg_at_3 |
| | value: 25.158 |
| | - type: ndcg_at_5 |
| | value: 23.427 |
| | - type: precision_at_1 |
| | value: 28.571 |
| | - type: precision_at_10 |
| | value: 21.429000000000002 |
| | - type: precision_at_100 |
| | value: 6.714 |
| | - type: precision_at_1000 |
| | value: 1.473 |
| | - type: precision_at_3 |
| | value: 27.211000000000002 |
| | - type: precision_at_5 |
| | value: 24.490000000000002 |
| | - type: recall_at_1 |
| | value: 2.585 |
| | - type: recall_at_10 |
| | value: 15.418999999999999 |
| | - type: recall_at_100 |
| | value: 42.485 |
| | - type: recall_at_1000 |
| | value: 79.536 |
| | - type: recall_at_3 |
| | value: 6.239999999999999 |
| | - type: recall_at_5 |
| | value: 8.996 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/toxic_conversations_50k |
| | name: MTEB ToxicConversationsClassification |
| | config: default |
| | split: test |
| | revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
| | metrics: |
| | - type: accuracy |
| | value: 71.3234 |
| | - type: ap |
| | value: 14.361688653847423 |
| | - type: f1 |
| | value: 54.819068624319044 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/tweet_sentiment_extraction |
| | name: MTEB TweetSentimentExtractionClassification |
| | config: default |
| | split: test |
| | revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
| | metrics: |
| | - type: accuracy |
| | value: 61.97792869269949 |
| | - type: f1 |
| | value: 62.28965628513728 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/twentynewsgroups-clustering |
| | name: MTEB TwentyNewsgroupsClustering |
| | config: default |
| | split: test |
| | revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
| | metrics: |
| | - type: v_measure |
| | value: 38.90540145385218 |
| | - task: |
| | type: PairClassification |
| | dataset: |
| | type: mteb/twittersemeval2015-pairclassification |
| | name: MTEB TwitterSemEval2015 |
| | config: default |
| | split: test |
| | revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
| | metrics: |
| | - type: cos_sim_accuracy |
| | value: 86.53513739047506 |
| | - type: cos_sim_ap |
| | value: 75.27741586677557 |
| | - type: cos_sim_f1 |
| | value: 69.18792902473774 |
| | - type: cos_sim_precision |
| | value: 67.94708725515136 |
| | - type: cos_sim_recall |
| | value: 70.47493403693932 |
| | - type: dot_accuracy |
| | value: 84.7052512368123 |
| | - type: dot_ap |
| | value: 69.36075482849378 |
| | - type: dot_f1 |
| | value: 64.44688376631296 |
| | - type: dot_precision |
| | value: 59.92288500793831 |
| | - type: dot_recall |
| | value: 69.70976253298153 |
| | - type: euclidean_accuracy |
| | value: 86.60666388508076 |
| | - type: euclidean_ap |
| | value: 75.47512772621097 |
| | - type: euclidean_f1 |
| | value: 69.413872536473 |
| | - type: euclidean_precision |
| | value: 67.39562624254472 |
| | - type: euclidean_recall |
| | value: 71.55672823218997 |
| | - type: manhattan_accuracy |
| | value: 86.52917684925792 |
| | - type: manhattan_ap |
| | value: 75.34000110496703 |
| | - type: manhattan_f1 |
| | value: 69.28489190226429 |
| | - type: manhattan_precision |
| | value: 67.24608889992551 |
| | - type: manhattan_recall |
| | value: 71.45118733509234 |
| | - type: max_accuracy |
| | value: 86.60666388508076 |
| | - type: max_ap |
| | value: 75.47512772621097 |
| | - type: max_f1 |
| | value: 69.413872536473 |
| | - task: |
| | type: PairClassification |
| | dataset: |
| | type: mteb/twitterurlcorpus-pairclassification |
| | name: MTEB TwitterURLCorpus |
| | config: default |
| | split: test |
| | revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
| | metrics: |
| | - type: cos_sim_accuracy |
| | value: 89.01695967710637 |
| | - type: cos_sim_ap |
| | value: 85.8298270742901 |
| | - type: cos_sim_f1 |
| | value: 78.46988128389272 |
| | - type: cos_sim_precision |
| | value: 74.86017897091722 |
| | - type: cos_sim_recall |
| | value: 82.44533415460425 |
| | - type: dot_accuracy |
| | value: 88.19420188613343 |
| | - type: dot_ap |
| | value: 83.82679165901324 |
| | - type: dot_f1 |
| | value: 76.55833777304208 |
| | - type: dot_precision |
| | value: 75.6884875846501 |
| | - type: dot_recall |
| | value: 77.44841392054204 |
| | - type: euclidean_accuracy |
| | value: 89.03054294252338 |
| | - type: euclidean_ap |
| | value: 85.89089555185325 |
| | - type: euclidean_f1 |
| | value: 78.62997658079624 |
| | - type: euclidean_precision |
| | value: 74.92329149232914 |
| | - type: euclidean_recall |
| | value: 82.72251308900523 |
| | - type: manhattan_accuracy |
| | value: 89.0266620095471 |
| | - type: manhattan_ap |
| | value: 85.86458997929147 |
| | - type: manhattan_f1 |
| | value: 78.50685331000291 |
| | - type: manhattan_precision |
| | value: 74.5499861534201 |
| | - type: manhattan_recall |
| | value: 82.90729904527257 |
| | - type: max_accuracy |
| | value: 89.03054294252338 |
| | - type: max_ap |
| | value: 85.89089555185325 |
| | - type: max_f1 |
| | value: 78.62997658079624 |
| | language: |
| | - multilingual |
| | - af |
| | - am |
| | - ar |
| | - as |
| | - az |
| | - be |
| | - bg |
| | - bn |
| | - br |
| | - bs |
| | - ca |
| | - cs |
| | - cy |
| | - da |
| | - de |
| | - el |
| | - en |
| | - eo |
| | - es |
| | - et |
| | - eu |
| | - fa |
| | - fi |
| | - fr |
| | - fy |
| | - ga |
| | - gd |
| | - gl |
| | - gu |
| | - ha |
| | - he |
| | - hi |
| | - hr |
| | - hu |
| | - hy |
| | - id |
| | - is |
| | - it |
| | - ja |
| | - jv |
| | - ka |
| | - kk |
| | - km |
| | - kn |
| | - ko |
| | - ku |
| | - ky |
| | - la |
| | - lo |
| | - lt |
| | - lv |
| | - mg |
| | - mk |
| | - ml |
| | - mn |
| | - mr |
| | - ms |
| | - my |
| | - ne |
| | - nl |
| | - 'no' |
| | - om |
| | - or |
| | - pa |
| | - pl |
| | - ps |
| | - pt |
| | - ro |
| | - ru |
| | - sa |
| | - sd |
| | - si |
| | - sk |
| | - sl |
| | - so |
| | - sq |
| | - sr |
| | - su |
| | - sv |
| | - sw |
| | - ta |
| | - te |
| | - th |
| | - tl |
| | - tr |
| | - ug |
| | - uk |
| | - ur |
| | - uz |
| | - vi |
| | - xh |
| | - yi |
| | - zh |
| | license: mit |
| | --- |
| | |
| | ## Multilingual-E5-large |
| |
|
| | [Multilingual E5 Text Embeddings: A Technical Report](https://arxiv.org/pdf/2402.05672). |
| | Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024 |
| |
|
| | This model has 24 layers and the embedding size is 1024. |
| |
|
| | ## Usage |
| |
|
| | Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset. |
| |
|
| | ```python |
| | import torch.nn.functional as F |
| | |
| | from torch import Tensor |
| | from transformers import AutoTokenizer, AutoModel |
| | |
| | |
| | def average_pool(last_hidden_states: Tensor, |
| | attention_mask: Tensor) -> Tensor: |
| | last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0) |
| | return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] |
| | |
| | |
| | # Each input text should start with "query: " or "passage: ", even for non-English texts. |
| | # For tasks other than retrieval, you can simply use the "query: " prefix. |
| | input_texts = ['query: how much protein should a female eat', |
| | 'query: 南瓜的家常做法', |
| | "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", |
| | "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"] |
| | |
| | tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-large') |
| | model = AutoModel.from_pretrained('intfloat/multilingual-e5-large') |
| | |
| | # Tokenize the input texts |
| | batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt') |
| | |
| | outputs = model(**batch_dict) |
| | embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask']) |
| | |
| | # normalize embeddings |
| | embeddings = F.normalize(embeddings, p=2, dim=1) |
| | scores = (embeddings[:2] @ embeddings[2:].T) * 100 |
| | print(scores.tolist()) |
| | ``` |
| |
|
| | ## Supported Languages |
| |
|
| | This model is initialized from [xlm-roberta-large](https://huggingface.co/xlm-roberta-large) |
| | and continually trained on a mixture of multilingual datasets. |
| | It supports 100 languages from xlm-roberta, |
| | but low-resource languages may see performance degradation. |
| |
|
| | ## Training Details |
| |
|
| | **Initialization**: [xlm-roberta-large](https://huggingface.co/xlm-roberta-large) |
| |
|
| | **First stage**: contrastive pre-training with weak supervision |
| |
|
| | | Dataset | Weak supervision | # of text pairs | |
| | |--------------------------------------------------------------------------------------------------------|---------------------------------------|-----------------| |
| | | Filtered [mC4](https://huggingface.co/datasets/mc4) | (title, page content) | 1B | |
| | | [CC News](https://huggingface.co/datasets/intfloat/multilingual_cc_news) | (title, news content) | 400M | |
| | | [NLLB](https://huggingface.co/datasets/allenai/nllb) | translation pairs | 2.4B | |
| | | [Wikipedia](https://huggingface.co/datasets/intfloat/wikipedia) | (hierarchical section title, passage) | 150M | |
| | | Filtered [Reddit](https://www.reddit.com/) | (comment, response) | 800M | |
| | | [S2ORC](https://github.com/allenai/s2orc) | (title, abstract) and citation pairs | 100M | |
| | | [Stackexchange](https://stackexchange.com/) | (question, answer) | 50M | |
| | | [xP3](https://huggingface.co/datasets/bigscience/xP3) | (input prompt, response) | 80M | |
| | | [Miscellaneous unsupervised SBERT data](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | - | 10M | |
| |
|
| | **Second stage**: supervised fine-tuning |
| |
|
| | | Dataset | Language | # of text pairs | |
| | |----------------------------------------------------------------------------------------|--------------|-----------------| |
| | | [MS MARCO](https://microsoft.github.io/msmarco/) | English | 500k | |
| | | [NQ](https://github.com/facebookresearch/DPR) | English | 70k | |
| | | [Trivia QA](https://github.com/facebookresearch/DPR) | English | 60k | |
| | | [NLI from SimCSE](https://github.com/princeton-nlp/SimCSE) | English | <300k | |
| | | [ELI5](https://huggingface.co/datasets/eli5) | English | 500k | |
| | | [DuReader Retrieval](https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval) | Chinese | 86k | |
| | | [KILT Fever](https://huggingface.co/datasets/kilt_tasks) | English | 70k | |
| | | [KILT HotpotQA](https://huggingface.co/datasets/kilt_tasks) | English | 70k | |
| | | [SQuAD](https://huggingface.co/datasets/squad) | English | 87k | |
| | | [Quora](https://huggingface.co/datasets/quora) | English | 150k | |
| | | [Mr. TyDi](https://huggingface.co/datasets/castorini/mr-tydi) | 11 languages | 50k | |
| | | [MIRACL](https://huggingface.co/datasets/miracl/miracl) | 16 languages | 40k | |
| |
|
| | For all labeled datasets, we only use its training set for fine-tuning. |
| |
|
| | For other training details, please refer to our paper at [https://arxiv.org/pdf/2402.05672](https://arxiv.org/pdf/2402.05672). |
| |
|
| | ## Benchmark Results on [Mr. TyDi](https://arxiv.org/abs/2108.08787) |
| |
|
| | | Model | Avg MRR@10 | | ar | bn | en | fi | id | ja | ko | ru | sw | te | th | |
| | |-----------------------|------------|-------|------| --- | --- | --- | --- | --- | --- | --- |------| --- | --- | |
| | | BM25 | 33.3 | | 36.7 | 41.3 | 15.1 | 28.8 | 38.2 | 21.7 | 28.1 | 32.9 | 39.6 | 42.4 | 41.7 | |
| | | mDPR | 16.7 | | 26.0 | 25.8 | 16.2 | 11.3 | 14.6 | 18.1 | 21.9 | 18.5 | 7.3 | 10.6 | 13.5 | |
| | | BM25 + mDPR | 41.7 | | 49.1 | 53.5 | 28.4 | 36.5 | 45.5 | 35.5 | 36.2 | 42.7 | 40.5 | 42.0 | 49.2 | |
| | | | | |
| | | multilingual-e5-small | 64.4 | | 71.5 | 66.3 | 54.5 | 57.7 | 63.2 | 55.4 | 54.3 | 60.8 | 65.4 | 89.1 | 70.1 | |
| | | multilingual-e5-base | 65.9 | | 72.3 | 65.0 | 58.5 | 60.8 | 64.9 | 56.6 | 55.8 | 62.7 | 69.0 | 86.6 | 72.7 | |
| | | multilingual-e5-large | **70.5** | | 77.5 | 73.2 | 60.8 | 66.8 | 68.5 | 62.5 | 61.6 | 65.8 | 72.7 | 90.2 | 76.2 | |
| |
|
| | ## MTEB Benchmark Evaluation |
| |
|
| | Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results |
| | on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316). |
| |
|
| | ## Support for Sentence Transformers |
| |
|
| | Below is an example for usage with sentence_transformers. |
| | ```python |
| | from sentence_transformers import SentenceTransformer |
| | model = SentenceTransformer('intfloat/multilingual-e5-large') |
| | input_texts = [ |
| | 'query: how much protein should a female eat', |
| | 'query: 南瓜的家常做法', |
| | "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 i s 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or traini ng for a marathon. Check out the chart below to see how much protein you should be eating each day.", |
| | "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮 ,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右, 放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油 锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅" |
| | ] |
| | embeddings = model.encode(input_texts, normalize_embeddings=True) |
| | ``` |
| | |
| | Package requirements |
| | |
| | `pip install sentence_transformers~=2.2.2` |
| |
|
| | Contributors: [michaelfeil](https://huggingface.co/michaelfeil) |
| |
|
| | ## FAQ |
| |
|
| | **1. Do I need to add the prefix "query: " and "passage: " to input texts?** |
| |
|
| | Yes, this is how the model is trained, otherwise you will see a performance degradation. |
| |
|
| | Here are some rules of thumb: |
| | - Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval. |
| |
|
| | - Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval. |
| |
|
| | - Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering. |
| |
|
| | **2. Why are my reproduced results slightly different from reported in the model card?** |
| |
|
| | Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences. |
| |
|
| | **3. Why does the cosine similarity scores distribute around 0.7 to 1.0?** |
| |
|
| | This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss. |
| |
|
| | For text embedding tasks like text retrieval or semantic similarity, |
| | what matters is the relative order of the scores instead of the absolute values, |
| | so this should not be an issue. |
| |
|
| | ## Citation |
| |
|
| | If you find our paper or models helpful, please consider cite as follows: |
| |
|
| | ``` |
| | @article{wang2024multilingual, |
| | title={Multilingual E5 Text Embeddings: A Technical Report}, |
| | author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu}, |
| | journal={arXiv preprint arXiv:2402.05672}, |
| | year={2024} |
| | } |
| | ``` |
| |
|
| | ## Limitations |
| |
|
| | Long texts will be truncated to at most 512 tokens. |
| |
|
