RAGoon is a set of NLP utilities for multi-model embedding production, high-dimensional vector visualization, and aims to improve language model performance by providing contextually relevant information through search-based querying, web scraping and data augmentation techniques.
Here is an example of code to produce Embeddings for a given list of models:
```python
from ragoon import EmbeddingsDataLoader
from datasets import load_dataset
# Initialize the dataset loader with multiple models
loader = EmbeddingsDataLoader(
token="hf_token",
dataset=load_dataset("louisbrulenaudet/dac6-instruct", split="train"), # If dataset is already loaded.
# dataset_name="louisbrulenaudet/dac6-instruct", # If you want to load the dataset from the class.
model_configs=[
{"model": "bert-base-uncased", "query_prefix": "Query:"},
{"model": "distilbert-base-uncased", "query_prefix": "Query:"}
# Add more model configurations as needed
]
)
# Uncomment this line if passing dataset_name instead of dataset.
# loader.load_dataset()
# Process the splits with all models loaded
loader.process(
column="output",
preload_models=True
)
# To access the processed dataset
processed_dataset = loader.get_dataset()
```
In particularly, this tool also provides the functionality to load embeddings from a FAISS index, reduce their dimensionality using PCA and/or t-SNE, and visualize them in an interactive 3D graph.
1 comment
[ 4.3 ms ] story [ 9.6 ms ] threadLink to GitHub: https://github.com/louisbrulenaudet/ragoon Link to Hugging Face Space: https://huggingface.co/spaces/louisbrulenaudet/ragoon
Here is an example of code to produce Embeddings for a given list of models: ```python from ragoon import EmbeddingsDataLoader from datasets import load_dataset
# Initialize the dataset loader with multiple models loader = EmbeddingsDataLoader( token="hf_token", dataset=load_dataset("louisbrulenaudet/dac6-instruct", split="train"), # If dataset is already loaded. # dataset_name="louisbrulenaudet/dac6-instruct", # If you want to load the dataset from the class. model_configs=[ {"model": "bert-base-uncased", "query_prefix": "Query:"}, {"model": "distilbert-base-uncased", "query_prefix": "Query:"} # Add more model configurations as needed ] )
# Uncomment this line if passing dataset_name instead of dataset. # loader.load_dataset()
# Process the splits with all models loaded loader.process( column="output", preload_models=True )
# To access the processed dataset processed_dataset = loader.get_dataset() ```
In particularly, this tool also provides the functionality to load embeddings from a FAISS index, reduce their dimensionality using PCA and/or t-SNE, and visualize them in an interactive 3D graph.