Package {ragR}

ragR: Retrieval-Augmented Generation + RAGAS Evaluation in R

Description

The ragR package provides a Retrieval-Augmented Generation (RAG) system implemented in R. It includes:

• A data ingestion pipeline for PDF, text, and Word documents

• An R-native vector store for chunk embeddings

• A configurable RAG pipeline that uses OpenAI embeddings and chat models

• Persistent QA logging for reproducible evaluation

• LLM-scored RAGAS-style metrics, including context precision, context recall, answer relevance, and faithfulness

• A Plumber-based HTTP API suitable for chatbot and evaluation frontends

Architecture

The package is organized into several modules:

• Ingestion (ingestion_files.R) Extracts text from PDF/TXT/DOCX files, chunks it, and prepares data for embedding and storage. Ingestion is handled through package functions and development scripts, not through the minimal HTTP API.

• Embeddings & Chat (embeddings_openai.R) Wraps the OpenAI REST API for embeddings, such as "text-embedding-3-small", and chat models, such as "gpt-4o-mini". Reads the API key from OPENAI_API_KEY.

• Vector Store (vectorstore_interface.R) Implements an R-native vector store using tibbles and numeric matrices, with functions to add, query, and clear stored embeddings. Also includes vectorstore_clear_all() for wiping the entire vector store in one operation.

• RAG Pipeline (rag_pipeline.R) Given a user question, the pipeline:

  1. Computes an embedding for the question

  2. Retrieves top-k similar chunks from the vector store

  3. Builds a grounded context-aware prompt

  4. Calls the OpenAI chat model to generate an answer

• QA Logging & Persistence (qa_logging.R, qa_persistence.R) Stores each interaction in db/qa_log.rds, including the question, model answer, retrieved chunks, final grounded prompt, model names, and timestamps. Provides helpers to load, save, and clear logs.

• RAGAS Metrics & Summary (ragas_metrics.R, ragas_summary.R) Computes LLM-scored RAGAS-style metrics, including context precision, context recall, answer relevance, faithfulness, and an overall score. Provides summary helpers for aggregating metric results across QA pairs.

• Reports (ragas_report.R) Generates a performance report by:

  • Computing LLM-scored metrics from the QA log

  • Saving db/qa_metrics.rds

  • Writing reports/ragas/ragas_summary.csv

  • Writing a bar chart of mean metrics to reports/ragas/ragas_means.png

• API Handlers (api_handlers.R) Functions that connect the core logic to HTTP endpoints. These are called by the Plumber router defined in ⁠inst/api/⁠.

HTTP API

When scripts/dev/run_api.R is executed, a Plumber server exposes:

• POST /chat Runs the RAG pipeline for a user question and logs the result into db/qa_log.rds.

• GET /ragas Computes and returns LLM-scored RAGAS-style metrics and a summary for the logged QA pairs.

• POST /ragas/report Computes LLM-scored metrics from the QA log, saves the metrics and report artifacts to disk, and returns a small summary object.

• POST /ragas/clear Clears the QA log, QA metrics, and report files.

• POST /clear Clears one vector-store collection.

• POST /clear_all Clears the entire vector store across all collections. Uses api_clear_all_handler(), which calls vectorstore_clear_all().

Usage

Programmatic usage inside R:

library(ragR)

# Run a RAG query
res <- query_rag(
  question        = "What is this document about?",
  collection      = "default",
  top_k           = 5,
  embedding_model = "text-embedding-3-small",
  chat_model      = "gpt-4o-mini",
  system_prompt   = "You are a helpful assistant."
)

cat(res$answer)

As a backend service:

# From the project root
source("scripts/dev/run_api.R")
# Visit http://127.0.0.1:8000/__docs__/ for interactive docs

Author(s)

Maintainer: Muhammad Aimal Rehman rehman.aimal@gmail.com

Authors:

• Zhili Lu

• Chi-Kuang Yeh

See Also

Useful links:

• https://github.com/aimalrehman92/ragR

• Report bugs at https://github.com/aimalrehman92/ragR/issues

API handler: chat with the RAG pipeline

Description

This handler receives a parsed JSON request body, runs the RAG pipeline, appends the interaction to the QA log, and returns the model answer.

Usage

api_chat_handler(body)

Arguments

Value

A list suitable for JSON serialization.

API handler: clear the entire vector store

Description

This handler wipes the R-native vector store by calling vectorstore_clear_all(), removing all collections and stored embeddings.

Usage

api_clear_all_handler()

Value

A list with elements status and message, suitable for JSON serialization.

API handler: clear one vector-store collection

Description

This handler deletes one collection from the R-native vector store. If no collection is supplied, "default" is used.

Usage

api_clear_handler(body = NULL)

Arguments

Value

A list with status, collection, and message.

API handler: clear RAGAS/QA evaluation files

Description

Clears the saved QA log, QA metrics file, and generated RAGAS report files.

Usage

api_ragas_clear_handler(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas"
)

Arguments

Value

A list with status and message.

API handler: compute RAGAS metrics and summary from saved QA log

Description

API handler: compute RAGAS metrics and summary from saved QA log

Usage

api_ragas_handler(path = "db/qa_log.rds", judge_model = "gpt-4o-mini")

Arguments

Value

A list with status, n_qa, metrics, summary.

API handler: generate RAGAS performance report

Description

API handler: generate RAGAS performance report

Usage

api_ragas_report_handler(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas",
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list with status, n_qa, qa_metrics_path, summary_csv_path, and plot_path.

Build a RAG prompt from question and retrieved chunks

Description

Build a RAG prompt from question and retrieved chunks

Usage

build_rag_prompt(question, retrieved)

Arguments

Value

Character scalar: the constructed grounded RAG prompt.

Chunk text into overlapping character segments

Description

Chunk text into overlapping character segments

Usage

chunk_text_character(text, chunk_size = 500L, chunk_overlap = 50L)

Arguments

Value

Character vector of chunks.

Chunk text into sentences (strict)

Description

Splits text into individual sentences. Each returned element is intended to be exactly one sentence (best-effort based on punctuation).

Usage

chunk_text_sentence(text)

Arguments

Value

Character vector; one sentence per element.

Clean raw extracted text (minimal)

Description

• Forces UTF-8

• Replaces carriage returns and newlines with spaces

• Removes control characters

• Collapses multiple whitespace

• Trims

Usage

clean_raw_text(text)

Arguments

Value

Cleaned character scalar

Clean text for embeddings (minimal, deterministic)

Description

Clean text for embeddings (minimal, deterministic)

Usage

clean_text_for_embeddings(text)

Arguments

Value

Cleaned character scalar

Clear the QA log on disk

Description

This helper overwrites the QA log file with an empty QA log, as created by qa_log_empty(). It is intended for starting a fresh evaluation session.

Usage

clear_qa_log(path = "db/qa_log.rds")

Arguments

Value

Invisibly, the path to the saved file.

Clear QA metrics on disk

Description

This helper overwrites the QA metrics file with an empty metrics tibble, as created by qa_metrics_empty().

Usage

clear_qa_metrics(path = "db/qa_metrics.rds")

Arguments

Value

Invisibly, the path to the saved file.

Compute RAGAS metrics

Description

Convenience wrapper for computing LLM-scored RAGAS-style metrics.

Usage

compute_ragas_metrics(
  qa_log,
  judge_model = "gpt-4o-mini",
  seed = NULL,
  embedding_model = "text-embedding-3-small",
  answer_relevance_strictness = 3L
)

Arguments

Value

A metrics tibble.

Compute RAGAS-style metrics (LLM scored)

Description

This implementation mirrors the structured Python RAGAS workflow rather than asking the judge model for one direct scalar score per metric.

Usage

compute_ragas_metrics_llm(
  qa_log,
  judge_model = "gpt-4o-mini",
  seed = NULL,
  embedding_model = "text-embedding-3-small",
  answer_relevance_strictness = 3L
)

Arguments

Details

Notes:

• context_precision uses answer_reference if available; otherwise it uses answer_model, matching the Python with-reference / without-reference split.

• answer_relevance requires an embedding helper to be available in the package environment.

Value

A tibble with one row per qa_id and metric columns.

Embed texts robustly using OpenAI embeddings helper

Description

Calls get_openai_embeddings() on the provided texts. On transient failures, retries with exponential backoff. On HTTP 400 (often request too large), the batch is split recursively.

Usage

embed_texts_batched(
  texts,
  model,
  retry = TRUE,
  max_retries = 5L,
  verbose = TRUE
)

Arguments

Value

A numeric matrix of embeddings aligned with texts

Extract text from a document

Description

Detects file type and extracts raw text from PDF, DOCX, or TXT.

Usage

extract_text(path)

Arguments

Value

Character scalar containing extracted text.

Generate a chat completion from OpenAI

Description

Used by the RAG pipeline and RAGAS-style metric prompts to generate text from a constructed prompt.

Usage

generate_openai_chat(
  prompt,
  model = "gpt-4o-mini",
  system_message = NULL,
  temperature = 0,
  max_output_tokens = 512L,
  seed = NULL,
  api_key = NULL
)

Arguments

Value

Character scalar containing the model's answer.

Generate a RAGAS performance report

Description

Loads a QA log from disk, computes LLM-scored RAGAS-style metrics, and writes:

• metrics RDS (qa_metrics_path)

• summary CSV (ragas_summary.csv)

• means bar plot PNG (ragas_means.png)

Usage

generate_ragas_report(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas",
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list with: status, n_qa, qa_metrics_path, summary_csv_path, plot_path.

Generate RAGAS report using LLM scoring

Description

This is an explicit alias for generate_ragas_report().

Usage

generate_ragas_report_llm(
  qa_log_path = "db/qa_log.rds",
  qa_metrics_path = "db/qa_metrics.rds",
  output_dir = "reports/ragas",
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list with: status, n_qa, qa_metrics_path, summary_csv_path, plot_path.

Get an environment variable or fail with a clear error

Description

Get an environment variable or fail with a clear error

Usage

get_env_or_stop(name)

Arguments

Value

The value of the environment variable as a character scalar.

Get embeddings from OpenAI

Description

Calls the OpenAI embeddings endpoint to obtain vector representations for one or more input texts. Used by both ingestion and query-time retrieval.

Usage

get_openai_embeddings(texts, model = "text-embedding-3-small", api_key = NULL)

Arguments

Value

A numeric matrix with one row per input text.

Ingest documents into the vector store

Description

Reads PDF, DOCX, or TXT files, extracts text, cleans it lightly, chunks it, embeds chunks, and stores them in a vector store collection.

Usage

ingest_documents(
  paths,
  collection = "default",
  chunk_size = 500L,
  chunk_overlap = 50L,
  chunking_strategy = c("character", "sentence"),
  embedding_model = "text-embedding-3-small",
  embedding_batch_size = 128L,
  embedding_max_chars = 8000L,
  retry = TRUE,
  max_retries = 5L,
  resume = TRUE,
  checkpoint_path = NULL,
  use_openai = TRUE,
  verbose = TRUE
)

Arguments

Details

Chunking strategies:

• "character": overlapping fixed-size character windows.

• "sentence": strict sentence splitting via chunk_text_sentence().

Cleaning strategy (minimal, deterministic):

• Replace carriage returns and newlines with spaces

• Remove control characters

• Collapse multiple whitespace to a single space

• Trim leading/trailing whitespace

Embedding strategy (robust by default):

• Embeddings are computed in batches (default embedding_batch_size = 128)

• Batches are retried with exponential backoff on transient failures (e.g., 429/5xx)

• If a 400 occurs due to request size, the batch is automatically split

• Each chunk is truncated to a safe maximum length before embedding

• Optional resumable ingestion via a local checkpoint file

Value

A tibble with a per-file ingestion summary (collection, path, n_chunks).

Load a QA log from disk

Description

This helper loads a QA log tibble from an RDS file. If the file does not exist, it returns an empty QA log created by qa_log_empty().

Usage

load_qa_log(path = "db/qa_log.rds")

Arguments

Value

A tibble containing the QA log.

Load QA metrics from disk

Description

This helper loads QA metrics from an RDS file. If the file does not exist, it returns an empty metrics tibble created by qa_metrics_empty().

Usage

load_qa_metrics(path = "db/qa_metrics.rds")

Arguments

Value

A tibble containing the QA metrics.

Load RAG configuration

Description

Placeholder for a configuration loader. In the future, this can read a YAML file and merge it with environment variables and defaults.

Usage

load_rag_config(path = NULL)

Arguments

Value

A list of configuration values. Currently returns an empty list.

Append one RAG interaction to the QA log

Description

Adds one row to an in-memory QA log tibble.

Usage

log_rag_interaction(
  qa_log,
  question,
  rag_result,
  collection,
  chat_model = "gpt-4o-mini",
  embedding_model = "text-embedding-3-small",
  qa_id = NULL,
  timestamp = Sys.time()
)

Arguments

Value

Updated QA log tibble.

Plot mean RAGAS metrics to a PNG

Description

Creates a simple bar chart of mean metric values.

Usage

plot_ragas_means(summary_df, output_path, width = 1200, height = 700)

Arguments

Value

Invisibly, output_path.

Create an empty QA log tibble

Description

Standard schema for storing Q/A interactions produced by the RAG pipeline.

Usage

qa_log_empty()

Value

A tibble with zero rows and the standard QA log columns.

Create an empty QA metrics tibble

Description

This helper creates an empty tibble with the columns used to store RAGAS-style evaluation metrics for each QA interaction.

Usage

qa_metrics_empty()

Details

All metric values are expected to be in [0,1].

Value

A tibble with zero rows and the standard QA metrics columns.

Query the RAG pipeline

Description

Takes a user question, embeds it, retrieves relevant chunks from a vector store, constructs a grounded RAG prompt, and calls an LLM to generate an answer.

Usage

query_rag(
  question,
  collection = "default",
  top_k = 4L,
  embedding_model = "text-embedding-3-small",
  chat_model = "gpt-4o-mini",
  temperature = 0,
  max_output_tokens = NULL,
  score_threshold = 0,
  system_prompt = ""
)

Arguments

Value

A list with:

Save a QA log to disk

Description

This helper saves a QA log tibble, as produced by log_rag_interaction(), to an RDS file. By default, it writes to db/qa_log.rds inside the project directory.

Usage

save_qa_log(qa_log, path = "db/qa_log.rds")

Arguments

Value

Invisibly, the path to the saved file.

Save QA metrics to disk

Description

This helper saves a QA metrics tibble, as produced by compute_ragas_metrics(), to an RDS file. By default, it writes to db/qa_metrics.rds inside the project directory.

Usage

save_qa_metrics(qa_metrics, path = "db/qa_metrics.rds")

Arguments

Value

Invisibly, the path to the saved file.

Summarize RAGAS metrics across QA pairs

Description

Computes mean, standard deviation, minimum, and maximum for each metric column in a QA-metrics tibble.

Usage

summarize_ragas(qa_metrics)

Arguments

Value

A tibble with columns: metric, mean, sd, min, max.

Delete ALL collections from the vector store

Description

Completely wipes the vector store by replacing it with an empty tibble.

Usage

vectorstore_clear_all()

Value

Invisibly, TRUE.

Delete a collection from the R-native vector store

Description

Delete a collection from the R-native vector store

Usage

vectorstore_delete_collection(collection)

Arguments

Value

Invisibly, TRUE on success.

Load the R-native vector store

Description

This function loads the current contents of the R-native vector store from db/vectorstore.rds. If the file does not exist yet, it returns an empty tibble with the expected columns: collection, id, text, embedding, and metadata.

Usage

vectorstore_load()

Value

A tibble with one row per stored chunk, or zero rows if the store is empty.

Query the R-native vector store for nearest neighbors

Description

Query the R-native vector store for nearest neighbors

Usage

vectorstore_query(collection, query_embedding, top_k = 4L)

Arguments

Value

A tibble with columns collection, id, text, score, and metadata.

Upsert embeddings into the R-native vector store

Description

Upsert embeddings into the R-native vector store

Usage

vectorstore_upsert(collection, ids, embeddings, documents, metadatas = NULL)

Arguments

Value

Invisibly, TRUE on success.