Package {bigrquerystorage}


Type: Package
Title: An Interface to Google's 'BigQuery Storage' API
Version: 1.3.0
Maintainer: Bruno Tremblay <openr@neoxone.com>
Description: Easily talk to Google's 'BigQuery Storage' API from R (https://docs.cloud.google.com/bigquery/docs/reference/storage/rpc).
License: Apache License (≥ 2)
Depends: R (≥ 4.1)
Imports: nanoarrow (≥ 0.6.0), rlang, assertthat, Rcpp, lifecycle, bit64, tibble
Suggests: blob, bigrquery, testthat, hms, wk (≥ 0.3.2), tzdb, base64enc, data.table
LinkingTo: Rcpp, nanoarrow
Encoding: UTF-8
URL: https://github.com/meztez/bigrquerystorage
BugReports: https://github.com/meztez/bigrquerystorage/issues
SystemRequirements: grpc/protobuf headers and compilers. For example libprotobuf-dev libgrpc++-dev protobuf-compiler-grpc (Debian) or protobuf-devel grpc-devel grpc-plugins (Fedora/RHEL)
Biarch: TRUE
Config/roxygen2/version: 8.1.1
NeedsCompilation: yes
Packaged: 2026-11-22 01:09:17 UTC; bruno
Author: Bruno Tremblay [aut, cre], Google LLC [cph, fnd]
Repository: CRAN
Date/Publication: 2026-11-22 08:10:02 UTC

bigrquerystorage: An Interface to Google's 'BigQuery Storage' API

Description

Easily talk to Google's 'BigQuery Storage' API from R (https://docs.cloud.google.com/bigquery/docs/reference/storage/rpc).

Author(s)

Maintainer: Bruno Tremblay openr@neoxone.com

Authors:

Other contributors:

See Also

Useful links:


Initialize bigrquerystorage client

Description

Initialize bigrquerystorage client

Close bigrquerystorage client

Usage

bqs_auth()

bqs_deauth()

Details

Will attempt to reuse bigrquery credentials.

About Credentials

If your application runs inside a Google Cloud environment that has a default service account, your application can retrieve the service account credentials to call Google Cloud APIs. Such environments include Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run, and Cloud Functions. We recommend using this strategy because it is more convenient and secure than manually passing credentials.

Additionally, we recommend you use Google Cloud Client Libraries for your application. Google Cloud Client Libraries use a library called Application Default Credentials (ADC) to automatically find your service account credentials. ADC looks for service account credentials in the following order:

  1. If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set, ADC uses the service account file that the variable points to.

  2. If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set, ADC uses the default service account that Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run, and Cloud Functions provide.

  3. If ADC can't use either of the above credentials, an error occurs.

Value

No return value, called for side effects.


Download table data

Description

This retrieves rows block in a stream using a grpc protocol. It is most suitable for results of larger queries (>100 MB, say).

Usage

bqs_table_download(
  x,
  parent = getOption("bigquerystorage.project", ""),
  snapshot_time = NA,
  selected_fields = character(),
  row_restriction = "",
  sample_percentage,
  n_max = Inf,
  quiet = NA,
  as_tibble = lifecycle::deprecated(),
  bigint = c("integer", "integer64", "numeric", "character"),
  max_results = lifecycle::deprecated(),
  type = c("tibble", "data.frame", "data.table", "arrow"),
  max_stream_count = 0L,
  compression = c("auto", "zstd", "lz4", "none")
)

Arguments

x

Table reference ⁠{project}.{dataset}.{table_name}⁠

parent

Used as parent for CreateReadSession. grpc method. Default is to use option bigquerystorage.project value.

snapshot_time

Table modifier ⁠snapshot time⁠ as POSIXct.

selected_fields

Table read option selected_fields. A character vector of field to select from table.

row_restriction

Table read option row_restriction. A character. SQL text filtering statement.

sample_percentage

Table read option sample_percentage. A numeric ⁠0 <= sample_percentage <= 100⁠. Not compatible with row_restriction.

n_max

Maximum number of results to retrieve. Use Inf or -1L retrieve all rows.

quiet

Should information be printed to console.

as_tibble

Should data be returned as tibble. Deprecated; use type instead.

bigint

The R type that BigQuery's 64-bit integer types should be mapped to. The default is "integer" which returns R's integer type but results in NA for values above/below +/- 2147483647. "integer64" returns a bit64::integer64, which allows the full range of 64 bit integers.

max_results

Deprecated

type

The output container type: "tibble" (default), "data.frame", "data.table", or "arrow" (returns a nanoarrow_array_stream).

max_stream_count

Maximum number of streams to request from BigQuery Storage (0L lets the server choose an optimal number of streams).

compression

Arrow record batch compression codec: "auto" (default, uses "zstd" when supported by nanoarrow), "zstd", "lz4", or "none".

Details

More details about table modifiers and table options are available from the API Reference documentation. (See TableModifiers and TableReadOptions)

Multi-stream reads are downloaded in parallel, each worker on its own gRPC connection, and the Arrow record batches of each stream are then decoded in parallel. Up to 16 streams are read concurrently by default; set options(bigrquerystorage.threads = n) to change the number of download and decoding threads.

Value

By default (type = "tibble"), a tibble::tibble. Alternatively a data.frame, data.table::data.table, or nanoarrow_array_stream according to type.


Upload table data

Description

This uploads rows to a BigQuery table using the BigQuery Storage Write API (gRPC protocol with Arrow RecordBatch serialization).

Usage

bqs_table_upload(
  x,
  values,
  fields = NULL,
  create_disposition = c("CREATE_IF_NEEDED", "CREATE_NEVER"),
  write_disposition = c("WRITE_EMPTY", "WRITE_APPEND", "WRITE_TRUNCATE"),
  stream_type = c("AUTO", "PENDING", "DEFAULT"),
  row_batch_size = 100000L,
  quiet = getOption("bigrquery.quiet", NA),
  max_stream_count = 0L,
  compression = c("auto", "zstd", "none"),
  ...
)

Arguments

x

Table reference (⁠{project}.{dataset}.{table_name}⁠, ⁠{project}:{dataset}.{table_name}⁠, or a bigrquery::bq_table).

values

Data to upload: a data.frame, tibble, data.table, nanoarrow_array, nanoarrow_array_stream, or any object coercible via nanoarrow::as_nanoarrow_array_stream().

fields

Optional schema specification (bq_fields or list of bq_fields). If NULL, inferred from values or retrieved from the destination table when it already exists.

create_disposition

Behavior if the destination table does not exist: "CREATE_IF_NEEDED" (default, creates the table) or "CREATE_NEVER" (errors if missing).

write_disposition

Behavior if the destination table already has data: "WRITE_EMPTY" (default, errors if non-empty), "WRITE_APPEND" (appends rows), or "WRITE_TRUNCATE" (replaces existing rows).

stream_type

Write stream mode: "AUTO" (default, uses high-speed parallel "DEFAULT" streams with automatic table cleanup on failure when creating a brand-new table, and atomic "PENDING" streams with BatchCommitWriteStreams when writing to an existing table), "PENDING" (always use atomic CreateWriteStream / FinalizeWriteStream / BatchCommitWriteStreams), or "DEFAULT" (always write to ⁠_default⁠).

row_batch_size

Maximum number of rows per AppendRows request when values is a data.frame (default 100000L). Automatically scaled down for wide tables to stay well below the 20 MB gRPC message limit.

quiet

Should progress information be suppressed? Default NA (shows progress in interactive sessions).

max_stream_count

Maximum number of parallel write streams to open (0L automatically scales concurrency based on payload size and options(bigrquerystorage.threads)).

compression

Arrow record batch buffer compression codec: "auto" (default, uses "zstd" when compiled with libzstd), "zstd", or "none".

...

Additional arguments passed to bigrquery::bq_table_create() when creating a table.

Details

More details about the BigQuery Storage Write API are available in the API Reference documentation.

Upstream bigrquery can delegate bq_table_upload(..., api = "arrow") directly to bqs_table_upload().

Value

Invisibly returns x.