Skip to contents

Performance notes

Source: vignettes/articles/performance-note.Rmd
performance-note.Rmd

seekr is not a fast text search tool. It is a structured text search and replacement workflow for R. These are different goals, and it helps to be clear about the trade-off before working with large file sets.

Why seekr is slower than CLI tools

When seekr searches a file, it does more than check whether a pattern matches.

For every match it finds, it stores the source path, the absolute character positions, the line and column positions, the matched text, the planned replacement if any, the line containing the match, surrounding context lines, the encoding, and a hash of the searched text that replace_files() uses later to verify the file has not changed.

The cost is real. seekr reads each file as a complete string, builds a structured R object for every match, and does all of this sequentially in a single R process. For most source code projects this is not a problem. For very large repositories, it can be.

The main cases where performance is likely to matter:

  • Large repositories with tens of thousands of files.
  • Large individual files, because each file must fit in memory as text.
  • Files with very large lines, because the complete matching line and, by default, surrounding context are stored for every match. Minified JavaScript, generated JSON, and one-line data dumps are common examples for which seekr may not be the best fit.
  • Workloads where you only need file names, not structured match objects. For this, a CLI tool such as ripgrep that will stop as soon as it finds a match is a better fit.

Practical strategies for larger workloads

Use Git-aware file discovery

Before using an external search tool, one simple option for Git repositories is to start with Git-aware file discovery:

files <- list_files(use_git = TRUE)
x <- seek("pattern", use_git = TRUE)

Pre-filter with ripgrep

Note: ripgrep is not managed by seekr: installing it, making it available on your system PATH, and dealing with platform-specific command-line details are left to the user. The goal here is only to show a possible pattern for advanced users with heavy workloads who are willing to use ripgrep for performance.

If the main bottleneck is the number of files or volume of data, you can use ripgrep to identify which files contain the pattern, then pass only those files to match_files().

# requires ripgrep to be installed and available on PATH
pattern <- "function"
path <- system.file("extdata", package = "seekr")

# list the files with at least a match
files <- system2(
  command = "rg",
  args = c("-l", shQuote(pattern), shQuote(path)),
  stdout = TRUE
)

# find the matches
x <- match_files(files, pattern, toupper)

This keeps seekr’s structured match objects while offloading the initial scan to a tool optimized for raw throughput.

Parallelize with mirai

If you want to parallelize the search itself, you can split the file vector into chunks and run match_files() on each chunk with mirai.

The idea:

  1. List and filter files as usual.
  2. Split the file vector into N roughly equal chunks.
  3. Start N mirai daemons.
  4. Call match_files() on each chunk in parallel.
  5. Combine the resulting seekr_match vectors.
library(seekr)
library(mirai)

files <-
  list_files() |>
  filter_files(extension = "R")

# Sequential version, useful as a reference
x <- match_files(files, "foo", "bar")
attr(x, "exclusions") <- NULL

N <- 8L

mirai::daemons(N)
mirai::everywhere(library(seekr))

split_files <- unname(split(files, cut(seq_along(files), N)))

matches_list <- mirai::mirai_map(
  split_files,
  \(files) match_files(files, "foo", "bar", .progress = FALSE)
)[]

mirai::daemons(0)

y <- Reduce(c, matches_list, init = new_seekr_match())

identical(x, y)
#> TRUE

As a rough example: searching a simple pattern across about 48,000 R files (roughly 500 MB of text) took about 80 seconds sequentially and about 20 seconds with 8 daemons on one machine. Files were likely cached by the OS at that point. This is a single data point, not a benchmark. Actual results depend on the storage device, number of cores, file sizes, pattern complexity, and how many matches need to be materialized.

For small projects, the overhead of starting workers and combining results is probably not worth the added code. The sequential workflow is simpler and usually fast enough.