GitHunt
LE

lejeunerenard/hyperbee-parallel-readstream

Add support for reading hyperbee nodes in parallel for larger sets of nodes.

Hyperbee Parallel ReadStream

Note: Currently experimental, not production ready

This module wraps the Hyperbee
class to add the parallelReadStream() method. parallelReadStream() will
create worker threads to spread the task across multiple cores allowing for
faster reading of nodes on larger databases & ranges.

Usage

import HyperbeeParallel from '@lejeunerenard/hyperbee-parallel-readstream'

const db = new HyperbeeParallel(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'json'
})
const range = { gte: 'beep', lt: 'boop' }
for await (const node of db.parallelReadStream(range)) {
  console.log(node)
}

See example.js for an example on a large data set (1 million
keys).

TODO

  • Add support for createReadStream()'s second argument opt
  • Add support for limit being distributed on workers
  • Further test how keys are split based on hyperbee's b-tree

Languages

JavaScript100.0%

Contributors

Created July 28, 2023
Updated September 1, 2025
lejeunerenard/hyperbee-parallel-readstream | GitHunt