LE
lejeunerenard/hyperbee-parallel-readstream
Add support for reading hyperbee nodes in parallel for larger sets of nodes.
Hyperbee Parallel ReadStream
Note: Currently experimental, not production ready
This module wraps the Hyperbee
class to add the parallelReadStream() method. parallelReadStream() will
create worker threads to spread the task across multiple cores allowing for
faster reading of nodes on larger databases & ranges.
Usage
import HyperbeeParallel from '@lejeunerenard/hyperbee-parallel-readstream'
const db = new HyperbeeParallel(core, {
keyEncoding: 'utf-8',
valueEncoding: 'json'
})
const range = { gte: 'beep', lt: 'boop' }
for await (const node of db.parallelReadStream(range)) {
console.log(node)
}See example.js for an example on a large data set (1 million
keys).
TODO
- Add support for
createReadStream()'s second argumentopt - Add support for limit being distributed on workers
- Further test how keys are split based on
hyperbee's b-tree