This is an automated archive made by the Lemmit Bot.
The original was posted on /r/opensource by /u/massus on 2024-01-10 16:28:15.
Hi there! I’m one of the maintainers of Quickwit, a distributed search engine. We just reached a significant milestone, so I thought it was time to tell a bit about what we accomplished in the open-source world.
We started 3 years ago with the dream of replacing great but expensive software such as Datadog with a great OSS one…
And you know what, it’s way harder than we expected :)
We decided to build a distributed search engine on top of tantivy, a (now popular) search library created by my cofounder u/fulmicoton in his spare time. And each time there was a missing OSS library, we implemented it ourselves and open-sourced it: ChitChat (Cluster membership protocol), mrecordlog (WAL), Whichlang (fast language detection), witty actors (actor framework), bitpacking (About SIMD algorithms for integer compression)…
And 3 years later, I’m here to tell you that we reached a significant milestone with the release of version 0.7: we now have a good Elasticsearch-compatible API and good integration within the OSS ecosystem (Grafana, Jaeger, OpenTelemetry). We also witnessed users using the nightly version at a large scale (hundreds of nodes, hundreds of terabytes of data daily ingested) with minimal hardware and enjoyed lovely cost reductions.
Regarding use cases, Quickwit is a great fit for logs, traces, security data, or any append-only datasets (and soon metrics). We store all the index data on an object storage. Because storage is separated from compute, it is much easier to remove or add nodes.
But you may wonder: “bummer, now I need to have my object storage”. Fortunately, you have at least two good options in the OSS world: you can use a classic local disk if you don’t care about losing your data (if original data is saved elsewhere, for example), OR you can use Garage, an OSS distributed object storage tailored for self-hosting. We had excellent feedback from users deploying it. OSS for the win!
We are still a small company, and any feedback is always appreciated!
Github repo -
Fun fact - some smart engineers working on petabytes of data sent us some nice pictures of their garage plenty of old servers and disks.