CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene. ...
@i @ahmad @verita84Any DB that uses SQL can be made to work with this project.
Originally I wanted the DB itself to be any Nostr relay, but the relay would need special characteristics (like indexes on particular tags) that I can't guarantee unless Ditto itself acts as the relay.
@ahmadIt's stored in the SQLite database. Everything is a Nostr event (pretty much), so it only needs one table for the events, and another table for indexing tags.
Remote data is automatically pruned after 7 days. And since it's connected to the Nostr network, data can be refetched in realtime if you scroll back far enough. The experience will degrade when looking at remote content older than 7 days. Local content will load well and offer the best experience.
It won't have the same problems as Pleroma with remote data making the database too huge.
@ahmadI'm not building a system that's infinitely scalable. I'm building a system that's minimalist, low cost, and with account portability.
It should be able to handle traffic from Poast, for example. I'm not designing it to handle Truth Social or Gab levels of traffic. But it doesn't matter, because users can move freely and seamlessly between servers, so if one Ditto server is struggling or offline, people can just use a different one with their same account.
it's almost 3-5 steps just to get to reply to your post from nostr. i didn't see your post until i visited your rebased site. then i had to search for your profile, search for your post post, find a client that accepts your post and finally reply. these steps differ each time for any activitypub content
will ditto solve this discovery of post and reply (nostr <-> activitypub) problem?
Gab traffic is probably 50% spambots tbh and I don't imagine TS is any better. I'm glad that shit hasn't infested the fedi yet but these retards are already trying with their stupid chat gpt troll accounts
@sjwData is portable on Nostr, so I don't see a need to have a decentralized database. The Nostr network IS the decentralized database. If Ditto struggles to scale, the SQLite part could be replaced with a pool of dedicated Nostr relays.
@sjwLooks good. If I swap out SQLite it would be for a key-value store like LMDB or bbolt as I've just learned. Main reason I haven't is because it's a skill issue.
Clout Chaser 🗿
•Alex Gleason
•verita84 :Debian_logo: :firefox: :bing: :android:
•@opphunter88
Does not show how Client interacts with Ditto.
maybe good to add a box or something around Mastodon API and Mastodon Streaming API or combine them?
Clout Chaser 🗿
•Alex Gleason
•Alex Gleason
•@ahmadI hate Postgres. It's a goal of mine to make this work with an embedded database.
So far, I haven't optimized it. If performance is poor, there are lots of things I can try.
Failing all that, the query builder I use should allow us to use any SQL compatible backend including Postgres.
gvs
•Alex Gleason
•gvs
•Alex Gleason
•Alex Gleason
•Alex Gleason
•@ahmadThere are only 3 database tables. In the future there might only be 2. The whole codebase is built around NIP-01 filters, and the SQL part uses Kysely. It would be trivial to swap out SQLite for Postgres: https://gitlab.com/soapbox-pub/ditto/-/blob/develop/src/db/events.ts?ref_type=heads#L55-108
And it would be possible to use a different backing store such as LMDB.
src/db/events.ts · develop · Soapbox / Ditto · GitLab
GitLabverita84 :Debian_logo: :firefox: :bing: :android:
•@ahmad
Cassandra =)
Alex Gleason
•@verita84 @ahmadLooks cool, but I'm aiming for minimalism. Cassandra is maximalist.
If I were to use a non-embedded DB, ElasticSearch is weirdly the best option, except that it isn't FOSS.
:gcp: is happening
•@ahmad @verita84 apart from the forks that still are https://github.com/crate/crate
there's always https://sqlite.org/hctree/doc/hctree/doc/hctree/threadtest.wiki on the horizon though
GitHub - crate/crate: CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.
GitHubAlex Gleason likes this.
Alex Gleason
•@i @ahmad @verita84Any DB that uses SQL can be made to work with this project.
Originally I wanted the DB itself to be any Nostr relay, but the relay would need special characteristics (like indexes on particular tags) that I can't guarantee unless Ditto itself acts as the relay.
Alex Gleason
•@ahmadIt's stored in the SQLite database. Everything is a Nostr event (pretty much), so it only needs one table for the events, and another table for indexing tags.
Remote data is automatically pruned after 7 days. And since it's connected to the Nostr network, data can be refetched in realtime if you scroll back far enough. The experience will degrade when looking at remote content older than 7 days. Local content will load well and offer the best experience.
It won't have the same problems as Pleroma with remote data making the database too huge.
Juanro49
•One question. It's possible to search fediverse posts from Nostr with ditto/mostr?
#ditto #mostr
Alex Gleason
•@ba31f050a66d6aee73fb258405a44fcffa662e24784e6650944f8fc2d3089427None of the relays Mostr publishes to support NIP-50 search. But I bet things would show up in primal.net search since they index everything.
In Ditto I will experiment with SQLite FTS. We'll see how it goes.
Primal
primal.netAlex Gleason
•@ahmadI'm not building a system that's infinitely scalable. I'm building a system that's minimalist, low cost, and with account portability.
It should be able to handle traffic from Poast, for example. I'm not designing it to handle Truth Social or Gab levels of traffic. But it doesn't matter, because users can move freely and seamlessly between servers, so if one Ditto server is struggling or offline, people can just use a different one with their same account.
dscvr
•it's almost 3-5 steps just to get to reply to your post from nostr. i didn't see your post until i visited your rebased site. then i had to search for your profile, search for your post post, find a client that accepts your post and finally reply. these steps differ each time for any activitypub content
will ditto solve this discovery of post and reply (nostr <-> activitypub) problem?
Alex Gleason
•meso
•Alex Gleason likes this.
Alex Gleason
•NonPlayableClown
•>It can handle Poast level of traffic but not so much of Gab and TS.
Oof really hurting
feelings right there 🤣
Well I suppose it's a good thing
bans many people.
Senator_Armstrong
•Alex Gleason likes this.
Electric Pants :prince_dance:
•Alex Gleason
•Electric Pants :prince_dance:
•Then use CouchDB because it's the most comfy DB.
Also what about Revolver?
Alex Gleason
•@sjwCouchDB is cool, but I have a strong preference for embedded databases. I looked into PouchDB, but it's for something else.
Also look at how good SQL is at turning Nostr filters into queries, it's phenomenal: https://gitlab.com/soapbox-pub/ditto/-/blob/develop/src/db/events.ts?ref_type=heads#L66-128
Revolver doesn't exist, show me the code.
src/db/events.ts · develop · Soapbox / Ditto · GitLab
GitLabElectric Pants :prince_dance:
•https://pkg.go.dev/go.etcd.io/bbolt#section-readme
bbolt package - go.etcd.io/bbolt - Go Packages
pkg.go.devAlex Gleason
•