Let’s assume I have a software application with millions of files. The files receive frequent updates.
Is there a way to distribute those updates across all those devices?
I’m thinking something like Torrents/IPFS/Resilio. . . But they all seem like overkill.
I’ve been looking at my Search Engine, and thought it would be cool to allow people to download the database themselves, and run a local UI, for maximum privacy.
I would just be creating records within the main database. I could offer searches through the main site and allow people to explore it on their desktops.
i would put the small distributed files behind an API and set a cache on the client?
if you don’t want to do that due to wanting local offline then maybe separate the databases as plugins and download/subscribe based on if users enable them, similar to how CoMaps works.
Maybe syncthing?
rsync:// server
All receivers (I call client) must get all the files from the source (I call server) ?
Then, does the clients are known ? (Example : you can access, or you can left a tutorial and they can install a new software)
If yes for all files :
Git could work, but you have to host the server, cause no free online website do allow big content.If no for all files : Then a solution could be to share with https://zeronet.io/
This post is not done writing, i’ll complete later.
if there’s not so many devices just rsync or something similar from one central server
if there’s a large number of devices i’d try torrent
Let’s assume I have a software application with millions of files.
rsync is generally good at things like this.
I would just be creating records within the main database.
So is it one file? Or millions?
maybe caldav/nextcloud
webdav*
Bless your heart op.
Thank you 🙏️
Please take this as support: You need to look a little bit more into theories. You are reinventing things that don’t need reinventing (categorizing data), but are not thinking about the real pain points of building search that return relevant results (full text search performance, scalability, semantics, extracting user intention from queries).
Here’s an article that I think is very good on the topic
That’s because I don’t want that.
I remember, as a kid, wandering the local library. Looking at all the books. Seeing the call numbers and author names. Exploring the shelves, as they transitioned from topic to topic to topic.
My idea is to treat the Internet as a giant library, with domains being books on a shelf, and allowing my users to start at a random location, and then peruse the different listings as they transition from slight variations of the main topic.
With all due respect, you don’t have the vision, man. To start looking at cultural anthropology, then have a link to a biological anthropology blog 3 lines down? That’s the goal. It’s not about a narrowed search. It’s about cataloging something, so people can go looking for it. To enjoy the hunt.
There are a couple of features in the BitTorrent specs for updating torrents, like embedding a signed RSS URL in the torrent file itself, or using mutable DHT entries, but I’ve never tried them out and I couldn’t tell you which clients support those features, and you’d have to manage signing keys since those features are completely decentralized and use public and private keys to ensure authenticity. Definitely worth looking into, but I apologize I don’t have anything concrete to link for you, I’m just aware of the features in the spec.
I think most clients also support pulling torrents from plain RSS feeds containing links to torrent files. Some sites like showrss do that.
