Ask HN: Is it possible to have forum type community running without a database?

One of the main points of static sites is that they don't update terribly often. Forums, on the other hand, update constantly. You are wanting something with good caching (server side and client side), and thats about it. Otherwise, you are going to be regenerating the pages constantly on every post/update.

You can run a community of sorts, but the feature list will be sparse. The only viable system sans-database that comes to mind is plain-text.

I'm a member of a forum that had an old system in the beginning that used plain text "database" for a very long time. Performance was fairly reasonable at 5000 visitors per day, if the daily hit counter was to be trusted. They since moved it to SQLite, which is very reliable in its own right, and now it's Postgres. It was only now that we got searching as a "feature".

The system was written originally in PHP 4.x, but they did move it to 5.x. I'm not sure what version they use now. I do know they were running ancient hardware and was up solely due to the charity of the admin.

I think there was an index file that stores new topic summaries, as there was no field for titles, when they were created.

They used microtime() as the ID, but the topics were stored by splitting it to 3 digit directories.

Seconds as 1234567890 and decimal 0.12345618 were combined to create /123/456/789/012/345618.html

There's no registration system or other way to identify the user so I believe the posts were separated by some kind of entry separator in a single file.

I think the pagination system created an array of sequential IDs and checked to see if files existed in those directories. There were no numbered links for pagination, only next/previous and if you reached the end, the next page would be blank.

There's no reason you should ever get laughed at for being curious.

File systems are databases of some sort: there's a way of storing and retrieving data. Using the FS won't necessarily be simpler or less resource-intensive than using sqlite or something similar, because you _will_ store and retrieve data, and the helper layers you build on top of the FS will probably be less optimized than existing lightweight databases.

Have a look at 4chan - once the thread is done, it's gone, no need for db for that and the community is still there.

you can write it to disk

you can also hold all the data in memory but when the memory is full you have to get rid of the old threads, but if your server app restarts you lose all threads

you could also use a message queue, something like firebase

Several people have made use of a lightweight webserver like Lighttpd in combination with PHP/SQLite in order to set up boards or wordpress installations on Raspberry Pis. Would that be something for you?

Sure. HN itself has no database, only the filesystem (or rather the filesystem is the DB).

If the only requirement is to run it on a raspberry pi, and you find that a RDBMS will be too resource intensive, you can always use one of the cloud data storage services out there. Amazon has an RDS offering as part of AWS, MySQL, Postgres, etc. Then your app only needs to talk to the database over the wire with zero local resource consumption by the database.

Hi there, you ask interesting questions. I have two things for you that aren't exactly what you're looking for, but they are definitely relevant.

First of all, a while ago I made a serverless chat for in your browser. It's just a proof of concept, it does work however. There will be nobody there, but you can test it by opening two browser tabs or using a second device. It works in the same way that Skype works, peer to peer. It's cool, because it is in the browser. http://codepen.io/Azeirah/pen/BHnbz

Second of all, you know bitcoins, right? It has this blockchain thing. People are starting to realize the potential of the blockchain. I've seen multiple initiatives and examples of applications using the blockhain to store the data without a static database. Note however, the data is still stored somewhere, on the users' pc's!

This is not a database-less application, it is decentralized. It's still very cool however, and may have a huge influence on the internet in the future if it stabilizes.

https://eris.projectdouglas.org/ (and their github: https://github.com/project-douglas/eris)

This is a fun exercise :)All right. Let us first analyze what the requirements should be. Obviously I am assuming a lot of things and thinking about this in a way I would go about building a (minimal)forum on raspberry pi:-

1. There should be a couple of predefined categories.

2. There should be a way to authenticate/authorize users.

3. There should be a way to search the previous posts.

4. There should be a facility to moderate.

Static forums

Very neat idea. All we need is a simple webserver, I am thinking, nginx that transforms and renders our posts to a user. Our first requirement that of predefined categories can be easily met by creating sub directories inside the directory that will be served. And we can avoid duplicating the pages between the sub directory (ie. a tag) and the main directory by using symbolic links[1] and configuring nginx to serve the symlinks [2]. Moderation of posts can be done by migrating a post in and our of a `moderate` folder that is not served by the application. But that obviously limits the moderation only to the webmaster. The trouble arises in two cases

a) When we need a facility to manage sessions. There is no way we can do that in static pages.

b) It will be quite difficult to create a search index for our application.

Now nginx is a good choice as it is pretty light weight and we will keep it. But what if we could have another application that can easily talk with nginx, manage user authentication for us, allow us to create search indexes, takes care of our data and still be light weight. Other have already suggested sqllite.How about couchdb?[3]

First couchdb has built in authentication and authorzation support. It will take you no more than 3 http requests to implement your own register-login-logout-scenario[4]. Second it is pretty easy creating simple search indexes with couchdb[5]. You can serve your forum directly from couchdb. It does not statisfy your orignal requirement of no database but it certainly gives you a way to work without any application layer. But perhaps most importantly it is very low on memory consumption. Not as low as sql lite but still quite low.

[1](http://stackoverflow.com/questions/1951742/how-to-symlink-a-...)

[2](http://nginx.org/en/docs/http/ngx_http_core_module.html#disa...)

[3](http://couchdb.apache.org/)

[4](http://www.staticshin.com/programming/easy-user-accounts-man...)

[5](http://wiki.apache.org/couchdb/View_collation#String_Ranges)

You're probably overthinking this. While the Pi is limited in processing power and I/O, if it's really a small community it will run a forum just fine.

Here's what to look out for:

1) Make sure you run NginX, it's fast and memory-efficient.

2) Install PHP-FPM and have NginX connect to it using a socket file. Keep the number of FPM workers small.

2a) Zend Opcache comes standard with PHP now, but there are scenarios where you might want to look into using APC instead, especially if you want to hand-tweak some caching into the forum app. But see if it's necessary first.

3) I had good experiences with MariaDB, a MySQL fork, which comes with the Aria table type. If possible, change the forum tables to that type, it's very fast. The worst table type on a Pi is InnoDB, it's pretty is much unusable.

4) Beware of flash medium write wear, buy a larger-than-needed SD card or place files that update frequently on an external USB hard drive.

I haven't seen anyone mentionning it yet: what about a mailing list ? You could even offload its presentation layer to a third party nntp reader.

A file database like sqlite or leveldb might be a good compromise between the features of a database and the ease of a flat filesystem

Yes, it's totally possible to write a blog, forum, or pretty much any "Web 2.0" application without using a database. You just need to be very, very careful with how you design your filesystem layout and file format.

As for the file format, it would be best to stick with standard formats like JSON, XML, or YML. Or the standard serialization method for your favorite language, such as pickle() for Python and serialize() for PHP. Try not to invent a brand-new format, it's going to be error-prone and generally slower than using a standard format.

A more difficult task is to maintain an index of some sort, separate from all the individual files, so that you won't have to read and parse every single file in order to generate the forum listing. Whenever a reply is posted, you'll need to bump a thread to the top of the listing. Think about how you can implement this without modifying or renaming several files at a time.

And then you'll need to ask yourself how you're going to prevent inconsistencies in your data over the next few years. What if you decide to add a new field to the JSON schema and all the old files don't have that field? What if you delete a thread but an error occurs halfway through and some of the individual posts still remain? You'll need to write logic to handle such edge cases as well.

SQLite solves a lot of these problems while offering the same sort of performance, if not better, on resource-constrained environments. As far as SQLite is concerned, a Raspberry Pi is a very powerful machine. There's no reason why you sholdn't be able to enjoy all the benefits of flat files together with all the benefits of a relational database.

  As far as SQLite is concerned, a Raspberry Pi is a very 
  powerful machine. There's no reason why you sholdn't be
  able to enjoy all the benefits of flat files together
  with all the benefits of a relational database.

I hate it when people change the question so they can give the answer they want to give, but I'm going to do it anyway, as I think its a relevant data point; I think you're dismissing the ability to run a database too quickly.

I have done almost exactly what you're talking about with a raspberry pi, using Ruby on Rails (3.0.x era) and sqlite. the Rasppi (model B) had enough horsepower to run in development mode on webrick and handle ~5 rps with page rendering and database calls, and that was in develop! when running in production, rails does less reloading of resources, so should be even more efficient.

I sure do think it is possible. I am not sure how soon you will actually run into problems, but I have been using Dokuwiki, which uses text files rather than a database for all storage purposes. Although it is not in itself a good example for a community-centric software, it shows what can be done, the contents are versioned, there are access control lists and so on.

If you can build a wiki based on files, why would you not be able to do so with some kind of forum community? Maybe Dokuwiki is already a good solution for you.

I feel that the others readers reactions show that you could elaborate your motivation some more. A database can be a quick and useful solution and solves many problems, such as distributed access, central management and administration, security and backup.

http://en.wikipedia.org/wiki/DokuWiki

I'm not sure if this counts as a database, but you can try out things like Firebase or Parse, which allow you to fetch data from a cloud server, and render it on your server. So your application will be just a dumb static client containing templates and application logic in HTML/CSS/JS, while the data would be stored on the firebase servers, directly fetched from there by your visitors on page-load, making your work pretty easy.

I don't see why the Pi would be unable to run a database. In any case, sqlite would work.

I would look into running a Citadel BBS[0] if it's something on the order of a personal project. Essentially, it's a BBS with Telnet, API, web, and email interfaces.

[0] http://www.citadel.org/

I've used SimpleForum for that purpose a long time ago:

http://freecode.com/projects/simpleforum

Unfortunately it doesn't seem to be in active development any more.

Checkout NoNonsense forum (camendesign [.] com/code/nononsense_forum). It works without a database and stores everything as RSS.

If the point of getting a forum site running on a Raspberry Pi is just for novelty & challenge, ignore this. But I think the most practical solution here is to forget the Raspberry Pi altogether or to set up the majority of processing and storage on a separate machine and have the Raspberry Pi server talk to it.

Yes.

People (including me) are thinking about and working on personally owned and hosted data. Your forum posts live in _your_ cloud, and in a way that the features as brickcap describes are achievable.

See https://news.ycombinator.com/item?id=8049890

Depends on your goal.

If you want to learn a lot about development, write a lot of code, you can use flat files and treat them as a small database. Implement simple SQL analogues of WHERE, etc. It could definitely be fun to implement! A lot of educational value. You would learn more about designing APIs consumed by other applications, etc.

But if you just want something that would run on r-pi, you would be better off by using a standard database. Some of them are around for over a decade, written in C and thousands of men hours were spent on optimizing them. As long as you are not doing something a bit crazy (e.g. 3 fields in WHERE statement, non of them are indices), you will be just fine.

  So I wondered if it is possible to run a forum type 
  community - something along the lines of [elgg][1] or
  [Friendica][2] - but without the MySQL or other type of
  database backend?

You're attempting to optimize the wrong part of the architecture, I think.

A simple read-only query to SQLite or a NoSQL database is fast - generally just a few milliseconds and often less than one millisecond. The odds of you doing it much faster yourself are low.

Rendering pages is the slow part. Because each forum page you render is going to involve multiple queries to your storage engine (whether you go with SQL or roll your own) and a ton of string manipulation/concatenation.

So what I'd do is....

- Use SQLite for my storage engine. The 256 or 512MB of RAM on a RaspberryPi is plenty for SQLite. - Cache rendered pages (and/or page fragments) to disk, rather than writing my own storage engine.

I'd use a lazy caching/prerendering strategy. Suppose a discussion thread has 50 pages. One of your mods deletes a post on page 1. Now all 50 pages need to be re-rendered. You have two choices. You can either re-render all 50 pages right away, or you can simply delete all 50 pages right away and re-render & re-cache them as needed, when a user actually requests one of them. I'd do the latter.

I've used this strategy myself. It was a very common paradigm back in the 90s and early 2000s when web servers commonly had hardware specs (700mhz, 256/512MB RAM) that was quite similar to what a RaspberryPi has today. The hardest part of this strategy is getting cache invalidation correct. Every time your code writes to the database, it has to also be aware of which cached pages/fragments it needs to blow away.

(You know the old joke: "There are only two hard things in Computer Science: cache invalidation and naming things.")