Sure, there are 3 parts. The Gatherer, the API and the Client.
The Gatherer is a Node server, that continuously fetches board catalogs from the public-Sup Forums-api.
It keeps a history of recent update-cycles, from which it takes the last hour to calculate most of the live board stats.
At set times it also calculates a new hour and day history-entry from those cycles, which are used for the timeline, avg.posts/day and activity %.
Everything is kept in memory, but for persistency all updates are written to a levelDB instance.
(I know it's key-value, but you can query and stream sorted key-ranges, which fit's really well, since all keys are made up of identifiers and unix timestamps. A key looks like [board,"hour",entryTime]. The only time the DB is actually read form is on server-startup.)
The API connects to the Gatherer via socket and gets streamed all new board-, thread- and history-updates.
In turn it also accepts socket connections from Clients, to which it forwards any board-stats updates right away.
It also has API endpoints for board-, thread- and history-stats.
The client is using Vue.js to render the view.
Board-stats updates are streamed in, but thread- and history-data is requested via ajax, either when a client clicks on a board in the list, or when the currently selected board receives an update via the socket, which indicates to the client, that an update to the threads should also be available. (no point streaming in threads and history, the client might never view)
The client knows how long threads and timeline data is valid, so switching between boards should make no unnecessary requests to the API.
I made the first version in order to learn React, but switched to Vue, which I am more familiar with, after working on it for a while
(was React)
The Gatherer and API were combined in one application previously, but separating the client-facing part seemed like a good idea.
