Deep Dive: The Microservices Migration
A BBS emulator that outgrew localStorage: gRPC services, NATS messaging, a gateway, and the question of whether any of it was necessary.
On this page
Two people posted to the same bulletin board and neither could see the other’s messages. I stared at that for a while before the obvious explanation arrived: each browser had its own localStorage, its own little universe of data. Which is how localStorage works. It is not how a BBS works.
That was the point where I stopped pretending the emulator was a client-side toy. Shared state had to live somewhere authoritative, and that somewhere wasn’t the browser.
Why I Went with Server Ownership
I spent a few days on two approaches and didn’t love either.
The first was client-side replication: keep everything in localStorage and sync across browsers over WebSockets. No server infrastructure, no database, just peer-to-peer updates. But every client owns the full dataset, so two people editing the same thing means conflict resolution, which means vector clocks or CRDTs or operational transforms, and now a BBS emulator has a distributed consensus problem. I spent three weeks once debugging a collaborative editor built on operational transforms. It worked perfectly right up until it didn’t, and then nobody could tell me why.
The second was server ownership: a database behind everything, real APIs, server as the single source of truth. More infrastructure, more things that can break at 3am, but no merge conflicts and no divergent state.
I went with server ownership because it matches what a BBS is — a central system that everyone dials into. The browser becomes a terminal, not a peer. The cost is real infrastructure: Postgres, Redis (well, Dragonfly), NATS, three separate services. The benefit is that “who owns this message” always has one answer.
The Shape of the Stack
┌─────────────────┐
│ PostgreSQL │
│ (Supabase) │
└────────┬────────┘
│
┌──────────┐ ┌─────────────┐ ┌─────┴────────┐
│ Browser │────►│ Gateway │───►│ Services │
│ Terminal │ WS │ (Axum) │gRPC│ (Tonic) │
└──────────┘ └──────┬──────┘ └──────────────┘
│ │
┌──────┴──────┐ ┌──────┴──────┐
│ Dragonfly │ │ NATS │
│ (cache) │ │ (events) │
└─────────────┘ └─────────────┘An Axum gateway handles WebSocket connections from browsers and translates REST requests into gRPC calls. Behind it sit three Rust services, each on its own port. The users service on 50051 handles profiles and online presence, including the “who’s online” listing you’d expect from any BBS. The messaging service on 50052 covers BBS boards, Usenet-style newsgroups, and private email; they’re different enough that I nearly split them and similar enough that I didn’t. If I ever add threading to email I may regret that. The market service on 50053 handles stock quotes, portfolios, and simulated trades.
Each service owns its domain completely. The gateway calls UsersService::GetProfile without knowing how profiles are stored.
The Users Service
Profiles and online presence:
// server/services/users-service/src/service.rs
pub struct UsersServiceImpl {
pool: PgPool,
cache: Option<Cache>,
/// Fallback in-memory store when cache unavailable
online_users: Arc<RwLock<HashSet<String>>>,
}That online_users field is a fallback for when the cache isn’t available. I wanted presence to degrade rather than fail outright, because cache servers go down at inconvenient times. My Dragonfly instance has been solid so far; I’ve been burned by Redis outages before.
For profile lookups, cache first, database second:
async fn get_profile(
&self,
request: Request<GetProfileRequest>,
) -> Result<Response<Profile>, Status> {
let handle = request.into_inner().handle;
// Try cache first
if let Some(profile) = self.get_cached_profile(&handle).await {
return Ok(Response::new(profile));
}
// Cache miss - fetch from database
let row = sqlx::query(
r#"
SELECT id, account_id, handle, display_name, location, bio, interests,
created_at, last_seen_at, is_sysop
FROM bbs_profiles
WHERE handle = $1
"#,
)
.bind(&handle)
.fetch_optional(&self.pool)
.await
.map_err(BackendError::from)?
.ok_or_else(|| BackendError::NotFound(format!("Profile not found: {}", handle)))?;
// ... convert row to Profile, cache it, return
}Profile cache TTL is 5 minutes: long enough to keep profile views off the database, short enough that a bio edit shows up reasonably soon. I picked the number by feel and haven’t load-tested it.
The Messaging Service
Boards, newsgroups, and email share a service but get their own proto definitions:
// server/crates/backend-proto/proto/messaging.proto
service MessagingService {
// BBS
rpc ListBoards(ListBoardsRequest) returns (ListBoardsResponse);
rpc ListBoardMessages(ListBoardMessagesRequest) returns (ListMessagesResponse);
rpc CreateMessage(CreateMessageRequest) returns (BbsMessage);
rpc DeleteMessage(DeleteMessageRequest) returns (DeleteMessageResponse);
// Usenet
rpc ListGroups(ListGroupsRequest) returns (ListGroupsResponse);
rpc ListGroupArticles(ListGroupArticlesRequest) returns (ListArticlesResponse);
rpc CreateArticle(CreateArticleRequest) returns (UsenetArticle);
// Email
rpc ListMailbox(ListMailboxRequest) returns (ListMailboxResponse);
rpc SendEmail(SendEmailRequest) returns (EmailMessage);
}Posting a message involves a bit of ceremony: verify authentication, check whether the board is read-only, then invalidate the cache so the board listing reflects the new message count.
pub async fn create_message(
pool: &PgPool,
cache: Option<&Cache>,
request: Request<CreateMessageRequest>,
) -> Result<Response<BbsMessage>, Status> {
let user_id = auth::require_user_id(&request)?;
let req = request.into_inner();
// Check if board is read-only
let read_only: bool = sqlx::query_scalar("SELECT read_only FROM bbs_boards WHERE id = $1")
.bind(req.board_id)
.fetch_optional(pool)
.await
.map_err(BackendError::from)?
.ok_or_else(|| BackendError::NotFound(format!("Board not found: {}", req.board_id)))?;
if read_only {
return Err(BackendError::Forbidden("Board is read-only".to_string()).into());
}
// Insert message...
// Invalidate boards cache (message counts changed)
invalidate_boards_cache(cache).await;
// Return the created message
}Board listings include message counts, so any post or delete makes the cached list stale. I could update the count in place, but blowing away the entry is simpler and posting isn’t a high-frequency operation.
The Gateway Layer
The gateway translates REST to gRPC so the frontend never has to know about protocol buffers or RPC semantics. It talks HTTP like any other web app.
A typical handler:
// server/src/handlers/backend_bbs.rs
pub async fn create_message(
State(state): State<MessagingClient>,
Path(board_id): Path<i32>,
headers: HeaderMap,
Json(body): Json<CreateMessageBody>,
) -> Response {
// Extract user_id from JWT
let user_id = match extract_user_id(&headers, &state.jwt_config) {
Ok(id) => id,
Err((status, msg)) => return (status, msg).into_response(),
};
let mut client = state.client.clone();
let mut request = tonic::Request::new(CreateMessageRequest {
board_id,
parent_id: body.parent_id.unwrap_or(0),
subject: body.subject,
body: body.body,
});
add_user_metadata(&mut request, &user_id);
let result = client.create_message(request).await;
match result {
Ok(response) => {
let m = response.into_inner();
(StatusCode::CREATED, Json(SchemaBbsMessage::from(m))).into_response()
}
Err(e) => {
let status = match e.code() {
tonic::Code::NotFound => StatusCode::NOT_FOUND,
tonic::Code::PermissionDenied => StatusCode::FORBIDDEN,
tonic::Code::Unauthenticated => StatusCode::UNAUTHORIZED,
_ => StatusCode::INTERNAL_SERVER_ERROR,
};
(status, e.message().to_string()).into_response()
}
}
}JWT validation happens in the gateway, not in the services. An invalid token never reaches gRPC at all, which keeps the services focused on their domain instead of auth ceremony. There’s a defense-in-depth argument for validating in each service too, but then every service needs to understand JWTs, and I didn’t want that coupling.
The gRPC-to-HTTP error mapping is explicit: NotFound becomes 404, PermissionDenied becomes 403, Unauthenticated becomes 401, everything else is a 500 with a message.
NATS for Real-Time Events
This is the part I’m least confident about. Services publish events when state changes, and anyone who cares subscribes.
// server/crates/backend-common/src/nats.rs
pub mod topics {
pub const BBS_MESSAGE_CREATED: &str = "backend.bbs.message.created";
pub const BBS_MESSAGE_DELETED: &str = "backend.bbs.message.deleted";
pub const USENET_ARTICLE_CREATED: &str = "backend.usenet.article.created";
pub const EMAIL_MESSAGE_SENT: &str = "backend.email.message.sent";
pub const USER_ONLINE: &str = "backend.users.online";
pub const STOCK_QUOTE_UPDATED: &str = "backend.stocks.quote.updated";
}If you’re watching board 3 and someone posts, you don’t poll. The messaging service publishes backend.bbs.message.created, the gateway receives it and pushes an update to your browser.
The decoupling is what I like. The messaging service doesn’t know which clients are watching which boards; it publishes and moves on, and the gateway handles fan-out. Adding gateway instances doesn’t change anything about the services.
Whether it’s warranted at my scale is another question. On a good day there might be a dozen concurrent users, and polling would probably be fine. NATS wasn’t hard to set up and it’s the foundation I’d want if this grew, which is the same reasoning I’ve used before on things that never grew.
Docker Compose Deployment
The whole stack runs in containers, so local development is docker compose up and you get roughly what runs in production.
# docker-compose.yml (simplified)
services:
users-service:
build:
context: .
dockerfile: Dockerfile
target: users-service
environment:
- BIND_ADDR=0.0.0.0:50051
- NATS_URL=nats://nats.devlab.ca:4222
- REDIS_URL=redis://dragonfly:6379
ports:
- "50051:50051"
messaging-service:
build:
context: .
dockerfile: Dockerfile
target: messaging-service
environment:
- BIND_ADDR=0.0.0.0:50052
- NATS_URL=nats://nats.devlab.ca:4222
- REDIS_URL=redis://dragonfly:6379
ports:
- "50052:50052"
websocket:
environment:
- USERS_SERVICE_URL=http://users-service:50051
- MESSAGING_SERVICE_URL=http://messaging-service:50052
- MARKET_SERVICE_URL=http://market-service:50053
depends_on:
- users-service
- messaging-service
- market-service
dragonfly:
image: docker.dragonflydb.io/dragonflydb/dragonfly:latest
ports:
- "6379:6379"The cache is Dragonfly rather than Redis. It speaks the Redis protocol, so the services can’t tell the difference, and it was a drop-in swap on a recommendation. Whether it’s actually better for this workload I couldn’t tell you; I haven’t measured.
The Proto Definitions
gRPC types are defined in Protocol Buffers and compiled at build time:
// server/crates/backend-proto/proto/users.proto
syntax = "proto3";
package backend.users;
service UsersService {
rpc ListProfiles(ListProfilesRequest) returns (ListProfilesResponse);
rpc GetProfile(GetProfileRequest) returns (Profile);
rpc UpdateProfile(UpdateProfileRequest) returns (Profile);
rpc GetOnlineUsers(GetOnlineUsersRequest) returns (GetOnlineUsersResponse);
rpc SetOnlineStatus(SetOnlineStatusRequest) returns (SetOnlineStatusResponse);
}
message Profile {
string id = 1;
string account_id = 2;
string handle = 3;
string display_name = 4;
string location = 5;
string bio = 6;
repeated string interests = 7;
string created_at = 8;
string last_seen_at = 9;
bool is_sysop = 10;
}The backend-proto crate generates Rust types from these definitions, so RPC is type-safe without hand-written serialization. Change the proto and forget to update a handler, and the compiler says so. I’ve worked on systems where the API contract was enforced by “please remember to update both sides,” and once shipped a breaking change because I updated the client and not the server. Three hours to work out why half the requests were failing.
What This Buys Me
The constraints, some of which I only recognized partway through: shared state has to live on the server; the browser keeps its REST ergonomics, because I wasn’t rewriting the frontend to speak gRPC; services don’t know about each other; and local development is one command, because anything harder than docker compose up I won’t actually run.
That combination pushed me toward a gateway with an event bus. It’s the standard microservices playbook, not an original design, but naming the constraints kept me from overthinking it.
The technology choices aren’t the interesting part. gRPC and NATS are well understood and I’m not doing anything clever with them. What matters is the ownership boundaries. Message missing? Look at the messaging service. Presence wrong? Users service. That’s a much better position than poking through a monolith where everything touches everything.
The clarity costs wiring. The gateway handlers are full of boilerplate, and a new endpoint means touching three places. I’ve built the monolith version of this before — profiles and messages and stocks sharing a codebase and a connection pool — and it’s fine until it isn’t, and by then you’re stuck.
For a hobby project with a dozen users this is probably overengineered. It was also fun to build, and two people posting to the same board now see each other’s messages.
See also: Journey Day 10: Microservices & Market Data — when the services went live.
See also: Deep Dive: Passkey Authentication — for the auth side of this story.