HN Explorer

Show HN: RemoteStorage – sync localStorage across devices and browsers (github.com)

129 points by pancomplex ↗ HN
Hey HN! Wanted to share a new library and server I've recently been working on called remoteStorage.

When building frontends with Javascript, the native localStorage API is super useful for keeping track of state between sessions in the same browser, but it's not as good a solution when your data needs to be shared across multiple devices or browsers.

For instance, let's say you want to show a welcome modal to all new users that sign up for your product. If you use localStorage to track if a user has already seen this modal, your users will end up getting the experience repeatedly every time they switch devices or browsers, or whenever Chrome decides to nuke your data.

I built remoteStorage to help fix that. Using the same API as localStorage, remoteStorage allows you to easily read and write data on the fly while maintaining state across browsers and devices in order to provide a better user experience.

The project is built as a monorepo in Typescript and contains a JS library that works across any Javascript stack (including Node.js and React Native) and is lightweight (~1 kb minified). The server is built with Nest.js with a disk-persisted Redis Database. It can be deployed in a few minutes using Docker.

One of the biggest challenges when building this was coming up with a secure scheme for handling authentication while still keeping the API dead simple for frontend devs to use. While the project is intended to store non-sensitive data like impression events/preferences, I still wanted to make sure data couldn’t easily leak or be tampered with.

One solution has been to generate a unique secret UUID per user on your own backend to identify the user. Alternatively, you could create a simple wrapper/proxy API around remoteStorage that uses your own authentication method to verify the user's identity before allowing them to access the data (this is super simple to build with React Server Components). Then, you could pick a secure and secret Instance ID that is not publicly available to ensure that only your application can access the data.

Has anyone felt the same pain points with localStorage before? Is this solution useful? Let me know what you think or ideas for how I can improve it :)

68 comments

[ 5.1 ms ] story [ 132 ms ] thread
thatwasunusual
> let's say you want to show a welcome modal to all new users that sign up for your product.

Not sure if this is a good example; how many users sign up for the same product multiple times?

eddd-ddde
They mean that on first login you should see the popup. If you used local storage, login in in a different device would wrongly consider it a first time login.
pancomplex
Yup, exactly. I’ll update the readme to make this more clear.
toomuchtodo
Cookies or passkeys? Seems very complex when you can use something simply to track state and identity. Passkeys sync across cloud storage, for example, even if you’re using a uuid as the username.
cantSpellSober
> while still keeping the API dead simple for frontend devs to use

Selling point seems to be the API, I'd prefer an alternative to document.cookie.split(";"), a sane way to store objects, etc, frontend friendly.

Typically use cookies to store data for the server (session id etc).

pancomplex
That’s right — my main idea with the library is to make it super simple for front end engineers to store preferences/flags server side without needing backend engineering work. Basically as simple as using localStorage itself :)
vidarh
The name Remotestorage is already used for a protocol for portable user-controlled storage. See e.g. Remotestorage.io.
[deleted]
(comment deleted)
paulgb
Your landing page at https://remote.storage/ is so good, I love the minimalism. Six lines of code tell me exactly what it does.
SeanAnderson
It's pretty good... but what if it showed a different landing page after visiting the website on a second machine? :)
pancomplex
Haha good call! Let's do it.
peeters
What mechanism for user tracking could you possibly use to do this?
[deleted]
(comment deleted)
culopatin
I can’t seem to be able to get to this URL. Safari iOS says the url is invalid. I wonder if that means dns issue or “that’s not a url, silly”
peeters
> Six lines of code tell me exactly what it does

I like the minimalism too, but I don't think it tells me "exactly what it does". I have no idea from that page what the backend requirements are. Is it a hosted solution? Is it using some browser profile functionality? Do I have to set up a server?

It demonstrates the simplicity of the client side code for sure. But the server stack is obviously where the details are going to be.

SushiHippie
Not to be confused with https://remotestorage.io/
[deleted]
(comment deleted)
kevincox
Yeah, I thought it was about this old project for a while. Curious why the author started something new, the goals seem very similar.
amadeuspagel
How does this compare to PouchDB[1]?

[1]: https://pouchdb.com/

pancomplex
I didn't actually know about PouchDB but it looks very useful!

I think the main difference here is the use case. remoteStorage is not intended to be used as a database (just like localStorage is not intended to either), but rather to store non PII data such as preferences, impression events, etc. without having to roll and stand up your own DB and API.

maxwellg
On the authentication front - JWTs would be a great fit here!

The developer's backend could issue a JWT, which is validated by the remoteStorage backend. Then, use the `sub` field as the user ID. JWTs are pretty ubiquitous in frontend applications, so many people would be able to reuse their existing auth setup.

pancomplex
Love this idea. Especially because JWTs are so ubiquitous like you mentioned, this could be a very lightweight way to secure the calls without doing too much if any backend work. My main goal is to make it as easy to use as localStorage itself but with all the benefits and security of having a centralized state/store.
shunyaekam
What is special with the server? I was surprised you offered a community-hosted server.

An OpenAPI spec should be enough, right?

lxe
So, just a UUID to authenticate? How do you recover/reset in case it gets lost?
pancomplex
You'd need to keep track of that somewhere for now, but as someone else suggested here, we could soon extend the library to support JWTs for authentication.
matlin
I love the simplicity and kinda wish that was actually built into Chrome.

I know it would probably break things but if Google could automatically sync local storage and IndexedDB when I'm signed into Chrome on all my devices then a lot browser of apps wouldn't require making an account or even a server for that matter.

pancomplex
Thank you! My goal was to make it as stupidly simple as possible in the first version.

I am also pretty surprised Google hasn't yet built something that solves this. When localStorage first came out, I was expecting a natural extension would be a way to sync it to a centralized store to avoid having state reset between sessions/devices.

DANmode
Something tells me they'll be able to find the resources now to productize your code - or something like it.

Thanks very much for improving the landscape and the help pushing the boundaroes of webapps.

This is super cool.

darknavi
I am going to nit pick you a little bit but if you want a good reason to use Firefox, then here it is.

"built into Chrome" implies that Chrome is the only web browser.

simlevesque
That's so cool ! I was thinking how I'd make work syncing my offline first app and this looks promising
matlin
Your app might not need it but if you want partial replication or relational querying in addition to offline-first syncing then check out https://github.com/aspen-cloud/triplit (disclaimer: I'm working on this)
internetter
how does this compare to https://electric-sql.com/
matlin
I think our goals are pretty similar but Triplit offers more flexible storage options (Durable Objects, IndexedDB, SQLite), deeper Typescript integration, and is probably a good deal simpler to operate--we are closer to Firebase/Firestore in that regard. ElectricSQL instead builds on the robustness of Postgres and SQLite.
pancomplex
That's a super valid use case. Let me know how it goes!
ocdtrekkie
> Has anyone felt the same pain points with localStorage before?

For Sandstorm, we've always want to make sure app data is stored on a server, but an increasing number of web apps love to use localStorage as a cheap hack to avoid writing a backend. I recall a bunch of attempts back in the day, but we were always looking for a simple way to include something with an app written to use localStorage which could store the data on a server backend instead.

joshstrange
Very cool overall but the sync->async change means you can't always use it as a drop-in replacement. I assume it's making network calls in those cases which will take significantly longer that local writes. I wonder if something could be done so that it write the data locally and kicks off something async in the background to sync it up to the backend. Maybe with just a timestamp in case you go offline to handle conflicts (or even way to register a callback if there is a conflict so you can decide how to handle it). I know that all makes this more complicated but I'd be worried about adding waiting for a network call where I currently do localStorage calls.

But still a very cool project, thank you for sharing!

pancomplex
Good question. I like the idea of having a configuration that includes optimistic writes and perhaps even reads from localStorage for projects that have very low latency/unstable network requirements.

That being said, the server is blazingly fast as it leverages Redis and Fastify under the hood. In most of my testing, even on WiFi in SF with a server in Oregon, I would achieve <20ms reads/writes.

joshstrange
Totally fair point and I understand that as-is it’s way simpler than what I proposed. I just know there are places I have to change a whole chain of functions from sync to async to use this (not the end of the world).

I’m tempted to rewrite the backend in lambda+dynamoDB for a completely serverless backend (though for anything serious you’d probably want provisioned lambdas to avoid the cold start).

pancomplex
That sounds super useful. If you follow through with it feel free to open a PR with either code or a link to an AWS recipe.

Ideally we have as many different server and client implementations as possible in the monorepo.

thih9
Can this be treated as tracking or storing information about the user in the context of data regulations (like gdpr and similar)?
pancomplex
Depends on what kind of data you are storing? The good news is you can self-host, so it's trivial to spin up a server in your own datacenter that meets your compliance requirements.
eps
How are the conflicts resolved?

Machine A is online and sets "x" to 1.

Machine B is offline and sets "x" to 2.

Machine B goes online.

What the resulting state on A and on B?

What if initially A was offline and B was online?

pancomplex
This is handled by Redis, as such, the last write will win. As the primary key is tied to the user id, there will not really be a concept of Machine A vs Machine B as the end user can only be in one place at a time.
eps
I see. The case in question is when the user switches between two devices, each of which may be sometimes offline. This can cause older changes hitting the server later than the new ones, in which case, ideally, they should be rejected.

What happens to storage requests made while offline? Are they cached locally and propagated once online, or are they lost?

[deleted]
(comment deleted)
Wowfunhappy
I feel like, if the answers to these questions are important, you should be using a more complex solution.
mondrian
Users can have an app in multiple tabs though, and apps can read/write local storage in the background.
bandergirl
The funny thing is that localStorage supports that.

Judging by the title, I thought this tool actually just synced the existing localStorage instead of being its own storage. Since localStorage has onChange events it would be trivial to built, it would still be sync, it wouldn’t require any changes to the existing code, etc

valsk
Based on the comments I think this is just, not yet implemented. But is one of the next steps for truly cross device storage.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
airtonix
how is this different from remotestorage.io ?
stevage
The bit about finding a unique ID for the user is the whole pain point though. The only way to do that really is for the user to create an account on some service that I host. And if I have that, I probably have my own hosted data storage anyway.
pancomplex
You totally _should_ have your own backend / auth when using this. It is not a database.

But it’s dead simple if e.g. using something like Clerk.dev for your auth.

tcper
The use case is limited, a project has it's own backend, why need to add a new sub-system for tiny data storage?
buro9
I would state the opposite, the use case is huge.

To add something new, that needs to persist, will not require going to a backend team and begging for a new API from them.

A common difficulty is the simplest scenario, where do I store this frontend user preference? How do I store this that prevents nagging multiple times across devices?

Backend teams will argue this is a frontend piece of data, and they provide APIs and backend storage that is agnostic of a specific frontend.

This feels like a valuable addition.

antifa
This sounds like something that should just be an extra JSON column in the users table of any SQL database. Backend team shouldn't be withholding backend support if remembering this data is actually important.
mxuribe
At first I got confused, because i thought this was a refreshed version of https://remotestorage.io ....but, i see now that its different/separate.
notso411
What if you assigned each user a unique ID, used this to determine the user session data and persisted this data in a server that was accessible by both web browsers. You could authenticate yourself against this user id by means of a secret only this user knows

It sounds a bit radical but it may work!

valsk
I like this but umm yeah I can roll my own pretty easily. If you had a simple auth solution built in I'd use it!
pancomplex
Stay tuned! Would JWT support fit your use case?
novoreorx
I think you should give an API specification so that people don’t bother using public API or hosting another service could implement it directly in the existing backend.