One thing I would love to see is the addition of wildcard addresses like the way google has and microsoft added (user+site_string@domain.com).

Thanks for your hard work on a great product!

My colleague wrote down our thoughts in https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri....

The other aspect of CRDT that's in some scary is what @mdpye explains in https://news.ycombinator.com/item?id=24621113. With OT, if we get the corner cases right, we can tune and extend the implementation. With CRDT that might be game over.

In the link, OT is aliased to "Operational Transformations"

It's equally "disrespectful" to waste reader's time on 101 content if that's now what the post is about.

The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future.

The material scientist may hit a dead end, or their company may axe their entire department to "cut costs".

One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded.

I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.

> So, having said all of that, is there hope for adding things like this to my toolbelt?

Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know.

We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.

Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all.

Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way.

Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.

See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move.

Good luck. I believe in you.

In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory.

I included all the resources I used to learn the concept on the page too.

It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources ).

I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating.

I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.

I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work.

I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.

HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms.

Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly).

For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.

Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)

I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas.

I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough.

That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!

And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore.

If you need to understand certain things at a deeper level, then explore the classes and other guided tuition.

I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft.

You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.

Get your intuition down really good and 10+ years is not required.

I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0].

[0] https://www.confluent.io/stream-processing/

Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.

Turns your front end app multiplayer with a library.

They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.

There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.

https://news.ycombinator.com/item?id=19886883

I see text editing as the beachhead for this tech. Text editing is hard enough that other systems don’t work well, so you’re kind of forced to use OT or CRDTs to make it work. And text documents are lists of characters - so once you’ve made it work there you have an implementation that can also sync arbitrary lists. At that point, editing maps (js objects) and adding support for arbitrary moves and reparenting will allow us to add real-time editing to a lot more software.

I think there’s a lot of interesting software architectures that can be enabled by making everything in a system a CRDT document. But text is still the beachhead. And a good litmus test for performance. And an important application in its own right.

There's also 2-year old list of CRDTs used in prod [2].

I know Azure CosmosDB (NoSQL) advertises it's own use of CRDTs for global replication.

[1] https://www.infoworld.com/article/3305321/when-to-use-a-crdt...

[2] https://github.com/ipfs/notes/issues/404

However in my experience most games's shared world state cannot be reconciled if one player's action is inserted in time prior to another player action. Or future actions reveal hidden information.

Even turn based games have turn timers. Including Jackbox games!

Among public databases there may only be two correct CRDT implementations in the world - Cassandra's and etcd's. Compare to how many database products make CRDT or CRDT-like promises, and how few probably actually work.

Lots of software promises CRDTs-like-over-actors, e.g. Erlang, the new Cloudflare product. Not really qualified to talk about them.

Most applications normal people use boil down to chat conversations - chat apps, Gmail, Facebook, collaborative text like you said, all sorts of log-like things - or serial, recommendation-driven consumption - YouTube, Netflix, which use incremental / batched cells in a matrix to update their recs. This stuff is insensitive to order and time. Then there's e-commerce, which is inherently linear, centralized, single-actor, all-or-nothing transactions, CRDTs provide no advantages here.

It's tricky. On the one hand you'd get this interesting discussion of CRDT applications. On the other it may wither under intense scrutiny.

We released the our collaborative Jupyter notebook in 2014 as a plugin to Jupyter classic. We then iterated on what we learned over the years, completely rewriting everything multiple times, including the entire realtime collaboration stack. Cocalc's Jupyter support is pretty mature and battle tested at this point, and also includes a TimeTravel slider that lets you view all past versions of a Jupyter notebook and integrated chat.

I was a college professor (at Univ of Washington), I started a company around this in 2015, so CoCalc has soo far been mainly aimed at serving the needs of academics teaching courses. It's been increasingly popular lately, e.g., in the last month over a half million distinct Jupyter notebooks were edited on https://cocalc.com. Of course, many of these notebooks are homework problems. Anyway, our company is doing very well, and we hope it will eventually be a "10M startup opportunity". :-)

The problem spaces are quite different. Problems that emerge on a minute-by-minute basis in collaborative editing emerge on a week-by-week basis in source control, and when the problems emerge in the latter, they tend to be much larger (because you can build up a much bigger merge conflict on a routine basis with the big chunks you're making).

Yes, it's true that if you squint hard, it looks like version control is a subset of collaborative editing, but I'd be really hesitant to, say, try to start a start-up based on that observation, because even if we take for the sake of argument that it's a good idea to use the same underlying data structures, the UI affordances you're going to need to navigate the problem space are going to be very different, and some of the obvious ways of trying to "fix" that would be awful, e.g., yes, you could give me a "collaborative space" where I see what everybody's doing in their code in real time... but it's a feature, not a bug, that when I'm working on a feature I'm isolated from what everyone else is doing at that exact moment. When I run the compiler and it errors out, it's really, really nice to have a good idea that it's my change that produced that result.

(I'm aware that collaborative editing also has the "I was offline for a week and here's a bunch of conflicts", but I'm thinking in terms of UI paradigms. That's not the common case for most/all collaborative editing systems.)

[1]: Not saying the only solution is the one we had now. A magic genie that watched over the code and made commits for you at exactly the right level of granularity would be great, so you'd never lose any useful context. But key-by-key isn't that level of granularity.

So we ended up essentially implementing what you laid out, an in-memory revision control system, although using a bit more formal methods to reason about divergence/convergence of clients. The most basic operation was the "diamond merge": given operation x:A->B, y:A->C, construct x':C->D, y':B->D such that x' . y == y' . x It also had to satisfy certain other algebraic laws, notably diamond composition, which allowed us to compose these merging operations whenever we wanted, guaranteeing that the clients will eventually converge to the same data state. It was quite neat! Shame that it's all proprietary.

Good old days. I remember, the most pesky operation was implementing a good undo-redo algorithm, it's quite tricky, even once you add inverses.

There's a next level of VCS forming on the horizon, in some combination of CRDTs, patch theory, and grammar-aware diffing.

Which should also learn from fossil, and consider metadata such as issues and surrounding discussions to be a part of the repo.

A really robust solution would also be aware of dependencies and build systems, and even deployment: I see these as all fundamentally related, and connected to versioning in a way that should be reflected and tracked through software.

Much of real-time collaboration goes back to networking and real-time networking used in distributed multi-user systems like games, where simulations need to sync on a server. In games though, Dead Reckoning [2] is used as well as interpolation and extrapolation in prediction, much of it can be slightly different for instance with physics/effect, but messages that are important to all like scores or game start/end are reliably synced and determined on the server.

[1] https://visualstudio.microsoft.com/services/github-codespace...

[2] https://www.gamasutra.com/view/feature/131638/dead_reckoning...

People think of OT / CRDT as realtime algorithms for realtime collaborative editing because they're always programmed and used that way. But the conflict resolution approach doesn't have to merge everything as-is. You could build a CRDT or OT system that generated VCS-style conflicts if concurrent edits happen on the same line of code. To make it a valid OT / CRDT algorithm the main constraint is just that every peer needs to resolve conflicts the same way. (So if I merge your changes or you merge my changes, we end up with identical document states). It would be easier to implement using OT because you only have to consider the interaction between two peers. But I think its definitely doable in a CRDT as well.

I think having something that seamlessly worked in both pair programming setups and with git style feature branches & merging would be fantastic.

I have a lot of thoughts about this and would be happy to talk more about it with folks in this space.

Version control works wonders with line-oriented stuff, which covers more or less every programming language in existence.

It doesn't do so well with non-line-oriented structured formats such as XML (not sure how JSON or TOML) fits in here).

Given that collaborative editing typically works with non-line-oriented data formats, you can see the issue, I think.

CRDT as a service.

Does it make sense for us as an opensource community to invest our time and energy making one really good CRDT (with implementations in a few languages). Or does it make sense for us to distribute that energy between a bunch of CRDT and OT implementations, with different performance tradeoffs?

My take is that its been hugely beneficial to us all that JSON is a standard, because I can use it from every language, and have confidence that the implementations are fast and good quality. I think we have an opportunity to make that for a good CRDT too. Even if OT would work fine in your architecture, if we have a great, capable, fast CRDT kicking around, it could be a reasonable default for most people. And my claim is that the performance difference between CRDTs and OT is smaller than the difference between high and low quality implementations. (I expect a well written CRDT in wasm will outperform my OT code in javascript.)

I believe it's better to focus on kits of parts--API's and/or self-contained functions--that can be combined or ignored as needed, along with a variety of reference application samples.

Having lots of ways to easily filter and sort content is also very useful. For example, filtering and/or sorting annotations by person, group, date, content (sub-strings) is very useful. A query-by-example kind of interface is nice for this.

So yes, hundreds of people can edit a string and produce a coherent result at the end. Contiguous runs of characters will stick together and interleave with concurrent edits.

And here's the thing: Can 100 people edit a document, even in theory, and have it make sense? I think the answer is "no," with or without technology.

I'm sure there are other uses for these data structures, but shared editing is always the example I read about.

If I was going to take an attempt at justifying the importance of CRDTs, I would say:

CRDTs are the future because they solve digital document-level conflict.

They don't bypass the problem the way that diff/patch/git conflict resolution does, by requiring human intervention.

Instead they truly and utterly obliterate the digital conflict resolution problem: a group of people editing a document can separately lose network connectivity, use different network transports, reconvene as a subgroup of the original editors... and their collective edits will always be resolved automatically by software into a deterministic document that fits within the original schema.

If viable, this has far-reaching implications, particularly related to cloud-based document and sharing systems.

Any multiplayer game does this. Git does this as well.

So of course you can do this, it's a matter of how you reconcile conflicts. Real-time interactive games will generally choose a FIFO ordering based on what came into the server's NIC first. Git makes the person pushing the merge reconcile first.

For docs, live editing seems to work the same as in games. Reconciliation for the decentralized workflow will be interesting, but it's just going to be minimizing the hit to a user when their version loses the argument.

But for a contrived example, a game with hundreds of players, backed by an enormous JSON document, where the game engine is in charge of making sure each move makes sense: A CRDT could enable that, and each player could save a snapshot of the game state as a simple text file, or save the entire history as the whole CRDT.

Or as a less contrived example, instead of a game, it's a chat client, and it provides rich text a la Matrix, but there's no server, it's all resolved with CRDTs and all data is kept client-local for each client.

There are a lot of cool things you can build with a performant CRDT.

Here is an interview with someone using CRDTs to build an edge state product that answers this question at a high level.

https://www.infoq.com/articles/state-edge-peter-bourgon

I think in practice what you would do (if your use case allowed it) is use CRDTs, but periodically checkpoint and trim them when you know everyone has synced. That gives you very similar properties to the video game world and still has the features of not losing peoples edits when they make them offline.

Currently we are in the "discovery of new crdts" and "engineering and implementing of older crdts reliably" phase, and in some cases "discovering when not to use crdts".

The crux of the this issue is that crdts that play nice with human expectations in regards to collaborative document editing are not known, possibly excepting automerge (yjs). As it's a 'softer' concept will no good axioms, there is no solid theory on how to combine the theoretical requirements of crdts with human expectations.

https://docs.riak.com/riak/kv/latest/developing/data-types/i...

> 1. Adding wins over deleting.

yeah, so, _maybe_ you can remove elements from your set. If you're lucky. I dunno about all that...

I can't speak to its usability as I'm waiting on a 1.0

My sense is that it's not a particularly flexible approach, and falters under adverse network conditions, even though it feels very tempting.

[1] https://en.wikipedia.org/wiki/Worse_is_better

Any time you see mention of "a server", you can in fact replace it with "a synchronised collection of servers acting as one".

OT involves logic on the server, not just storage, so it's not really OT as generally meant, if using S3 and a collection of peer clients only running client logic. S3 doesn't have enough interesting logic for that.

However, if you try the thought experiment of stretching "a synchronised collection of servers" to be all of the peers, no S3 even required, and then do OT with that, you can!

The result behaves exactly like OT in terms of things like document editing, conflict resolution and history garbage collection, rather than behaving like a CRDT.

It has different timing and reliability characteristics from single-server OT though. Those characteristics depend on how the virtual server is implemented on top of the synchronised peers, and especially how it implements peers coming and going.

If that sounds like OT-on-p2p has similarities to CRDT-on-p2p - they do, and they are not the same. Roughly speaking, CRDT-on-p2p has lower latency relaying updates between interested peers, because it doesn't need to globally synchronise. However with some fancy techniques you can make OT-on-p2p fast most of the time as well, and retain some of the OT benefits.

Those two behave differently but there are some common characteristics. Once you have the cluster idea, it's not out of the question to mix and match bits of OT, CRDT and other kinds of Distributed Transaction on a per-operation basis for different ops on the same data, depending on the characteristics you want them to have.

There are many trade-offs in the characteristics.

If you squint a lot, that's sort of, kind of, what realtime network games do implicitly, without a clear theory underlying it. They also add predictions and interpolations.

Actually, CRDT is not a single data structure or even algorithm. It is a term for several families of data structures and different algorithms on them. If your task is not editing text, you may find a simple and already implemented CRDT for your case.

Try Y.js: https://github.com/yjs/yjs

The most recent guideline we added says: "Please don't complain about website formatting, back-button breakage, and similar annoyances. They're too common to be interesting." I suppose that complaints about writing style fall under the same umbrella.

Not that these things don't matter (when helping people with their pieces for HN I always tell them to define jargon at point of introduction), but they matter much less than the overall specific topic and much less than the attention they end up getting. So they're basically like weeds that grow and choke out the flowers.

(This is not a personal criticism—of course you didn't mean to have this effect.)

https://news.ycombinator.com/newsguidelines.html

[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

I found the article quite good, and if you had genuinely been motivated to engage with the content you could have highlighted the acronym and searched for it. There is a wealth of good info for "CRDTs" that comes up on the first page of Google, Bing or DDG.

Does the acronym actually illuminate what they are or how they function? I submit to you that it probably doesn't.

It's called the link. All an author has to do is link the first instance of an acronym or piece of jargon to some authoritative description, and you get the best of both worlds: readers familiar with e.g. CRDTs[0] can just keep reading, and the rest can click the link and find out.

[0]: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

I read the title, wondered what CRDT was, and started reading. In the back of my mind I was wondering what CRDT was, but reading the article felt like I was going on a journey. Every term that needed to be defined was defined. Finally, when CRDT was mentioned in the article, it was immediately defined.

I generally agree that throwing acronyms around without defining them is not fair to the reader, but I don't think this article did that at all.

Mods, can we expand the acronym in the title of this submission please?

This is way over the top.

I thought the author did an amazing job of discussing a highly technical topic in a very approachable way. Every blog on HN should aspire to write like this! It was so good it got me reading other posts even.

Yes, it would have been nice for us non- domain experts if the author had done the classic "Conflict-free replicated data type (CRDT)" thing, but you can easily just say that, ya know? "Hey, it would be helpful if you expanded CRDT early on."

Engineers, when talking about technical concepts with acronyms, always expand them for the first time to your readers!

In contrast, I like way more a different approach on explaining (mostly see it on Cyrillic forums) -- instead of guiding you by hand, they just give you clues where to look for. That way, knowledge givers are way more approachable, because it costs them very little to chat back something like "look for CRDT", than go into in-depth explaining. In the end -- there's way more information, and from top experts in the fields.

I’m a little embarrassed to admit I didn’t even notice.