This is looking very good and it could lower the pain of developping mobile apps, especially for independent studios. Looking forward to learn more about the details of data syncing.
Data syncing is an obvious need and Realm has actually been designed for that from day one. The entire underlying core is build to seamlessly record changes and coordinate them with a backend.
It is not part of the initial launch as we want to nail down the local api's with the community first, but look out for announcements soon.
What kind of consistency do you expect to provide with this future syncing feature? I assume it will be eventually consistent. Is that right? How will conflicts be resolved?
Yes, on mobile sync really only make sense if also works when the device has spotty or no connectivity, so that naturally entails eventual consistency in some form.
We are not ready to into details about how our sync solution will work yet, but watch out for some announcements soon.
EnduroSync (https://orandolabs.com) does exactly the same thing, and it already has syncing. It's available for iOS and Android now, with more platform support coming.
Definitely feel that having good local/offline data handling capacity will help on wearables, because of the intermittent connectivity when your device is not tethered to a phone or tablet.
JP from Realm here, I worked on our (experimental) Swift support. Swift as a language is pretty unstable right now, but it's clear that Apple sees it as the future of iOS and OSX development. So we wanted to start experimenting with how to make Realm feel right at home in Swift.
Looks good. Would really like to see Swift support. Also, the example showing integration with REST services in interesting, seems like you could do more with that.
Realm currently supports Swift, though we're actively working to improve our Swift APIs[0]. You can find Swift sample code throughout our docs[1], and in our Swift examples on GitHub[2].
We learnt a lot about Swift introspection and interacting with Swift objects from the Objective-C runtime. We'll share our findings in some upcoming blog posts, so stay tuned!
The API approach vs. SQLite, etc. is interesting - particularly that it does not look to have very many moving parts. I'm curious about transactional semantics though. Are they 'always on' or is there API control (still haven't gone through docs completely yet, so I may answer my own question later).
Alexander from Realm here, all interactions with the database is through full ACID transactions, but it is a bit untraditional in that the interaction model is build specifically to mobile apps.
The transactions interact with the runloop, so that you are always in an up-to-date read transaction (which always give you a consistent view and never blocks, even when other threads modify the same data). To write, you have to do explicit write transactions.
So your data is always live, consistent and up-to-date without you having to do any explicit coordination (apart from the write transaction on changes) and notifications allow you to always keep your UI up-to-date with the latest changes.
For people using SQLite, it's trivial to do the same with WAL mode and a read-only transaction on the main thread (which is what you should be doing). All writes happen off the main thread, so performance is great.
What does this provide that CoreData doesn't? Whatever it is, it needs to be clearly explained on the landing page. Otherwise it's difficult to understand what problem this solves compared to CoreData.
Hey Tim, from Realm here. The big thing is really ease of use. Our blog post has a few more details on that point [0] but basically what we heard over and over and saw ourselves, is that Core Data quickly gets in your way if your app grows [1][2][3]. In essence, we tried with this initial release to offer a rich feature set with a simple API, less side-effects (especially for multi-threaded apps), and a performance that’s even better than using raw SQLite.
In early testing we got a lot of higher-level questions about what Realm was exactly, so we decided to go for a descriptive tagline instead, but you’re right that we should add more details about how we compare to Core Data!
Core Data has been improved and stabilized by Apple for over a decade, is used internally by many first party apps in iOS, is well documented and fully integrated into Xcode, and heavily "marketed" by Apple at WWDC. Any perceived performance issues can be eliminated or reduced by using faults, indexes and improving queries through NSPredicates. Even one of the blog posts you quote above [3] states that if he was starting a new app today, he'd go with Core Data. So I'm just not seeing a need for yet another ORM solution. I'm not saying you shouldn't continue building it, but you guys need to do a better job explaining the differentiation against Core Data. One area could be sync, where Core Data still needs work, and is limited to iCloud's backend. Perhaps that could be your differentiator? But you're not talking about sync yet. So there's work to do in crafting your message. Fwiw, I'm a senior iOS developer who evaluates these things for a living.
Just to be clear on the topic, since this is an easily misunderstood aspect of Realm: we are not an ORM. We’re not built on SQLite, we’re not mapping to a relational layer, nor are we serializing objects from a third-party data representation: your objects in the language are exactly the same data that’s stored on disk.
We heard a lot of dissatisfaction from the community as far as Core Data is concerned (including from Apple engineers themselves). I’m also not entirely sure you could ever get Core Data performance to the levels we enjoy, since Core Data cannot be any faster than the SQLite underneath, after all. I do want to say that your feedback on how we present ourselves and the work ahead of us is spot on and much appreciated. Many thanks for speaking up, we will work hard to heed your comments & improve.
Just as another data point, I agree with magsafe. I've used Core Data a bit recently and haven't really had any problems with it (though I agree it could be nicer).
My huge, huge, huge problem, on the other hand, is sync. I don't want to get locked into iCloud and the idea of Apple offering multi-platform support is unrealistic.
If you offered this as a cross-platform solution + syncing I'd sign up immediately (it sounds like this might be where you're headed anyway).
CoreData is not as stable as advertised. If you use it simply it works. However advanced use will quickly discover sharp edges. I devote a non-trivial amount of code managing objects between contexts and there are still FRC bugs. (e.g. sorted update.)
I see a lot of comments here about wanting a CoreData replacement that's better at sync, which is exactly the use case CouchbaseLite is meant to solve.
Seriously love this product. What the Realm guys are missing is the fact that Couchbase Lite is really a replication protocol overlayed SQLite. Couchbase Lite did not always use SQLite and is in the process of switching to ForestDB as the underlying persistent store. As far as API's go, I have not seen a more intuitive design than CBL's. It almost makes too much sense. Once you get Sync Gateway setup, you have a mini iCloud. Watch the CloudKit presentations and you'll see that CBLite solves what Apple is leaving up to developers as far as Sync goes.
I think that's because Cloudant's solution is the same replication protocol (CouchDB) talking to it's hosted databases. Thank's for pointing this out, I didn't know about.
The database is stored in a single file that is easily backed up. It does not have any explicit integration with iCloud, but you can easily use it to back up a copy of the db, or you can stream the copy back home yourself via ftp or http.
I'm Kenneth at Realm. As realm databases are files on the phone's file system, you can use the common classes for manipulating them. To do a backup, the NSFileManager class might be useful (the copyItemAtURL method might be what you're looking for). See https://developer.apple.com/library/mac/documentation/Cocoa/... for details.
A big plus for sqlite is that you can use it everywhere. For example, for a REST service instead of return a JSON I just return the sqlite database. Also, I can see the data in the desktop, use it on python, etc.
This could work elsewhere too? Be usable in python?
You can do the same thing with Realm. We actually have a couple of key benefits over SQLite there, which are that our file size is much smaller (usually about 50%), and we don’t require deserialization to get objects out of the file (Realm data can be streamed directly into memory as native objects in your language).
On the minus side, We don’t work from everywhere yet, but we have internal bindings for Python, Ruby, C#, Java, PHP and a few other languages. We focused on the iOS launch so far, but we do plan to wrap up those additional implementations and share them on GitHub.
Sorry about that. We’ve been seeing some issues with our CloudFront caches all morning, but if you give it a hard reload or two in your browser, you should eventually see our Swift samples — we’ll keep working to fix the issue.
The foursquare map video shows how the experience with realm is better (and does that well), but it didn't give me any sense of how it's achieving those results. As a developer, I'd love to see an accompanying explanation with details on how the original version works, and how the realm version works.
Absolutely. We didn’t get a chance to clean up the codebase for this app before the launch (it’s actually using an old version of our API that would be confusingly different than what our docs detail) but we’ll try to set up a blogpost about that since I may not be able to do the topic justice in a comment here :)
The short version is that most apps tend to do a query on click, fetch from an API and then cache the results in memory. The Realm version fetches from the API asynchronously in the back, pre-fetching results and writing them form the DB on one thread and reading & displaying them on the map from another. This has a few benefits (& drawbacks). On one hand, you do hit your servers a bit more often at first, and you may actually get so much data in the map that the UI component will start to lag. On the plus side, your data points are cached so you can do a lot less API queries over time (especially for something like Foursquare where users tend to stay in the same area and the dataset of venues doesn’t change that often). You also get better control as a designer or developer, and you can trim down & display relevant datasets on the fly.
To sum up, I’d say the key difference is the availability of a local, concurrent data structure that can easily be updated & read from multiple places (i.e. in a few lines of code if you look at the samples on http://realm.io)
Thank you! So in theory one could do the same thing writing your own threads and using vanilla local storage? But realm is removing all the boilerplate custom code you'd have to write yourself, and making the local side more performant than custom code would be? Is that it in a nutshell?
Yes, that’s the spirit, although I’m sure others would tell you that using Core Data across threads and at high throughput it much easier said than done :) You can achieve anything with almost any storage technology — we just try to let you do it well, easily and in a very performant & maintainable way.
"You can achieve anything with almost any storage technology" - is actually being very generous to quite a few other technologies :-)
One of the promising benefits of any technology that can make things run magnitudes faster, is actually that you can do totally new things you previous thought impossible.
Cool idea. I had the same problem with the intro video that others have mentioned. I'm gonna read more when the Android version is out, since this might actually be able to fix a common pain point I have with my apps with data persistence (I'd like to be able to read recently touched questions in my Stack Exchange app even when I don't have internet, which requires a damn complex caching layer).
Just a side note: The SO link you have in the footer seems to not have any questions about your actual product in the tag, but about Java realms instead.
Yup, offline mode is definitely a big area of focus for us. Regarding the tag, I’m not sure what the protocol is for these things, but since it wasn’t used for a clear product/project/topic, we figured it was as good a destination as any, since we’d prefer to let everybody benefit from answers we may give to support questions, via SO.
I’m drafting something to dive deeper into the video now; anything in particular you’d like us to dive into?
I'd love deeper info about "we don’t require deserialization to get objects out of the file", but that might be somewhere in the documentation. I'm assuming you're faster than SQLite, so a demo showing those side by side might be a good selling point too.
Realm is not an ORM, but rather an integral part of your language. One key difference is that it does not have to copy all values in the objects back and fourth as you read them and save them, as most ORM's does. So that gives a huge performance improvement.
There are some benchmarks in our intro blogpost[0]. So I can recommend reading that for a bit more background information. We will have more blog posts coming out soon about the technical details.
Wow - this looks really great! As someone who has struggled with Core Data in the past, the API looks great and things seem, on the whole, a lot cleaner. Looking forward to giving it a go.
EDIT: I just came across the REST API / JSON import section in the documentation - really cool stuff!!
I could do with some clarification on what problem the product solves, and actually, what the product even is. To me this looks like a new DB, and an ORM style interface to interact with it?
I like that it is multi-platform, but SQLLite is a pretty mature DB store available on all the major platforms anyway. I get that it is faster... but presumably there are there trade-offs for this speed gain. I would like a better understanding of what they are. Transaction support?
(PS: I would also much prefer if your objects can be typed in a way so they are plain, rather than forced inheritance from RLMObject)
Absolutely. There’s a lot more details in our launch blogpost [0], but you got the right impression: it’s a custom C++ storage engine underneath (not SQLite), that is tightly integrated with the object layer. Not an ORM per se, because the engine isn’t relational, and because there’s not copying/mapping/translation going on — the data on disk is the same data you manipulate in your language.
The big problems we try to solve are really ease of use & performance. We heard from many developers dissatisfied with the current options on mobile, and the relative complexity to accomplish tasks like accessing data across threads, JSON handling, performance on graph queries, etc. — which the reactions on Twitter seem to confirm. We’ll write a lot more about the tradeoffs in coming weeks, but we’ve really grabbed a lot of perf cycles just by taking advantage of modern techniques such as zero-copy architectures, bitpacking and vectorization. Of course the big implicit tradeoff is that you give up a bit of the relational paradigm in exchange for that speed, although we do maintain ACID transactions, immediate disk persistence by default and schemas.
Completely hear you on wanting plain objects — I personally feel that way too! Unfortunately, there’s no way to achieve what we do without making you inherit from RLMObject, although we do try to make those behave as close as possible to NSObject.
Although I don't know exactly what the implications are just yet, I've created a category on one of my RLMObject subclasses which accepts an NSArray or NSDictionary property and in the getter/setter converts to and from the NSData backing property. The only thing is after [Venue creatInDefaultRealmWithObject:] I have to explicity set the value for it to propogate to the backing NSData property.
It's extra work, but it lets me manipulate Realm into doing what I want it to do to store data which doesn't conform to an explicit schema. (e.g. a Venue which has an array of address strings, where I don't want the headache of iterating through venue.address[i].string values:
(note: this doesn't look pretty, its doing some funky formatting!)
"venue" : {
"name":"Test Venue",
"address" : [
"address line 1",
"address line 2",
"address line 3 for rare instances"
]
}
I would love to share the swift-specific portion of the getting started docs on www.sososwift.com but http://realm.io/docs/ios/0.80.0/#swift doesn't navigate directly to that. If there's a link to Swift instructions for realm, I'll post them and you will find them here http://www.sososwift.com/?a=realm.io Good luck, looks cool!
CloudFront is giving us a lot of issues at the moment, but I implemented + deployed that feature for you! Hopefully it will be live very soon. (btw, thanks for creating & running SoSoSwift!)
Hey great, it worked! Sorry for the delay, I had to work at my actual job. :/ BTW, we just deployed a new app last week, had a bunch of problems with CloudFront. Good luck!
I was going to say Android already has SQLite built-in, but they do a good job explaining advantages vs that (more object like syntax, more document like vs. table and schema like, better performance, etc.). Good job.
I don’t think anyone at Realm has ever tried, but I doubt it would work at this point. We do a lot of things specific to each language in the implementation, so we’d have to support RubyMotion explicitly for it work.
RubyMotion is Objective-C under some thin covers so it seems like it would be relatively straightforward to have it work with Realm without a lot of extra trouble.
True but there’s a lot of work that goes into making your Objective-C RLMObjects work seamlessly with the storage underneath (i.e. without copying), so at least that part wouldn’t work without an additional implementation in Realm.
Someone is already looking into it on the ML though[0] so we’ll see how far we can get it going.
LOVE this. I've dealt with the pain of multiple CoreData contexts in different threads and ugggh. This was easily the longest part of our Streak app development, as we needed it to be fast, offline, and fully synced. This is definitely a way better API that accomplishes the same thing as CoreData. If its actually faster too, wow, just amazing.
One question - CoreData works really well with built in UIViewControllers like TableViewControllers. Any chance you'll be releasing some code that makes wiring up changes in the models to the controls that are displaying that data?
Android is coming! We have an internal build that’s almost at parity with iOS but still needs a bit of tuning and is missing a few key features like Migrations. We’ll work to release & open-source it soon. You can register to be notified of that [0].
As good as Realm looks for iOS, I think it might be needed more in Android as there are no really viable alternatives other than using sql lite directly.
Thanks for the kind words! Regarding wiring this to a TableViewController, we have some code samples already for Swift[0] and Objective-C[1]. Right now you get notified of changes and refresh, but we’re very interested in exposing a reactive-style binding mechanism as well. The storage engine underneath supports that very well actually, so stay tuned…
Absolutely! We’ll be working towards this as fast as possible. We already have support for .Net on an internal repo but crafting a great API & performance is a long process.
The main benefit of using CoreData is seamless faulting behavior. It's not obvious from the example or the site that Realm supports something comparable. How does Realm deal with a lot of objects?
Realm does not have a concept of faulting like Core Data does. In Core Data every object is essentially a copy of a representation in the underlying data store (usually sqlite), so instantiating the object is quite expensive as all the properties have to be retrieved and copied into the object, and before you have done that it is very limited what you can do with the data.
In Realm the object you see is a direct representation of the underlying data, it does not have to copy in all the properties. When you access them, it is the property directly from the db you are getting, not a copy stored in the object. This means that it takes up a fraction of the memory space, and it also means that queries and similar operations can work on the objects directly in the db, without having to instantiate them in objective-c. This means that you can easily scan over millions of objects, without causing a single "fault".
When manually traversing objects you still have to instantiate them, but since they don't have to copy any data, it is a far more lightweight process than faulting.
The RealmBrowser (which can be found under /tools/ at the above source link) is already using Realm internally to show/edit any Realm file dynamically.
104 comments
[ 5.9 ms ] story [ 117 ms ] threadIt is not part of the initial launch as we want to nail down the local api's with the community first, but look out for announcements soon.
We are not ready to into details about how our sync solution will work yet, but watch out for some announcements soon.
https://news.ycombinator.com/item?id=8037029
We learnt a lot about Swift introspection and interacting with Swift objects from the Objective-C runtime. We'll share our findings in some upcoming blog posts, so stay tuned!
[0]: https://github.com/realm/realm-cocoa/pull/549 [1]: http://realm.io/docs [2]: https://github.com/realm/realm-cocoa/tree/master/examples/sw...
The transactions interact with the runloop, so that you are always in an up-to-date read transaction (which always give you a consistent view and never blocks, even when other threads modify the same data). To write, you have to do explicit write transactions.
So your data is always live, consistent and up-to-date without you having to do any explicit coordination (apart from the write transaction on changes) and notifications allow you to always keep your UI up-to-date with the latest changes.
In early testing we got a lot of higher-level questions about what Realm was exactly, so we decided to go for a descriptive tagline instead, but you’re right that we should add more details about how we compare to Core Data!
(EDITS: formatting)We heard a lot of dissatisfaction from the community as far as Core Data is concerned (including from Apple engineers themselves). I’m also not entirely sure you could ever get Core Data performance to the levels we enjoy, since Core Data cannot be any faster than the SQLite underneath, after all. I do want to say that your feedback on how we present ourselves and the work ahead of us is spot on and much appreciated. Many thanks for speaking up, we will work hard to heed your comments & improve.
My huge, huge, huge problem, on the other hand, is sync. I don't want to get locked into iCloud and the idea of Apple offering multi-platform support is unrealistic.
If you offered this as a cross-platform solution + syncing I'd sign up immediately (it sounds like this might be where you're headed anyway).
And it's all closed source.
http://developer.couchbase.com/mobile/#couchbase-lite
I see a lot of comments here about wanting a CoreData replacement that's better at sync, which is exactly the use case CouchbaseLite is meant to solve.
https://cloudant.com/product/cloudant-features/sync/
This could work elsewhere too? Be usable in python?
On the minus side, We don’t work from everywhere yet, but we have internal bindings for Python, Ruby, C#, Java, PHP and a few other languages. We focused on the iOS launch so far, but we do plan to wrap up those additional implementations and share them on GitHub.
The short version is that most apps tend to do a query on click, fetch from an API and then cache the results in memory. The Realm version fetches from the API asynchronously in the back, pre-fetching results and writing them form the DB on one thread and reading & displaying them on the map from another. This has a few benefits (& drawbacks). On one hand, you do hit your servers a bit more often at first, and you may actually get so much data in the map that the UI component will start to lag. On the plus side, your data points are cached so you can do a lot less API queries over time (especially for something like Foursquare where users tend to stay in the same area and the dataset of venues doesn’t change that often). You also get better control as a designer or developer, and you can trim down & display relevant datasets on the fly.
To sum up, I’d say the key difference is the availability of a local, concurrent data structure that can easily be updated & read from multiple places (i.e. in a few lines of code if you look at the samples on http://realm.io)
Just a side note: The SO link you have in the footer seems to not have any questions about your actual product in the tag, but about Java realms instead.
I’m drafting something to dive deeper into the video now; anything in particular you’d like us to dive into?
There are some benchmarks in our intro blogpost[0]. So I can recommend reading that for a bit more background information. We will have more blog posts coming out soon about the technical details.
[0] http://realm.io/news/introducing-realm/
EDIT: I just came across the REST API / JSON import section in the documentation - really cool stuff!!
I like that it is multi-platform, but SQLLite is a pretty mature DB store available on all the major platforms anyway. I get that it is faster... but presumably there are there trade-offs for this speed gain. I would like a better understanding of what they are. Transaction support?
(PS: I would also much prefer if your objects can be typed in a way so they are plain, rather than forced inheritance from RLMObject)
EDIT: The blog post at http://realm.io/news/introducing-realm/ answers some of these questions.
The big problems we try to solve are really ease of use & performance. We heard from many developers dissatisfied with the current options on mobile, and the relative complexity to accomplish tasks like accessing data across threads, JSON handling, performance on graph queries, etc. — which the reactions on Twitter seem to confirm. We’ll write a lot more about the tradeoffs in coming weeks, but we’ve really grabbed a lot of perf cycles just by taking advantage of modern techniques such as zero-copy architectures, bitpacking and vectorization. Of course the big implicit tradeoff is that you give up a bit of the relational paradigm in exchange for that speed, although we do maintain ACID transactions, immediate disk persistence by default and schemas.
Completely hear you on wanting plain objects — I personally feel that way too! Unfortunately, there’s no way to achieve what we do without making you inherit from RLMObject, although we do try to make those behave as close as possible to NSObject.
[0] http://realm.io/news/introducing-realm/
It's extra work, but it lets me manipulate Realm into doing what I want it to do to store data which doesn't conform to an explicit schema. (e.g. a Venue which has an array of address strings, where I don't want the headache of iterating through venue.address[i].string values:
(note: this doesn't look pretty, its doing some funky formatting!)
"venue" : { "name":"Test Venue", "address" : [ "address line 1", "address line 2", "address line 3 for rare instances" ] }
@interface Venue : RLMObject @property NSString name; @property NSData addressData; @end
@interface Venue (AddressToData) @property NSArray address; @end
+ (NSDictionary )defaultPropertyValues { return @{ @"addressData":[NSJSONSerialization dataWithJSONObject:@[] options:0 error:nil] }; }
- (NSArray) address { NSArray address = nil;
}- (void) setAddress:(NSArray )address { if (!address) { address = @[]; }
Someone is already looking into it on the ML though[0] so we’ll see how far we can get it going.
[0] https://groups.google.com/forum/#!topic/realm-users/Dn0wg0Uk...
One question - CoreData works really well with built in UIViewControllers like TableViewControllers. Any chance you'll be releasing some code that makes wiring up changes in the models to the controls that are displaying that data?
[0] http://realm.us5.list-manage1.com/subscribe?u=2aab5198c2f56b...
As good as Realm looks for iOS, I think it might be needed more in Android as there are no really viable alternatives other than using sql lite directly.
In Realm the object you see is a direct representation of the underlying data, it does not have to copy in all the properties. When you access them, it is the property directly from the db you are getting, not a copy stored in the object. This means that it takes up a fraction of the memory space, and it also means that queries and similar operations can work on the objects directly in the db, without having to instantiate them in objective-c. This means that you can easily scan over millions of objects, without causing a single "fault".
When manually traversing objects you still have to instantiate them, but since they don't have to copy any data, it is a far more lightweight process than faulting.
[0] https://github.com/realm/realm-cocoa#building-realm