Ugh, if you were linking against system sqlite you had problems RIGHT NOW before Android N.
sqlite.so isn't part of public Android API surface, which means the OEMs will regularly replace it, break it, compile it with strange options and deal a lot of pain to you and your users.
So if you weren't shipping your own native SQLite in NDK/Xamarin you already had problems, you just maybe didn't notice them.
Android N is now actually actively preventing linking against system libraries which aren't part of public API surface, which is good - it'll result in less developers doing dumb things which break on minor updates.
Facebook is one of the most feature-rich apps available for Android. With features like push notifications, news feed, and an embedded version of Facebook Messenger (a complete app in its own right) all working together in real-time, the complexity and volume of code creates technical challenges that few, if any, other Android developers face
I've never understood why the Facebook client is so heavy. That isn't a long list of features! It seems like the requirements could be written on the back of an envelope.
Obviously part of the reason is that they move fast and have a lot of developers, so there's much more pressure to add code than to keep it trimmed down. But the client still doesn't do all that much. Surely, surely most of the logic is handled on the server.
I've noticed they are now doing a similar thing now for push notifications.
Instead of using the platform-provided, battery efficient GCM (now FCM), they insist on using their own MQTT push solution.
Not only does this require a constantly running background service and a TCP socket which has to be reopened on every connectivity change (e.g. a switch from wi-fi to mobile data): They do it for every single one of their apps.
My phone has at least three of them (Instagram, Facebook, Messsenger), and I'm pretty sure that this is a major reason for battery life problems on Android.
The Facebook mobile web page has been going through so much churn lately, I feel like they are working to make it less desirable relative to the app. Disabling access to messages was one of the big ones, but they also regularly change how the sharing links work (to where you can or cannot share from a given action). One of the most recent and enjoyable changes is that the unread count badge on notifications updates in the background, but the actual notification drop down doesn't. You have to go to the notifications page (see all notifications) to see what the new notification actually is.
But the point is that you ship your own copy of SQLite you still have a problem. You need to make sure that a database is never accessed both through the android wrapper and you own sqlite copy, or you will face corruption. Seems trivial, but when you take into consideration things like Xamarin and third party libraries, it's not so clear-cut.
Kind of related: There is an outstanding request [0] on the Android issue tracker to offer an official support/compat library for Android's sqlite bindings, that would bridge Android's API (in a different java package) to a sqlite binary that you provide.
There are already implementations [1][2] of this, but because it is not provided by the Android team it is hardly supported by ORMs, libraries, etc.
I don't think so but I might also interpret the article wrong.
The problem - from what I understand - is that you used to be able to use the systems sqlite.so to access databases owned by your app from the native environment. With Android N you are no longer able to do this, you need to ship your own sqlite binary.
This can lead to problems when you want to access the same sqlite files from both the Android platform APIs and native (e.g. use it from native for business logic, debug it with a tool like Stetho [0] that uses android.database.sqlite) because of a version mismatch.
Would Google offer (a copy of) the Android API with the ability to plug in a different sqlite-compatible binaries, it would very likely find broad adoption. Then the problem stated by the article could be solved by shipping the app with a sqlite build that is used by both the native code and plugged into the compat library.
One big danger seems to be that POSIX locks don't like having two fds for the same file in the same process. But Linux has a newish kind of lock that is backwards-compatible and works better. I would love for SQLite to use it.
Patches are accepted, though.. SQLite is, by all accounts I've read, a very well written and tested piece of software, and I'd be surprised if there was an easy and safe win like this which hasn't already been proposed / considered.
Though, my guess shouldn't stop you looking for previous suggestions to implement this, and if there are none, submitting it yourself :)
POSIX locks semantics are implementation defined, hence why they are pretty much useless unless all applications access the same set of files play by the same rules.
If you have native code that wants to access a database, not round trip though Java, and not incur the size hit of shipping a separate copy OR (and this is key) want your Java code to be able to read the same database without writing your own Java->JNI SQLite wrapper.
You have no idea what version of sqlite the OS might be using, so you can't rely on consistent behavior. In many cases the vendor will have tinkered with it and messed it up. Comprehensively testing on all Android devices is basically impossible because there are so many models out there.
The reason they can tinker with it is that it's not public; if it were public it would (hopefully) be better-defined, locked down and stable, part of the Android compatibility test suite.
That feature request would let people supply their own sqlite.so, and also redirect the Java API to use it. Seems like a very promising approach.
Because using the NDK only solves a very specific set of problems (and brings it's own can of worms in the wild), not that many apps will directly be affected by this.
The article finishes by asking: Would it really be so bad to include libsqlite in the NDK? Good question!
Why is the NDK so underpowered? Why are they so reluctant (or unable) to publish some useful, stable native APIs? It's missing things like curl, sqlite, icu, OpenSSL. Most of these are built into the OS but not accessible via the NDK. (OpenSSL isn't a great library, I know, but it's patchable. And they could start now with something better.)
The only useful native third-party libraries you get are: zlib, OpenGL ES, OpenSL ES.
Apart from that, there are a bunch of wrappers for a small subset of Java APIs (AssetManager, Bitmap and so on). These are tedious to use, and seem to have been written by hand so in some cases they're buggy -- for example, incorrect string handling in AStorageManager: https://code.google.com/p/android/issues/detail?id=41983
Then they have to maintain those versions of those libs forever. Most devices don't have enough storage space for 9 copies of the past major versions of all of these libs.
Well, that's the importance of stable APIs. You only need a new copy on major revisions, which are rare. libcurl, for example, claims to have had a stable API and ABI since 2006 (https://curl.haxx.se/libcurl/features.html#stableapi). Why do we only get zlib?
I bet many Android devices already have at least 9 copies of sqlite, since so many apps ship their own copy.
Most likely Google (and OEMs) want to keep the flexibility of changing implementations without having to code and patch those libraries you listed forever.
NDK team doesn't look like it's expansive or prioritized at all, so I doubt they have resources or will to do that. TBH I'd rather they focus at improving the tooling and debugging, we can ship our own libraries with our own options compiled in. Heck even on iOS we have to ship a custom SQLite to get all the needed capability.
Can anybody speak to the current size of the NDK team? I met with one of them many years ago when the team had only two (2) full-time engineers.
Google has one of the largest software engineering populations of any company in the world. It supports one of the largest developer ecosystems in the world. Games are the largest catagory in their ecosystem. 2 engineers supporting the NDK...
As an earlier commenter mentioned, games are one of the biggest app categories and many games rely on the NDK. Pro audio could be another big category, but I think Apple has the market sewn up there because Android is still way behind.
Same thing seems to be the case with their messaging team. I don't know how many they are, but even before they spread themselves thin by releasing Allo and Duo they lagged behind the competition and released rarely and with few differences when they did. I have no idea what Google uses their engineers for, it doesn't seem to be Android related anyway. I think Reader went away because they didn't have anyone to maintain it.
There were plenty of people inside Google who would have liked to maintain Reader, but management explicitly made the decision to shut it down.
(I'm not saying that there's anything inherently unreasonable about that, although I happen to disagree with the decision. But these things cost money and they decided they didn't want to spend the money.)
I'm not sure Skia has a sufficiently stable API. And graphics are actually an OK case -- OpenGL ES is the key API here, and it's provided.
What you want is stable APIs for things that interact in fiddly ways with the rest of the system, so they're more easily managed at the OS level. Networking, shared databases. (Edit to add: localization)
zlib is nice to have, but it's also really portable and lightweight. It would be easy for apps to bundle their own copies. Something like libcurl is a lot more awkward (what do you use for https? bundle your own SSL too!)
Even if you don't need a whole 2D library, it would be very useful to have a stable API for reading the system fonts, so apps don't have to ship their own.
Because Android's OS APIs are in Java for a reason. Yes, if you insist on using C++ with the OS APIs, you are pushing a square peg into a round hole, so enjoy JNI.
iOS and UWP also were not designed with the intent to use single binary on any CPU architecture, independently from any appstore that could recompile for you.
iOS also is not easy to use with any language, it assumes ability to link with a module written in ObjC. The pain of using UIKit with Python is comparable to using JNI.
> iOS and UWP also were not designed with the intent to use single binary on any CPU architecture, independently from any appstore that could recompile for you.
It seems you never did UWP development.
UWP was designed to be architecture independent since Windows Phone 8.
Application's bytecode is uploaded to the store and gets compiled to each supported architecture using the so called "cloud compiler".
iOS development has moved into that model with bitcode on iOS 9.
Also there have been several C and C++ compilers across the years that could generate single binary on any CPU architecture.
For example compilers for OS/400, TenDRA, PocketPC.
> iOS also is not easy to use with any language, it assumes ability to link with a module written in ObjC. The pain of using UIKit with Python is comparable to using JNI.
Python is not a platform language on iOS SDK, C and C++ are platform languages on iOS, Android and UWP SDKs.
A good lesson in terms of productivity, is to only use SDK supported languages when writing production code any OS.
> Application's bytecode is uploaded to the store and gets compiled to each supported architecture using the so called "cloud compiler".
That's why I wrote: independently from any appstore that could recompile for you.
Google was and is under different constrain, that you refuse to acknowledge: everything, that puts an application onto the device, must be able to run locally at the development system. Amazon store, F-Droid, or whatever the Chinese or Russians are using cannot depend on Google Play Store to recompile for them. The result has to run not only on any combination of ARM/Neon/VFP3, but also on Intels, MIPSes or some future architecture, that someone somewhere used or will use in the Android device.
Yes, they could use pNACL or LLVM bytecode, or whatever. They used a higher level one, dex. Their target was Java class hierarchy, they didn't want their developers to reinvent strings and collections all time.
> Python is not a platform language on iOS SDK, C and C++ are platform languages on iOS, Android and UWP SDKs.
C and C++ are not platform languages for Android. Literally, just read the first two paragraphs of Android NDK's "Getting Started" document. Using C/C++ is only for certain scenarios, that don't require the rich platform APIs. These are written in Java and if you are going to use non-JVM language, you are going to go through the same pain as if you were using Python with ObjC frameworks.
Android and SQLite have had some adventures: pre-Honeycomb (v3/API 11/2011), corrupted databases were automagically deleted by the OS API used to open them!
SQLite was also shaking out a (very) few bugs out during this time period; pretty sure this auto-delete implementation wiped any chance of semi-easily recovering my first phone's SMS db.
I would suggest another approach:
1) Import the source android.database.sqlite package while renaming it to my.database.sqlite
2) modify it trivially to load your own instance of sqlite....
Wouldn't symbol versioning[1] allow more than one SQLite in a Java process, even if it is just to munge the system shipped symbols into an Android specific subspace? (I do this, but with a large C++ application so I might be missing some finer details).
Although I mentioned using dlopen and dlsym to find the SQLite functions in the current process, our actual production solution was to use JNI to call the android.database.sqlite classes.
53 comments
[ 3.7 ms ] story [ 123 ms ] threadsqlite.so isn't part of public Android API surface, which means the OEMs will regularly replace it, break it, compile it with strange options and deal a lot of pain to you and your users.
So if you weren't shipping your own native SQLite in NDK/Xamarin you already had problems, you just maybe didn't notice them.
Android N is now actually actively preventing linking against system libraries which aren't part of public API surface, which is good - it'll result in less developers doing dumb things which break on minor updates.
You can never stop the Facebook developers doing dumb things on Android.
Google have tried for years, but Facebook always manages to find new ways to fuck up.
https://www.facebook.com/notes/facebook-engineering/under-th...
I've never understood why the Facebook client is so heavy. That isn't a long list of features! It seems like the requirements could be written on the back of an envelope.
Obviously part of the reason is that they move fast and have a lot of developers, so there's much more pressure to add code than to keep it trimmed down. But the client still doesn't do all that much. Surely, surely most of the logic is handled on the server.
Instead of using the platform-provided, battery efficient GCM (now FCM), they insist on using their own MQTT push solution.
Not only does this require a constantly running background service and a TCP socket which has to be reopened on every connectivity change (e.g. a switch from wi-fi to mobile data): They do it for every single one of their apps.
My phone has at least three of them (Instagram, Facebook, Messsenger), and I'm pretty sure that this is a major reason for battery life problems on Android.
Xamarin, Cordova and others are pretty much a cesspool of catastrophically poor libraries so I'm not sure why Google has to take the blame for that.
There are already implementations [1][2] of this, but because it is not provided by the Android team it is hardly supported by ORMs, libraries, etc.
[0] https://code.google.com/p/android/issues/detail?id=202658
[1] https://www.sqlite.org/android/doc/trunk/www/index.wiki
[2] https://github.com/requery/sqlite-android
The problem - from what I understand - is that you used to be able to use the systems sqlite.so to access databases owned by your app from the native environment. With Android N you are no longer able to do this, you need to ship your own sqlite binary.
This can lead to problems when you want to access the same sqlite files from both the Android platform APIs and native (e.g. use it from native for business logic, debug it with a tool like Stetho [0] that uses android.database.sqlite) because of a version mismatch.
Would Google offer (a copy of) the Android API with the ability to plug in a different sqlite-compatible binaries, it would very likely find broad adoption. Then the problem stated by the article could be solved by shipping the app with a sqlite build that is used by both the native code and plugged into the compat library.
[0] https://facebook.github.io/stetho/
Does SQLite accept patches?
Though, my guess shouldn't stop you looking for previous suggestions to implement this, and if there are none, submitting it yourself :)
You have no idea what version of sqlite the OS might be using, so you can't rely on consistent behavior. In many cases the vendor will have tinkered with it and messed it up. Comprehensively testing on all Android devices is basically impossible because there are so many models out there.
The reason they can tinker with it is that it's not public; if it were public it would (hopefully) be better-defined, locked down and stable, part of the Android compatibility test suite.
That feature request would let people supply their own sqlite.so, and also redirect the Java API to use it. Seems like a very promising approach.
Why is the NDK so underpowered? Why are they so reluctant (or unable) to publish some useful, stable native APIs? It's missing things like curl, sqlite, icu, OpenSSL. Most of these are built into the OS but not accessible via the NDK. (OpenSSL isn't a great library, I know, but it's patchable. And they could start now with something better.)
The only useful native third-party libraries you get are: zlib, OpenGL ES, OpenSL ES.
Apart from that, there are a bunch of wrappers for a small subset of Java APIs (AssetManager, Bitmap and so on). These are tedious to use, and seem to have been written by hand so in some cases they're buggy -- for example, incorrect string handling in AStorageManager: https://code.google.com/p/android/issues/detail?id=41983
I bet many Android devices already have at least 9 copies of sqlite, since so many apps ship their own copy.
NDK team doesn't look like it's expansive or prioritized at all, so I doubt they have resources or will to do that. TBH I'd rather they focus at improving the tooling and debugging, we can ship our own libraries with our own options compiled in. Heck even on iOS we have to ship a custom SQLite to get all the needed capability.
Google has one of the largest software engineering populations of any company in the world. It supports one of the largest developer ecosystems in the world. Games are the largest catagory in their ecosystem. 2 engineers supporting the NDK...
It wasn't there until 2.2, only got added due to community pressure and they always did the minimum amount of work.
When the migration to Android Studio was announced, we got left with no migration plan.
Only after JET Brains announced CLion, almost 2 years later, did they announce the migration to Studio.
Likewise they moved into Gradle without plan for C NDK support and it took three generations of build tools until they decided to settle on CMake.
As an earlier commenter mentioned, games are one of the biggest app categories and many games rely on the NDK. Pro audio could be another big category, but I think Apple has the market sewn up there because Android is still way behind.
(I'm not saying that there's anything inherently unreasonable about that, although I happen to disagree with the decision. But these things cost money and they decided they didn't want to spend the money.)
Most likely due to game development community.
Specially when compared with the tooling support C and C++ enjoy on iDevices and UWP.
So anyone that wants 2D graphics from NDK side either needs to call into Java, or bring their own library on top of OpenGL.
And the whole mismanaged way how the C++ IDE and build system migration was handled.
What you want is stable APIs for things that interact in fiddly ways with the rest of the system, so they're more easily managed at the OS level. Networking, shared databases. (Edit to add: localization)
zlib is nice to have, but it's also really portable and lightweight. It would be easy for apps to bundle their own copies. Something like libcurl is a lot more awkward (what do you use for https? bundle your own SSL too!)
In any case that was just an example.
Android has nothing like Objective-C++ or C++/CX for easy interoperability with the OS APIs.
Even if you don't need a whole 2D library, it would be very useful to have a stable API for reading the system fonts, so apps don't have to ship their own.
Also, given that Google forked Java, they could have added something like CNI insteand of forcing everyone the JNI pain.
iOS also is not easy to use with any language, it assumes ability to link with a module written in ObjC. The pain of using UIKit with Python is comparable to using JNI.
It seems you never did UWP development.
UWP was designed to be architecture independent since Windows Phone 8.
Application's bytecode is uploaded to the store and gets compiled to each supported architecture using the so called "cloud compiler".
iOS development has moved into that model with bitcode on iOS 9.
Also there have been several C and C++ compilers across the years that could generate single binary on any CPU architecture.
For example compilers for OS/400, TenDRA, PocketPC.
> iOS also is not easy to use with any language, it assumes ability to link with a module written in ObjC. The pain of using UIKit with Python is comparable to using JNI.
Python is not a platform language on iOS SDK, C and C++ are platform languages on iOS, Android and UWP SDKs.
A good lesson in terms of productivity, is to only use SDK supported languages when writing production code any OS.
That's why I wrote: independently from any appstore that could recompile for you.
Google was and is under different constrain, that you refuse to acknowledge: everything, that puts an application onto the device, must be able to run locally at the development system. Amazon store, F-Droid, or whatever the Chinese or Russians are using cannot depend on Google Play Store to recompile for them. The result has to run not only on any combination of ARM/Neon/VFP3, but also on Intels, MIPSes or some future architecture, that someone somewhere used or will use in the Android device.
Yes, they could use pNACL or LLVM bytecode, or whatever. They used a higher level one, dex. Their target was Java class hierarchy, they didn't want their developers to reinvent strings and collections all time.
> Python is not a platform language on iOS SDK, C and C++ are platform languages on iOS, Android and UWP SDKs.
C and C++ are not platform languages for Android. Literally, just read the first two paragraphs of Android NDK's "Getting Started" document. Using C/C++ is only for certain scenarios, that don't require the rich platform APIs. These are written in Java and if you are going to use non-JVM language, you are going to go through the same pain as if you were using Python with ObjC frameworks.
SQLite was also shaking out a (very) few bugs out during this time period; pretty sure this auto-delete implementation wiped any chance of semi-easily recovering my first phone's SMS db.
https://stackoverflow.com/questions/7764943/what-can-be-done...
"8. Bugs in SQLite" (2013)
https://www.sqlite.org/howtocorrupt.html
1. https://refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Core-gene...
https://code.google.com/p/android/issues/detail?id=213433
Although I mentioned using dlopen and dlsym to find the SQLite functions in the current process, our actual production solution was to use JNI to call the android.database.sqlite classes.