This is a disappointing response. I don't have a horse in this race (besides wanting to have access to good hardware for FOSS platforms), but I do have my ear on the ground in this community, so here are my thoughts on this.
> [removing SPI] was based on the fact that for years SPI was largely unused on PINE64 devices.
People have been arguing that a u-boot firmware needs to be installed on that flash chip for years, too. Essentially everyone in the dev community has been pushing them to do this for essentially the entire lifetime of their product line.
Would you rather find, download, and flash a special-purpose, pinebook-specific image onto a microSD card, pop it in, and boot that up to install a distribution, or would you rather it supports a standard UEFI booting interface over USB, netboot, whatever else, like every other laptop can?
> We created a space for development talks to be held (as we always had)
My understanding from speaking to friends and colleagues who work with Pine64 is that these spaces are pretty toxic. Manjaro has the loudest voice and other projects don't get the respect they deserve. And I'm prepared to believe it if this flippant response quoted in Martijn's article is accurate: "people who want [an SPI chip] can just solder one on."
This is not the behavior that I expect from a project which claims to honor its community.
I'm very disappointed in this response and in PINE64 generally. I expected an apology and a promise to improve, and this isn't it. Pine, I believed in you from the start.[0] I thought you were better than this. The past few years of your behavior has disillusioned me to your project and I can no longer offer you my support.
It's an excellent soldering iron, hardware wise. Lightweight, heats up extremely fast, holds up heat very well. Never tried a custom firmware, but had no need either.
Me too, I was hoping to get a Linux phone at a price that match the current lack of polish (for me the pinephone had that price) but Linux means community and diversity for other OS'es to me. So if they don't work with the community, then half the value of the phone disappears...
I will now consider buying second hand phone with postmarketOS support. It means I'll have to accept proprietary hardware but, as it will be second hand, it means I'll be more in line with my eco-environemental-whatever-half-baked principles.
Nobody cares, I just wanted to illustrate what are the reasons I would like to buy a Pine64 and how those reasons revolves around some basic principles and trade off's.
You could also get a second-hand PinePhone (just buy postmarketOS CE or later to avoid bad hardware bugs).
Regarding other postmarketOS supported devices, I quite like the Xiaomi Pocophone F1 and the OnePlus 6 - but be aware that bootloader unlock on the Xiaomi is a bit painful and charging on the OnePlus 6 is really slow (although that might have been fixed lately/be fixed soon).
So as is practically stereotypical for the Linux community, we're going to let a squabble over distros send us to buying proprietary hardware and more than likely... straight up giving Google more money, as seems to be the top hardware choice of vendors of so-called "secure Android forks".
> Would you rather find, download, and flash a special-purpose, pinebook-specific image onto a microSD card, pop it in, and boot that up to install a distribution, or would you rather it supports a standard UEFI booting interface over USB, netboot, whatever else, like every other laptop can?
It's possible to write Tow-Boot to the eMMC to do exactly this. SPI isn't needed, it's just more convenient. I happen to prefer SPI and Pine64 was willing to go with SPI too.
I think this issue is blown out of proportion.
Pine64 listened to feedback. The problem here was that timeline's didn't match, not bad intention.
If you want to buy a phone that Just Works, buy a Samsung Galaxy or a Google Pixel.
You can also buy a Pine64 product and if you do not like the hardware you are free to modify it to your heart's content. However, they don't owe it to you to perform the modifications you want, certainly not if there is a shortage of the part you want them to install.
As an outsider with no insights (but owning a PineBookPro and considering a PinePhonePro), it surely looks like Manjaro is the king of the mountain - and that Martijn's blogpost is likely close to The Truth.
Listening and doing your own things anyways is the time-honored way of ignoring people...
Even as an Arch user (that hence "gets" Manjaro) that is sad :(
Much of the device still doesn't even work. And we're talking simple stuff like the cameras. Calling is also super sketchy. You can text but voice calls effectively is a fail.
The IMU does appear to work correctly, for things like rotation. It is pretty fast.
You can also fry your device (hardware damage!) if you get the pinephone keyboard and plug in USBC into the phone port instead of the keyboard port. You read that correctly. Hope you don't mess up if you're tired.
And if your battery on the PPP goes too low, the utter shit firmware goes into a boot loop and can't charge. You have to take off the case and hit the recessed button to put it in DFU mode. That bypasses the firmware causing bootloops. And it's slllloooowwwww charging.
I completely and utterly regret such a shit device. And I say that about the very hardware, AND the reticence in actually putting out base builds of working software.
They claim that it supports eMMC booting natively. I bought 4 of each. And I bought a eMMC USB dongle to pre-liad the images.
And .... No booting off of eMMC. And utterly NO help from anyone. No guides, no nothing.
And that's hardware stuff that Pine attested would work. And it completely doesn't. I still have to boot using mSD at a whole class 10.
Their whole "plan" and company is a shit show. It's all gongkai crap. Using or buying will give you pain and suffering if anything goes even slightly wrong.
The cameras are far from "the simple stuff", it's one of the more complicated systems in mobile devices with multiple high speed components from different vendors that have to work together and not very much documentation for the hardware existing.
Admittedly, that's doubly concerning. Pine chooses the BoM; we don't.
And if they're choosing unsupported/unsupportable chips without doing the barest of legwork to get them functional (or hell, get docs public), then that's a huge problem.
Pine should be showing on their pages a hardware matrix showing what works and what doesn't. The Pinephone Pro doesn't qualify for the definition "phone".
Martijn, I want to really give you some credit for these responses: It's rare to see someone who will step in and defend bad takes against the party they are also taking issue with.
Rear sensor was chosen as a balance of kernel code available and picture quality, I had some influence there, front sensor was originally the ov5640 but that just got replaced due to availability.
> Pine should be showing on their pages a hardware matrix showing what works and what doesn't. The Pinephone Pro doesn't qualify for the definition "phone".
I'm actually on board, but do note that that's going to be a chart full of subnotes and caveats - things like "camera produces an slightly red-shifted but passable image on default software shipped from factory, red-shift is fixed with kernel 5.19-patch-foo but that breaks focus, Mobian ships that kernel with a userspace correction filter that fixes it in the latest nightlies[...]". You'll end up with a small table that's 90% little yellow "works with caveats" boxes and then several pages describing the caveats and various working and shades-of-working configurations. Oh, and it'll need to be timestamped and constantly updated (as in "major bugs fixed, and sometimes added, on a weekly basis")
In short, summarizing active development is never going to be simple.
If it’s no, but has a dozen asterisks, to the end user, that’s still a “no, this component does not work properly.” The footnotes can be 5 pages long but the chart is still straightforward
One day I’ll probably end up with a Linux phone, but I mean, not if I’m worried I’ll get surprised when it arrives
Nobody wrote the driver for H.264 HW encoder support, yet. When that happens, Pinephone will be capable of that, too.
That's not a $200 proposition, though. :)
And I guess there's no huge motivation, given the expected quality of the resulting image. Maybe someone will write it for some other similar AW SoC that uses the same HW block.
If it's beta software and prototype hardware then they should be in the 'give it to some people willing to help with bugs and documentation' phase, not 'charge people for a phone that can't make phone calls' phase.
I used to be all for early adoption, but too many people ran off with my money via Kickstarter and Steam Early Access.
> If it's beta software and prototype hardware then they should be in the 'give it to some people willing to help with bugs and documentation' phase, not 'charge people for a phone that can't make phone calls' phase.
It can make phone calls.
It took a separate $31,000 kickstarter and 4 years of a multi-vendor cooperation to create and upstream then non-existent accelerated stateless video decoding API for Linux. Pine64 would have never released anything if they waited on Linux API standardization process (have multiple users of the API in kernel, etc.). And even after 2 years of availability, the only major user of the API is gstreamer, and it's still bubbling through the stack so that you can just open Firefox and play an accelerated video.
Pushing through this process even nicher accelerated encoding feature, usable pretty much only for the cameras (which are quite crappy on the OG pinephone, so motivation is already low, there), is just not going to happen for giving someone a 7 year old outdated phone worth $150 only to some enthusiasts and $0 to pretty much everyone else, lol. That's just ridiculous.
Your $5 bargain bin webcam doesn't work without the computer and the software that runs on it though. The PinePhone isn't the problem, the problem is the lack of software. It is ridiculous to compare the two things.
I’d expect my PinePhone to come with software too. I mean, I guess it did, I think I just expected more from the open source community than from the nameless fly-by-night vendor.
Are you sure it does and isn't just the bog-standard generic driver for your OS? Because likewise, you provided the computer, OS and everything else for it to work with.
I never said they didn't include SPI, it took an incredibly amount of arguing over several weeks to get it there on the PPP, then they again didn't want to ship it on the PBP. In the end the PBP does have SPI but still didn't ship with something flashed to it.
While they did in the end listen and add SPI _after_ manjaro finally agreed. that doesn't mean they listened to the whole community. Ofcourse PINE64 and Manjaro don't have to deal with how the other distributions boot.
Well the issue with this happening in private telegram channels is that this will now turn into a "he said" "they said" situation, none of this happened publicly and neither side can provide sources. At the time the argument was definitely not that the chips weren't available.
Instead of delaying the PPP production by one day to get U-Boot sorted they decided to not ship mainline U-Boot at all. Which was only possible because Manjaro provided them with an image to ship with that.
In the end the only response there is is a blog post that never claims anything is wrong or should happen differently, after acknowledging all that I've done.
I only wrote why I left, that they themselves think they're manjaro focussed because the hardware that can't ship Linux doesn't ship manjaro doesn't change that this whole behind the scenes situation really burned me out, and I left.
You have every right to leave, and obviously if you're feeling burnt out, you made the right call for yourself. But it is frustrating that it seems like a distro preference issue has expanded into what will probably end up selling more proprietary phones at the end of the day.
I never understood why the PinePhone does not ship with Mobian or PostmarketOS instead of Manjaro. Manjaro quickly felt buggy and fragile on the first minutes I tried it, while the two others felt more robust. I'd rather see the PinePhone ship with one of these distributions.
Why does Pine64 favor Manjaro so much? What do they gain by doing so?
Personally, I don't get why the focus was on yet another mobile OS variant instead of creating a reasonably open hardware with an AOSP userland first. Users won't buy devices where they can't even be sure that basic functionality works.
And for what it's worth, I know Android isn't without its issues - but why not build something on top of at least its Linux kernel and HAL and save so much effort in getting a system running? Why do people always have to reinvent the wheel despite it being almost impossible to compete with Android and iPhone anyway, leading to burn-out and failures?
Because AOSP is controlled by Google and the incentives to develop an actually open platform are not there. Even the SDK to develop for it is guarded by a silly license.
That could have been a first step though, but at the risk of endangering the motivation to develop for mobile GNU/Linux and still be dependent of AOSP with nowhere to go when it really goes towards the wrong direction. More and more (essential) components are proprietary and shipped with the GApps. Even something as essential as the notification system relies on Google's servers and proprietary bits (this part has an open source re-implementation though, microG). At this point you need to recompile Android apps to avoid that… if you have access to the source code.
It's hard to fight against this direction, release after release, I think it is a lost battle.
I could live with AOSP today but I'd rather have a functional GNU/Linux environment with no bullshit in a few years.
> Even something as essential as the notification system relies on Google's servers and proprietary bits.
Agreed, but a notification broker that's usable for more than just "on-device" generation of notifications or wants to avoid every app polling constantly will always need some sort of centralized server infrastructure that costs a lot of money to develop, keep running and secure (something like "user A gets notifications intended for user B" or "attacker manages to dump notifications" must never happen).
> I could live with AOSP today but I'd rather have a functional GNU/Linux environment with no bullshit in a few years.
Same, but as long as people continue bikeshedding and insisting on complete 100% purity every such effort is doomed to fail.
I told myself I'd stay off this site because there's so much distracting crap on here but I need to correct this one.
You don't need a notification broker unless you want to run non-free garbage. Well written apps will just service the socket they get data from just fine. This is already the way things work on Linux, the only thing missing for real time notifications is to have the network interfaces wake the device up instead of relying on eg rtc-wake.
>centralized server infrastructure that costs a lot of money to develop
Maybe but Mozilla seems to get away with doing it for free and they have way less money.
And the higher level comments are right. AOSP is absolute fucking garbage. Before Android 5 it wasn't clear if this was something intentional from Google but it's clear now. They hate people having control over their devices and are in many ways even worse than Apple here despite publishing some code.
>100% purity
Purity will be the only thing that results in success. Already the Pinephone works well enough for people who are used to desktop Linux (sans the hardware issues with the modem.)
> the only thing missing for real time notifications is to have the network interfaces wake the device up instead of relying on eg rtc-wake
Waking the device up for each incoming packet would probably drain the battery. But yeah, I had descent battery life on Android without GApps with mails and two chat apps (Signal and Element) so this can't be impossible to achieve. The modem has to stay up for incoming calls, listening a few additional TCP connections is probably fine. Maybe it's less power efficient than a centralized solution but still very usable. We could also imagine a solution where the used service is configurable.
> Purity will be the only thing that results in success
If you don't enforce 100% open source, you are less incentivized to fix broken stuff and develop open source alternatives.
The "open source but not quite" niche is already fulfilled and it's LineageOS and its friends. That's already done. So nobody loses here.
Unless I am missing something you need a notification broker due to the very widespread usage of CGNAT on most mobile networks. There has to be a reachable outside server.
Nope, because your phone initiates the connection. The network knows to find your phone again when there are incoming packets from this connection. This is no different for the service broker by the way, as far as I know.
> You don't need a notification broker unless you want to run non-free garbage.
Face reality: most of what the utter majority of the people on this planet use is proprietary and ffs purity bikeshedding won't lead to any success.
> Well written apps will just service the socket they get data from just fine.
The utter majority of providers use CGNAT with extremely short keepalive times for cost saving, the worst I've seen was some crap MVNO dumping connections after ten seconds idling. That means your usual phone will have 20-50 apps that all have to send keepalive data all the time meaning the modem never goes asleep and drains your battery, vs one centralized notification broker that even shoddy MVNOs configure to be exempt from frequent connection drops.
> Already the Pinephone works well enough for people who are used to desktop Linux (sans the hardware issues with the modem.)
So... like what, 3% of users given Desktop Linux market share? LOL the only ones used to Linux on desktop are either forced to do so by their employer or are utter masochists willing to overlook catastrophic flaws for the sake of purity.
If the connection dies, you just re-establish it when the device is scheduled to wake up (as directed by timers which can easily be auto-coalesced on Linux, so battery drain is a total non-issue regardless of how many apps are active). It's not a bigger issue than checking your email or refreshing your Slack/Discord/whatever.
There's no such thing as a single AOSP kernel. If you really want to save effort in getting hardware supported, the reasonable choice is to target the kernel mainline.
It's terrible in that it's huge piles of inefficient software that can barely run on hardware that's actually pretty powerful.
It's doomed in that as time passes, different FLOSS bits of it get "deprecated" and replaced with something proprietary. Or a new API appears, but Google phones actually ship a slightly better proprietary one.
There's a big conflict of interest when an ad company wanting to spy on people and lock them in is the steward of an open source OS.
EDIT: Also, don't forget that most devices out there use a custom kernel. There's no single "AOSP".
If Pine had targeted Android AOSP, I would have had zero interest in their product. I suspect I'm not the only one. Instead, I bought their pinephone because it could run Debian.
I went from Maemo (Debian based) to AOSP Android when my n900 no longer worked. I've never experienced such a user-hostile platform as Android-- you even have to expend effort just to gain and retain root on a device you nominally own. And, I don't trust anything in the Android ecosystem to not be spying and tracking.
The thing is, that bikeshedding is the approach that everyone has taken so far - and failed catastrophically. Literally every single project either has failed or is in the process of failing - and I count "shipping a product that is barely able to accept calls to a bunch of developers" just that.
So I'd argue it makes more sense to first either get a decent alternative OS running on a proprietary piece of hardware (which is almost impossibly hard) or get an open piece of hardware running the only thing that comes close to an open-source OS first (so users will actually buy it because they can actually use it as their daily driver).
It doesn't make any sense for any project without billions of dollars backing it to attempt the 100% purist approach from the beginning. Even Mozilla wasn't able to pull it off, so how and why should anyone else succeed?
Mozilla did not try a purist approach. Their Boot2Gecko was built on AOSP downstream kernels and proprietary drivers. (Then again, there was no such thing as a truly 'purist' device back then, even in non-AOSP land. So they just did what could work at the time.) If you want to explore that kind of approach, there's things like UBPorts and Droidian. They might even make sense, if they end up being able to function as AOSP-compatible Generic System Images (GSI's) and run on mostly any Project Treble-capable device. Make no mistake though, that's a lot harder than doing everything properly in upstream and restricting one's attention to hardware that's compatible with that approach. If only because you're not forced to keep up with churn in the AOSP-native interfaces.
They ship devices with a completely broken OS[1], I could never recommend their products to anyone, except maybe someone who is a technical type and comfortable going through the process of installing another distro + happy to spend time doing that.
[1] At least as of a couple of months ago. I'm talking basic functionality being broken, like for example trying to add an icon to the home screen and the phone crashes. I encountered multiple bugs in just the first few minutes of usage.
To be fair, I had a very similar experience on my very first minutes with an iPhone on one of the .0 iOS releases. So many glitches and broken things, even on the onboarding process, many of them reliably reproducible. That said, when I grabbed that iPhone the next time and updated it a few minor releases further, most of them were fixed.
the difference is that there were perfectly good distros for the PinePhone, but then it was decided to start shipping it with a half-baked one. It's as if Apple went back from those later iOS releases to the earlier one.
The original "Why I left Pine64" article adds more info about what went on.
As a veteran embedded engineer, I was shock to see that another variant of boot loader is being recommended.
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is not that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem then so be it but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
Tow-Boot is actually a really exciting project that wants to make it so that distributions don't have to manage device-specific u-boot packages for ARM devices, and can instead provide generic images like they do on x86.
This right here. As much as I would love to see more faster and powerful ARM processors come to laptops and desktops in the future, I dread the current ARM bootloader situation coming along with it.
As a veteran embedded engineer, I was shock to see that another variant of boot loader is being recommended.
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
I am a big fan, but I also find this situation and the response to be a bit disappointing.
I highly respect Martijn and the effort he has poured into pmOS and Pine64 and surrounding projects. Without wanting to make other important people feel left out, I consider him an absolute legend and the fact that he has felt compelled to write this blog post (and, unfortunately, to leave) is massively concerning.
I think it's very possible that there has been no malice here and it's all just an accident of one thing leading to another, but something has gone awry here and I think it would be nice to hear what Pine intend to do to put it right, or to know that there is such a plan at all.
> As for the reason why Pinebook Pro doesn’t ship with a bootloader on SPI – it isn’t caused by favoritism for a particular group of developers or disregard for good ideas, but rather something more trivial. Namerly, a functional Tow-Boot build for the Pinebook Pro wasn’t available at the time of manufacture and shipping.
One particular point I want to make: I'm willing to believe Pine about the above, but it's too late to hear about this after crucial people have been miffed. This seems to indicate that communication with the various projects has not been good. This seems like a crucial thing to improve on to prevent this recurring.
I don't have enough knowledge of the situation and am incapable of making any judgements as to what is really going on, but I sincerely hope that Pine will consider this situation carefully and whether something can be done to put things right again.
I really wish to see Pine and their hardware do well and flourish. That was never going to be easy, but it will be even less easy with the loss of people like Martijn.
There has been zero communication towards Tow-Boot developers during this process, nothing about the issues on the new batch. The moment Tow-Boot learned about this is when people received the new Pinebook Pro and it came with a different bootloader.
I found logs. For context, tl_lim is the founder of Pine64.
2022-06-01, the Pine64 dev IRC chat:
> [T] <tl_lim> I pause on move forward due to no one commit to maintain. If MayueIC interest to maintain, lets move forward and include Tow-Boot into the PBP factory build. @MayeulIC, you can send an email to info@pine64.org, we will ship a free Pinebook Pro to you so that you can maintain the Tow-Boot.
> [T] <tl_lim> On the integration, just follow the recent PinePhone Pro way that flash OS build to eMMC then flash Tow-Boot to SPI Flash.
> [T] <tl_lim> Frankly speaking, I am happy on developer commit maintaining Tow-Boot on Pinebook Pro. Thanks and a big shout out to @MayeulIC.
2022-06-02, in the Tow-Boot chat, which was discussing tl_lim's above comment:
> spikerguy: Tllim asked on. 25th April about pbp tow boot support i replied it just work fine as it should. But just because i am the one to work on factory images plus i am the manjaro arm pkg maintainer maybe he didn't wanted to hear from me instead the community that was assuring him about tow boot and none replied
> The current batch factory image is already submitted. So hopefully we can keep things ready for the next batch
I don't see all comms, but overall I see Tow-Boot on SPI at factory-time as nice-to-have but not essential (users can easily do this, distros don't need to require it, and the next batch can improve it if needded), Pine64 has showed willingness to use Tow-Boot. Tow-Boot has 1 developer, and I'm not sure if they were willing to commit to support this. I understand if they weren't.
This issue is blown out of proportion.
Maybe a good step would be for Pine64 to fund Tow-Boot's support of Pinebook Pro.
This is just after I left. And after me saying (in not the public channel) that samuel is maintaining Tow-Boot, other developers are helping (like me) but that didn't matter.
I'm a bit confused by the whole bootloader situation. Locking down a bootloader on an open device seems quite bad, but what's the issue here exactly? The user can install any bootloader they want, can't they? I know the ARM bootloader system isn't anything like the PC bootloader system, but even if the bootloader is loaded from eMMC, in what way is the user impeded?
I would guess all the user should need to do to use their own bootloader is to pacman -R manjaro-arm-uboot and pacman -S pmos-uboot to hand over control of the bootloader image if you buy a device flashed with Manjaro, or overwrite the eMMC bootloader with one that manually tries to boot from the SD card before resuming normal boot?
It seems that the frustrations about Manjaro go further than just the bootloader but I don't really understand why a SPI flash chip would even be a necessity in the first place.
> even if the bootloader is loaded from eMMC, in what way is the user impeded?
The "bootloader" isn't just a bootloader, it's effectively a critical firmware component. If the "bootloader" is missing or garbled, the board is bricked and can only be recovered by removing the eMMC, which would be very difficult for most users. Such a thing should not be stored on a data partition and managed like any other distribution package. If they aren't going to ship a separate SPI chip, the eMMC should at least be repartitioned to include a separate "firmware" partition and the bootloader should be shipped there. Distributions should then boot via a chainload mechanism, just like on x86 hardware. (This mechanism can be UEFI based but need not be.)
I think I'm on Team Pine here. I don't really understand the problem Martijn has. It seems to boil down to that they agreed to everything he wanted, just not fast enough?
Sure, but that's not a deal breaker when you can just change it? I too don't particularly like Manjaro but here's a situation where he's doing damage to basically the only brand that even remotely supports his interests (and that would even listen to him personally) at a low cost. Or maybe there's other better phone options to run Phosh that I don't know about?
If they proved anything it was that they can’t build a phone. The world got a useless brick because politics. Whenever I read about the pinephone I somehow narrate it in the voice of Jordan Peterson
I think the problem is that as an external observer, it's almost impossible to distinguish "we listen but don't/can't always act on it" and "we don't listen", because they produce the same actions and are only possible to tell apart by closely following the exact communications (which isn't always possible since some happen in private, and is unreliable anyways if some is trying to appear engaged while not actually listening).
This seems easy enough, right? If the SPI chip is in shipped units, and tow-boot is available for it now, then it should be simple enough to ship out of the box starting with the next batch of hardware, and it should be simple enough to post official instructions for installing it from Manjaro. While that wouldn't change the past, it would fix things going forward and let Pine back their words with actions.
> I may add that the decision was made among ourselves, without the input of any third parties or partners
Maybe this, right here, is actually a major part of the bigger problem and not exonerating like it seems to be intended
> The reasoning behind not including SPI on the PCB wasn’t motivated by any single project or outside source – it was based on the fact that for years SPI was largely unused on PINE64 devices.
So if they only talked amongst themselves, how could they have any idea of if the SPI was already being used or not?
> And yet, despite this, we agreed to include it on the PinePhone Pro because developers from multiple projects – postmarketOS being one of them – were adamant that it was an absolute necessity.
So yeah, sounds like if they had engaged earlier, a lot of unnecessary frustration and grief could have been completely avoided.
Unfortunately this post does not even hint at the more significant point in Martijns post, the one about the relationships between PINE64 and the community distributions.
> The example given in the blog post supposes that the community team and Pine Store employees were firmly intent on removing SPI on the PinePhone Pro and coerced not to ship Tow-Boot.
I don't see where in Martijn's blog post that "coercion" was "supposed" at all. What he actually wrote:
> Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.
> After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.
He doesn't even claim that Manjaro objected at all to the idea of adding an SPI chip and preloading Tow-Boot on it; only that they have a possible vested interest in resisting it, and that PINE64 therefore also does for as long as it considers Manjaro the preferred OS for PinePhones.
I get the feeling that Martijn's situation was very much death by a thousand cuts and that didn't quite get captured well by the focus on the SPI issue - which Pine64 seems to have defended reasonably well here.
108 comments
[ 1.1 ms ] story [ 174 ms ] thread> [removing SPI] was based on the fact that for years SPI was largely unused on PINE64 devices.
People have been arguing that a u-boot firmware needs to be installed on that flash chip for years, too. Essentially everyone in the dev community has been pushing them to do this for essentially the entire lifetime of their product line.
Would you rather find, download, and flash a special-purpose, pinebook-specific image onto a microSD card, pop it in, and boot that up to install a distribution, or would you rather it supports a standard UEFI booting interface over USB, netboot, whatever else, like every other laptop can?
> We created a space for development talks to be held (as we always had)
My understanding from speaking to friends and colleagues who work with Pine64 is that these spaces are pretty toxic. Manjaro has the loudest voice and other projects don't get the respect they deserve. And I'm prepared to believe it if this flippant response quoted in Martijn's article is accurate: "people who want [an SPI chip] can just solder one on."
This is not the behavior that I expect from a project which claims to honor its community.
I'm very disappointed in this response and in PINE64 generally. I expected an apology and a promise to improve, and this isn't it. Pine, I believed in you from the start.[0] I thought you were better than this. The past few years of your behavior has disillusioned me to your project and I can no longer offer you my support.
[0]: https://drewdevault.com/2019/12/18/PinePhone-review.html
Although the firmware mine came with had an issue and I did have to update it, I now prefer it to my Hakko.
I will now consider buying second hand phone with postmarketOS support. It means I'll have to accept proprietary hardware but, as it will be second hand, it means I'll be more in line with my eco-environemental-whatever-half-baked principles.
Nobody cares, I just wanted to illustrate what are the reasons I would like to buy a Pine64 and how those reasons revolves around some basic principles and trade off's.
Regarding other postmarketOS supported devices, I quite like the Xiaomi Pocophone F1 and the OnePlus 6 - but be aware that bootloader unlock on the Xiaomi is a bit painful and charging on the OnePlus 6 is really slow (although that might have been fixed lately/be fixed soon).
It's possible to write Tow-Boot to the eMMC to do exactly this. SPI isn't needed, it's just more convenient. I happen to prefer SPI and Pine64 was willing to go with SPI too.
I think this issue is blown out of proportion.
Pine64 listened to feedback. The problem here was that timeline's didn't match, not bad intention.
That's how many of the SBCs I own (from differen vendor) are distributed. There's a space to solder the chip. (and you can pick the size you want)
You can also buy a Pine64 product and if you do not like the hardware you are free to modify it to your heart's content. However, they don't owe it to you to perform the modifications you want, certainly not if there is a shortage of the part you want them to install.
Listening and doing your own things anyways is the time-honored way of ignoring people...
Even as an Arch user (that hence "gets" Manjaro) that is sad :(
Much of the device still doesn't even work. And we're talking simple stuff like the cameras. Calling is also super sketchy. You can text but voice calls effectively is a fail.
The IMU does appear to work correctly, for things like rotation. It is pretty fast.
You can also fry your device (hardware damage!) if you get the pinephone keyboard and plug in USBC into the phone port instead of the keyboard port. You read that correctly. Hope you don't mess up if you're tired.
And if your battery on the PPP goes too low, the utter shit firmware goes into a boot loop and can't charge. You have to take off the case and hit the recessed button to put it in DFU mode. That bypasses the firmware causing bootloops. And it's slllloooowwwww charging.
I completely and utterly regret such a shit device. And I say that about the very hardware, AND the reticence in actually putting out base builds of working software.
They claim that it supports eMMC booting natively. I bought 4 of each. And I bought a eMMC USB dongle to pre-liad the images.
And .... No booting off of eMMC. And utterly NO help from anyone. No guides, no nothing.
And that's hardware stuff that Pine attested would work. And it completely doesn't. I still have to boot using mSD at a whole class 10.
Their whole "plan" and company is a shit show. It's all gongkai crap. Using or buying will give you pain and suffering if anything goes even slightly wrong.
And if they're choosing unsupported/unsupportable chips without doing the barest of legwork to get them functional (or hell, get docs public), then that's a huge problem.
Pine should be showing on their pages a hardware matrix showing what works and what doesn't. The Pinephone Pro doesn't qualify for the definition "phone".
If they were serious, they'd put this page up front, with a list of all the stuff that they imply works, but doesn't: https://wiki.pine64.org/wiki/PinePhone_Pro_Software_State
inb4 What works: You tell me!
Actually camera sensor was changed on request from Martijn. (or so I was told, when I was asked to add support for the cameras in the kernel)
"The PinePhone Pro Explorer Edition is aimed at Linux developers with an extensive knowledge of embedded systems and/or experience with mobile Linux"
To me thats pretty a pretty clear implication that many things may work poorly or not at all.
And, they call it a Pine *PHONE* Pro.
It can't even do the "Phone" thing.
And read up about the hardware critical failures that can destroy stuff when you plug in the wrong usb-c connection.
Yes, I expect a minimum functionality. You know, like NOT FRYING SHIT. They couldn't even manage to do that.
There's nothing wrong about Pinehnone Pro's Type-C port supporting circuitry. It's pretty much a reference design this time.
I'm actually on board, but do note that that's going to be a chart full of subnotes and caveats - things like "camera produces an slightly red-shifted but passable image on default software shipped from factory, red-shift is fixed with kernel 5.19-patch-foo but that breaks focus, Mobian ships that kernel with a userspace correction filter that fixes it in the latest nightlies[...]". You'll end up with a small table that's 90% little yellow "works with caveats" boxes and then several pages describing the caveats and various working and shades-of-working configurations. Oh, and it'll need to be timestamped and constantly updated (as in "major bugs fixed, and sometimes added, on a weekly basis")
In short, summarizing active development is never going to be simple.
One day I’ll probably end up with a Linux phone, but I mean, not if I’m worried I’ll get surprised when it arrives
My $200 pinephone can, on a good day, do 0.5hz. I dunno, but I expected more.
That's not a $200 proposition, though. :)
And I guess there's no huge motivation, given the expected quality of the resulting image. Maybe someone will write it for some other similar AW SoC that uses the same HW block.
Such is the power of FOSS.
I used to be all for early adoption, but too many people ran off with my money via Kickstarter and Steam Early Access.
edit- add phase x2 =p
It can make phone calls.
It took a separate $31,000 kickstarter and 4 years of a multi-vendor cooperation to create and upstream then non-existent accelerated stateless video decoding API for Linux. Pine64 would have never released anything if they waited on Linux API standardization process (have multiple users of the API in kernel, etc.). And even after 2 years of availability, the only major user of the API is gstreamer, and it's still bubbling through the stack so that you can just open Firefox and play an accelerated video.
Pushing through this process even nicher accelerated encoding feature, usable pretty much only for the cameras (which are quite crappy on the OG pinephone, so motivation is already low, there), is just not going to happen for giving someone a 7 year old outdated phone worth $150 only to some enthusiasts and $0 to pretty much everyone else, lol. That's just ridiculous.
I’d expect my PinePhone to come with software too. I mean, I guess it did, I think I just expected more from the open source community than from the nameless fly-by-night vendor.
Software takes time. But their plan is "cobble shit together in an untested and unmaintained state and YOLO".
* Didn't have cameras working
* Wifi didn't come back after suspend
* Had numerous modem reliability issues
* Bluetooth, sensors & audio were available but unused by software
* Had some hardware bugs
So this seems about on-par. Though on the userspace side things have of course progressed hugely for all devices.
While they did in the end listen and add SPI _after_ manjaro finally agreed. that doesn't mean they listened to the whole community. Ofcourse PINE64 and Manjaro don't have to deal with how the other distributions boot.
Instead of delaying the PPP production by one day to get U-Boot sorted they decided to not ship mainline U-Boot at all. Which was only possible because Manjaro provided them with an image to ship with that.
In the end the only response there is is a blog post that never claims anything is wrong or should happen differently, after acknowledging all that I've done.
I only wrote why I left, that they themselves think they're manjaro focussed because the hardware that can't ship Linux doesn't ship manjaro doesn't change that this whole behind the scenes situation really burned me out, and I left.
Why does Pine64 favor Manjaro so much? What do they gain by doing so?
And for what it's worth, I know Android isn't without its issues - but why not build something on top of at least its Linux kernel and HAL and save so much effort in getting a system running? Why do people always have to reinvent the wheel despite it being almost impossible to compete with Android and iPhone anyway, leading to burn-out and failures?
That could have been a first step though, but at the risk of endangering the motivation to develop for mobile GNU/Linux and still be dependent of AOSP with nowhere to go when it really goes towards the wrong direction. More and more (essential) components are proprietary and shipped with the GApps. Even something as essential as the notification system relies on Google's servers and proprietary bits (this part has an open source re-implementation though, microG). At this point you need to recompile Android apps to avoid that… if you have access to the source code.
It's hard to fight against this direction, release after release, I think it is a lost battle.
I could live with AOSP today but I'd rather have a functional GNU/Linux environment with no bullshit in a few years.
Agreed, but a notification broker that's usable for more than just "on-device" generation of notifications or wants to avoid every app polling constantly will always need some sort of centralized server infrastructure that costs a lot of money to develop, keep running and secure (something like "user A gets notifications intended for user B" or "attacker manages to dump notifications" must never happen).
> I could live with AOSP today but I'd rather have a functional GNU/Linux environment with no bullshit in a few years.
Same, but as long as people continue bikeshedding and insisting on complete 100% purity every such effort is doomed to fail.
You don't need a notification broker unless you want to run non-free garbage. Well written apps will just service the socket they get data from just fine. This is already the way things work on Linux, the only thing missing for real time notifications is to have the network interfaces wake the device up instead of relying on eg rtc-wake.
>centralized server infrastructure that costs a lot of money to develop
Maybe but Mozilla seems to get away with doing it for free and they have way less money.
And the higher level comments are right. AOSP is absolute fucking garbage. Before Android 5 it wasn't clear if this was something intentional from Google but it's clear now. They hate people having control over their devices and are in many ways even worse than Apple here despite publishing some code.
>100% purity
Purity will be the only thing that results in success. Already the Pinephone works well enough for people who are used to desktop Linux (sans the hardware issues with the modem.)
> the only thing missing for real time notifications is to have the network interfaces wake the device up instead of relying on eg rtc-wake
Waking the device up for each incoming packet would probably drain the battery. But yeah, I had descent battery life on Android without GApps with mails and two chat apps (Signal and Element) so this can't be impossible to achieve. The modem has to stay up for incoming calls, listening a few additional TCP connections is probably fine. Maybe it's less power efficient than a centralized solution but still very usable. We could also imagine a solution where the used service is configurable.
> Purity will be the only thing that results in success
If you don't enforce 100% open source, you are less incentivized to fix broken stuff and develop open source alternatives.
The "open source but not quite" niche is already fulfilled and it's LineageOS and its friends. That's already done. So nobody loses here.
Face reality: most of what the utter majority of the people on this planet use is proprietary and ffs purity bikeshedding won't lead to any success.
> Well written apps will just service the socket they get data from just fine.
The utter majority of providers use CGNAT with extremely short keepalive times for cost saving, the worst I've seen was some crap MVNO dumping connections after ten seconds idling. That means your usual phone will have 20-50 apps that all have to send keepalive data all the time meaning the modem never goes asleep and drains your battery, vs one centralized notification broker that even shoddy MVNOs configure to be exempt from frequent connection drops.
> Already the Pinephone works well enough for people who are used to desktop Linux (sans the hardware issues with the modem.)
So... like what, 3% of users given Desktop Linux market share? LOL the only ones used to Linux on desktop are either forced to do so by their employer or are utter masochists willing to overlook catastrophic flaws for the sake of purity.
It's terrible in that it's huge piles of inefficient software that can barely run on hardware that's actually pretty powerful.
It's doomed in that as time passes, different FLOSS bits of it get "deprecated" and replaced with something proprietary. Or a new API appears, but Google phones actually ship a slightly better proprietary one.
There's a big conflict of interest when an ad company wanting to spy on people and lock them in is the steward of an open source OS.
EDIT: Also, don't forget that most devices out there use a custom kernel. There's no single "AOSP".
It may be less battery efficient than Google's builds but skipping all the spyware really saves a ton of battery.
I went from Maemo (Debian based) to AOSP Android when my n900 no longer worked. I've never experienced such a user-hostile platform as Android-- you even have to expend effort just to gain and retain root on a device you nominally own. And, I don't trust anything in the Android ecosystem to not be spying and tracking.
So I'd argue it makes more sense to first either get a decent alternative OS running on a proprietary piece of hardware (which is almost impossibly hard) or get an open piece of hardware running the only thing that comes close to an open-source OS first (so users will actually buy it because they can actually use it as their daily driver).
It doesn't make any sense for any project without billions of dollars backing it to attempt the 100% purist approach from the beginning. Even Mozilla wasn't able to pull it off, so how and why should anyone else succeed?
Mozilla did not try a purist approach. Their Boot2Gecko was built on AOSP downstream kernels and proprietary drivers. (Then again, there was no such thing as a truly 'purist' device back then, even in non-AOSP land. So they just did what could work at the time.) If you want to explore that kind of approach, there's things like UBPorts and Droidian. They might even make sense, if they end up being able to function as AOSP-compatible Generic System Images (GSI's) and run on mostly any Project Treble-capable device. Make no mistake though, that's a lot harder than doing everything properly in upstream and restricting one's attention to hardware that's compatible with that approach. If only because you're not forced to keep up with churn in the AOSP-native interfaces.
I'm glad you are a part of the community, championing for ease of installation across all distros.
Which other manufacturers provide any feedback channel, or engage with the community during manufacturing?
[1] At least as of a couple of months ago. I'm talking basic functionality being broken, like for example trying to add an icon to the home screen and the phone crashes. I encountered multiple bugs in just the first few minutes of usage.
The original "Why I left Pine64" article adds more info about what went on.
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is not that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem then so be it but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
If the device has the "ARM System Ready" certification it will run most linux distros just by using an arm build, similar to amd64.
Tianocore/EDK2 also supports many devices and adds a BIOS/UEFI that is similar to the BIOS you'd see on your typical amd64 mainboard.
I’m quite sure that the bootloop is more than fixable but have my doubts that SPI is that option.
Tow-wut? Sounds something like “extoll-embrace-engulf” and Balkanization to me. Not to stand in the way, but if that fixes the low-charge/bootloop problem but it doesn’t.
Still have the PinePhone Classic, still works, on my anti-static pad, but it may not serve us much longer.
I highly respect Martijn and the effort he has poured into pmOS and Pine64 and surrounding projects. Without wanting to make other important people feel left out, I consider him an absolute legend and the fact that he has felt compelled to write this blog post (and, unfortunately, to leave) is massively concerning.
I think it's very possible that there has been no malice here and it's all just an accident of one thing leading to another, but something has gone awry here and I think it would be nice to hear what Pine intend to do to put it right, or to know that there is such a plan at all.
> As for the reason why Pinebook Pro doesn’t ship with a bootloader on SPI – it isn’t caused by favoritism for a particular group of developers or disregard for good ideas, but rather something more trivial. Namerly, a functional Tow-Boot build for the Pinebook Pro wasn’t available at the time of manufacture and shipping.
One particular point I want to make: I'm willing to believe Pine about the above, but it's too late to hear about this after crucial people have been miffed. This seems to indicate that communication with the various projects has not been good. This seems like a crucial thing to improve on to prevent this recurring.
I don't have enough knowledge of the situation and am incapable of making any judgements as to what is really going on, but I sincerely hope that Pine will consider this situation carefully and whether something can be done to put things right again.
I really wish to see Pine and their hardware do well and flourish. That was never going to be easy, but it will be even less easy with the loss of people like Martijn.
This is a massive communication issue.
Pine64 was looking for somebody to commit to maintain PineBook Pro support in Tow-Boot (a relatively new project)
I don't know if the point-of-contact was made aware that PineBook Pro was not going to be shipped with Tow-Boot.
2022-06-01, the Pine64 dev IRC chat:
> [T] <tl_lim> I pause on move forward due to no one commit to maintain. If MayueIC interest to maintain, lets move forward and include Tow-Boot into the PBP factory build. @MayeulIC, you can send an email to info@pine64.org, we will ship a free Pinebook Pro to you so that you can maintain the Tow-Boot.
> [T] <tl_lim> On the integration, just follow the recent PinePhone Pro way that flash OS build to eMMC then flash Tow-Boot to SPI Flash.
> [T] <tl_lim> Frankly speaking, I am happy on developer commit maintaining Tow-Boot on Pinebook Pro. Thanks and a big shout out to @MayeulIC.
2022-06-02, in the Tow-Boot chat, which was discussing tl_lim's above comment:
> spikerguy: Tllim asked on. 25th April about pbp tow boot support i replied it just work fine as it should. But just because i am the one to work on factory images plus i am the manjaro arm pkg maintainer maybe he didn't wanted to hear from me instead the community that was assuring him about tow boot and none replied
> The current batch factory image is already submitted. So hopefully we can keep things ready for the next batch
I don't see all comms, but overall I see Tow-Boot on SPI at factory-time as nice-to-have but not essential (users can easily do this, distros don't need to require it, and the next batch can improve it if needded), Pine64 has showed willingness to use Tow-Boot. Tow-Boot has 1 developer, and I'm not sure if they were willing to commit to support this. I understand if they weren't.
This issue is blown out of proportion.
Maybe a good step would be for Pine64 to fund Tow-Boot's support of Pinebook Pro.
I would guess all the user should need to do to use their own bootloader is to pacman -R manjaro-arm-uboot and pacman -S pmos-uboot to hand over control of the bootloader image if you buy a device flashed with Manjaro, or overwrite the eMMC bootloader with one that manually tries to boot from the SD card before resuming normal boot?
It seems that the frustrations about Manjaro go further than just the bootloader but I don't really understand why a SPI flash chip would even be a necessity in the first place.
The "bootloader" isn't just a bootloader, it's effectively a critical firmware component. If the "bootloader" is missing or garbled, the board is bricked and can only be recovered by removing the eMMC, which would be very difficult for most users. Such a thing should not be stored on a data partition and managed like any other distribution package. If they aren't going to ship a separate SPI chip, the eMMC should at least be repartitioned to include a separate "firmware" partition and the bootloader should be shipped there. Distributions should then boot via a chainload mechanism, just like on x86 hardware. (This mechanism can be UEFI based but need not be.)
No, it would require an SD-card to recover.
A lot hinges on a fact like that. It's the difference between not-convenient and not-possible.
Will the device actually know how to boot from sdcard if the part they are talking about is broken?
Seems like a bad idea to me.
Pine64 has listened to feedback, and made attempts to get Tow-Boot flashed on the PineBook Pro. It just didn't fit Pine64's manufacturing timeline.
Does any other manufacturer do better in this regard?
I realize that’s healthy, but the way they’re going about it makes me feel like they’re trying to weasel out of it or something.
I’m left with a mild feeling of disappointment and I’m not sure why.
Maybe this, right here, is actually a major part of the bigger problem and not exonerating like it seems to be intended
> The reasoning behind not including SPI on the PCB wasn’t motivated by any single project or outside source – it was based on the fact that for years SPI was largely unused on PINE64 devices.
So if they only talked amongst themselves, how could they have any idea of if the SPI was already being used or not?
> And yet, despite this, we agreed to include it on the PinePhone Pro because developers from multiple projects – postmarketOS being one of them – were adamant that it was an absolute necessity.
So yeah, sounds like if they had engaged earlier, a lot of unnecessary frustration and grief could have been completely avoided.
Unfortunately this post does not even hint at the more significant point in Martijns post, the one about the relationships between PINE64 and the community distributions.
I don't see where in Martijn's blog post that "coercion" was "supposed" at all. What he actually wrote:
> Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.
> After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.
He doesn't even claim that Manjaro objected at all to the idea of adding an SPI chip and preloading Tow-Boot on it; only that they have a possible vested interest in resisting it, and that PINE64 therefore also does for as long as it considers Manjaro the preferred OS for PinePhones.