This blog post has been quite the effort—coming into it I thought I'd just learn some new things, document a simple process, and be done.
But along the way I probably spent 40 hours (and counting) trying to figure out the ins-and-outs of 4G LTE modems and Linux networking when it comes to these oddball devices.
I hope some parts of this post help anyone else new to LTE networking in Linux, and maybe somebody can help clear up some of my open questions, like:
Why doesn't Quectel have any firmware downloads on their site? (And is there any way to get them besides downloading random files posted to forums?)
Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)?
Why does it seem there's no real documentation for LTE modems on Linux? What little _good_ information I found was in random presentation slides scattered around the Internet.
Other than that it's either extremely technical documentation pages that expect a very high level of familiarity already, or some very cookie-cutter blog posts that give a walkthrough to get Internet going, but don't really explain anything.
As someone who as also spent way more hours on LTE/5G modems than I want to, this post resonates with me. I also learned a few things, namely about the ECM. I will have to look into that. Thanks!
As for the firmware. It seems to just be how these companies do things, unfortunately. At work, we use modems from both Quectel and Simcom and the experience described in OP is very similar. For Simcom we have gotten access to a "secret-but-public" Onedrive share with documentation, drivers and firmware where they put up stuff sporadically. Sometimes the firmware updates even works!
I have a theory that Qualcomm who makes the chipsets used by Quectel and Simcom (at least for the 5G modems) imposes some kind of heavy NDA which prevents free distribution of firmware and documentation. Everything seems very "hush, hush" at least.
Have you experimented with modemmanager/networkmanager to handle the configuration of the modem? For me that has worked quite reliably across a pretty large fleet of devices with a Quectel EG25 (qmi over usb).
Regarding FW updates, have you tried opening a support ticket with Quectel? For me they have been extremely responsive and helpful, providing extra documentation on special AT commands and stuff.
I have not—that was what I was trying (mmcli) early on in my exploration, but I kept hitting speedbumps, so switched gears to configuring the modem over TTY and using QMI utils directly.
I would like to use modemmanager though, it _seems_ like it's the nicer path.
With modemmanager you get a lot of nice stuff for "free", like automatic reconnect, DHCP, good logging, and an APN database (useful if you don't provide your own SIMs). It also allows you to easily set up correct priorities between WiFi and LTE, both for routing and DNS. Especially DNS is something I didn't think about at first, but the DNS server provided by your LTE connection is not always reachable over your WiFi/ethernet connection.
You can run modemmanager in debug mode, which allows you to send AT commands over dbus (or mmcli) in case you still want to configure some stuff manually or get debug info.
If firmware is your thing - you may want to look into Sierra Wireless EM7355, EM7455, etc series. They are linux compatible and have firmware, docs and SDK available from the Sierra site.
https://source.sierrawireless.com/devices/mc-series/mc7455/
You might want to look around the Pinephone and Pinephone Pro communities. Both phones use the EG25-G modem, chosen for better worldwide band availabilities, rather than splitting the Pinephone in "EU" and "NA" batches.
Users also are working on providing a mostly liberated firmware for the modem:
Yeah, things have become wildly fragmented since the "plain" 3G modem days, and there are no effective standards for USB multi-point devices (LTE made Ethernet emulation more popular, but weird pseudo-serials to configure them just became... weirder).
Heh, when I started researching and found that, yes, uart and AT commands are still the preferred way to interact with these modems, I was wondering if I had time-warped back to the 90s!
> Why doesn't Quectel have any firmware downloads on their site?
The telecoms industry in general is pretty shit and does a lot of anti-user nastiness like SIM locking or other arbitrary restrictions. It is not in the their best interests to let the user get too close to the inner workings of things. Security-wise it's also often terrible and security by obscurity is still very much the norm and would be threatened by more transparency. There's also only a handful of manufacturers of the underlying modem chips (Qualcomm is the main one) and they're all in bed with the industry and perpetuate the issue, and the barriers to entry to build your own module are huge both in terms of skill, factories/equipment needed to produce the chips and patents so there's little chance for a more open competitor to step up.
> And is there any way to get them besides downloading random files posted to forums?
If you can find a production-grade device that embeds the same modem that you have, search for firmware updates for that device - chances are it may embed the firmware for the modem too, which you can extract with a bit of reverse engineering. It's not great, but at least you're still getting the firmware from a more trusted source than random forum posts.
> Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)?
If you look carefully at the `ip addr` output of QMI vs ECM you would see that the IPs you got aren't the same.
In QMI mode, the LTE interface is directly connected to your system, the 10.... IP you got is the one assigned by your carrier (I am not sure where the initial 169.254.231.106 IP it had before you established the connection came from) and your machine is directly reachable from the Internet.
In ECM mode, you are given a fake Ethernet interface that's connected to a router and NAT gateway ran by the LTE module itself; this is also where that web interface comes from. In this mode, the module's router part is the one exposed to the internet and acts as a router/firewall/NAT gateway - your computer is not directly reachable unless you use that web interface to forward a port. Given the usual "quality" of these (as you witnessed by looking at the UI) I would not be surprised if there are exploits or bugs in there.
There should also be a third mode, where the modem exposes one or more serial ports that accept AT commands and you can establish a connection over it using wvdial/pppd just like good old dial-up. This is worse than QMI mode but preferable to ECM mode as you still get your system directly connected to the internet without relying on the modem's router/gateway part.
In general, I would recommend QMI mode - it is a breath of fresh air compared to the old school way of the modem exposing multiple serial ports and having to do dial-up on it. It does require some work but it's a solved problem - for desktop Linux look into ModemManager, or alternatively look into how OpenWRT deals with it for a potentially lighter alternative.
Thanks for all the advice! I'll definitely give modemmanager a spin now; I had some initial trouble but it sounds like if I can get it working it will solve a lot of the pain points I had with the manual QMI setup.
>In QMI mode, the LTE interface is directly connected to your system, the 10.... IP you got is the one assigned by your carrier (I am not sure where the initial 169.254.231.106 IP it had before you established the connection came from) and your machine is directly reachable from the Internet.
Maybe from the OS on the Raspberry Pi. I don't think it comes from the modem. Generelly I would recommend using MBIM if possible.
Some Huawei devices also had NCM, which is even better in terms of handling. You only have to send a command on the serial port to connect, then use a DHCP client on the ethernet interface and you'll get the external IP on that. Also bridge mode is way easier that way.
Kind of surprisingly, from the opposite end of complexity surprise, for me was getting a Google Fi data only SIM working on an esp32 arduino type environment. Using Lilygo's T-SIM7000G and about an hour of coding got me pinging an MQTT server. Not my video, but very helpful: https://www.youtube.com/watch?v=uMXzOegaY6U
This was something I was exploring a little bit, but as I had some early trouble—and since all the guides I could find seemed to _not_ use MM/NM—I didn't keep trying.
I would like to figure it out though, because it seems they'd be the more plug-and-play solution if they can work with the modem out of the box.
I found that using the Ubuntu Server build for RPi supported more modern modems out of the box than Raspberry Pi OS as the latter operating system links to very old builds of these tools.
> A SIM card with a 4G data plan. I bought a SixFab SIM and used their monthly pay-as-you-go service for testing. You could pull out the SIM from your mobile phone (on most carriers at least) but that is a bit inconvenient as your phone won't get service until you put the SIM back in. You could also get a plan and SIM from any major carrier (e.g. AT&T, T-Mobile, Verizon in the USA)—though you have to make sure the carrier you use works with the LTE modem you have!
As an aside: You should pull out the SIM from your mobile phone anyway, and use a VPN hotspot instead (and connect your phone to that with Wi-Fi). The carriers are monitoring all of your unencrypted DNS traffic (and your unencrypted SNI traffic that starts all of your TLS/SSL connections that says to which hostname you're connecting) and selling it, alongside your phone number and name. Use a hotspot that supports a VPN so you're only sending VPN traffic across a mobile carrier network.
(This is why T-Mobile in the US is now blocking Apple's "Private Relay" encrypted service and are complaining loudly - it's costing them sniffing-data-resale revenue.)
Same goes for the Pi: make sure you run a VPN client so you're only sending encrypted traffic across the wwan0 interface. Verify with 'tcpdump -i wwan0'.
iOS leaks requests around the VPN, sadly. I don't run an LTE base station and I don't have a jailbroken device, so I can't tell you how much, but I do know that with a VPN enabled and set to always, I get mobile-network IP DNS queries to my DNS server (not sent via VPN).
iOS is not a privacy-focused OS.
Having a discrete device means you can also connect multiple mobile devices via Wi-Fi on the same SIM, and do your own monitoring/filtering of traffic (such as blocking all unencrypted DNS, and installing DoH profiles on devices, or running an unencrypted-dns-to-DoH forwarding proxy).
By sending the DNS request out of the other interface: the mobile network… which is the expected scenario in the case of a network interruption on the Wi-Fi interface.
But after rereading the parent comment, I think they could have a misconfiguration with their VPN. I just ran a few DNS leak tests with my WireGuard setup on iOS and they all go over the VPN to my personal resolver.
Or maybe only portions of the OS is leaking around the VPN (i.e not safari but system components).
> (This is why T-Mobile in the US is now blocking Apple's "Private Relay" encrypted service and are complaining loudly - it's costing them sniffing-data-resale revenue.)
I'm fairly sure this was shown to be false. From memory, they were shown only to be blocking it on their "family monitoring" opt in plan - i.e where they sniff and intercept at the behest of the owner. As far as I know, no US mobile carriers are blocking it beyond this. Do you have evidence of this?
> The carriers are monitoring all of your unencrypted DNS traffic (and your unencrypted SNI traffic that starts all of your TLS/SSL connections that says to which hostname you're connecting) and selling it, alongside your phone number and name.
You're probably right they're collecting it, but adding PII and selling it doesn't pass the Litmus test for me. Do you have any evidence or sources of this?
Whilst the advice you provide is sound, it appears to be built on unfounded falsehoods.
> A mini PCI Express 4G LTE modem. I used a Quectel EC25-A, but there are some from Sierra Wireless I've heard recommended too.
This is actually just a mini-pci-e adapter board that exposes the USB pins to the USB type A connector seen in the photo. It's not an actual pci-e bus device LTE modem. As shown in the example, it appears to the kernel as a USB device.
This is something I haven't even researched yet, but are 5G modems (the ones that seem to be more M.2 slot style) also using USB, or are they finally running over the PCIe bus?
Seems like there would be no reason to upgrade to PCIe, since (I assume) it would be a pretty expensive update for no meaningful change in theoretical max speed.
Some have a PCIe mode, but as of now you are stuck with crappy vendor drivers (if any). Default is USB. Lenovo had an issue with their crappy notebooks in this regard. Whitelist triggered if you used it in USB mode.
I was racking my brain trying to figure out how they did PCI over USB 2 and then just had the modem show up as a USB device. Maybe USB-C would have made some sense, sorta. Had no idea that the mini-pci-e connector could also just be used for USB.
Looking for a similar solution, I had one Huawei USB 3G stick lying around and used that with RPI 3. Everything worked just fine but after a couple of days it would randomly disconnect and wouldn't restore the connection so currently, I am looking for some 3G/4G router to do the same job.
When I moved into my new house but the internet fiber was not yet activated I also used a Pi with a Huawei 3g dongle as gateway for my home. Most expensive internet I had to use for 3 weeks.
Didn't have any reconnects though but I had a script running checking the connection and if it failed it would re dial
I developed an integration for LTE for work a few years ago for a linux server, I found the experience the same in how little clear information there was. We tried quite a few different cards, though I can't remember the one that ended up "working" (the project wasn't ever deployed). I could not get qmi_wwan or wvdial to work consistently, which was crucial as if it went down someone would have to physically intervene. I ended up using plain old pppd and chatscripts, which was especially amusing to me as a late millennial who never expected to use such antiquated technologies. Another fidgety thing was the SIM pin, especially given the threat that multiple failed attempts to set it would lead to a lockout. That being said, it is far simpler to just get an LTE modem with ethernet ports and let that take care of the connection.
Super interesting. Does anyone here have an idea if the described setup also works in Europe ? And some pointers towards European alternatives if not ? There are some 4g wifi modems on the market where i live in belgium, but they are just home rooters (on 220V), not this integrated stuff in the post.
45 comments
[ 2.8 ms ] story [ 91.5 ms ] threadBut along the way I probably spent 40 hours (and counting) trying to figure out the ins-and-outs of 4G LTE modems and Linux networking when it comes to these oddball devices.
I hope some parts of this post help anyone else new to LTE networking in Linux, and maybe somebody can help clear up some of my open questions, like:
Why doesn't Quectel have any firmware downloads on their site? (And is there any way to get them besides downloading random files posted to forums?)
Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)?
Why does it seem there's no real documentation for LTE modems on Linux? What little _good_ information I found was in random presentation slides scattered around the Internet.
Other than that it's either extremely technical documentation pages that expect a very high level of familiarity already, or some very cookie-cutter blog posts that give a walkthrough to get Internet going, but don't really explain anything.
(Also in the course of my research I learned a lot from OSMOCOM's work reverse-engineering some of the LTE modems (including the QC25 I'm using): https://osmocom.org/projects/quectel-modems/wiki/EC25 ).
As for the firmware. It seems to just be how these companies do things, unfortunately. At work, we use modems from both Quectel and Simcom and the experience described in OP is very similar. For Simcom we have gotten access to a "secret-but-public" Onedrive share with documentation, drivers and firmware where they put up stuff sporadically. Sometimes the firmware updates even works!
I have a theory that Qualcomm who makes the chipsets used by Quectel and Simcom (at least for the 5G modems) imposes some kind of heavy NDA which prevents free distribution of firmware and documentation. Everything seems very "hush, hush" at least.
While I can't help answer your remaining questions, I can point to wiki I have found really helpful: http://trac.gateworks.com/wiki/wireless/modem
Regarding FW updates, have you tried opening a support ticket with Quectel? For me they have been extremely responsive and helpful, providing extra documentation on special AT commands and stuff.
I would like to use modemmanager though, it _seems_ like it's the nicer path.
This is the config I'm using: https://github.com/commaai/agnos-builder/blob/master/userspa...
You can run modemmanager in debug mode, which allows you to send AT commands over dbus (or mmcli) in case you still want to configure some stuff manually or get debug info.
Users also are working on providing a mostly liberated firmware for the modem:
- https://github.com/Biktorgj/pinephone_modem_sdk
It makes life worth living.
Thank you kind, brave soul.
The telecoms industry in general is pretty shit and does a lot of anti-user nastiness like SIM locking or other arbitrary restrictions. It is not in the their best interests to let the user get too close to the inner workings of things. Security-wise it's also often terrible and security by obscurity is still very much the norm and would be threatened by more transparency. There's also only a handful of manufacturers of the underlying modem chips (Qualcomm is the main one) and they're all in bed with the industry and perpetuate the issue, and the barriers to entry to build your own module are huge both in terms of skill, factories/equipment needed to produce the chips and patents so there's little chance for a more open competitor to step up.
> And is there any way to get them besides downloading random files posted to forums?
If you can find a production-grade device that embeds the same modem that you have, search for firmware updates for that device - chances are it may embed the firmware for the modem too, which you can extract with a bit of reverse engineering. It's not great, but at least you're still getting the firmware from a more trusted source than random forum posts.
> Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)?
If you look carefully at the `ip addr` output of QMI vs ECM you would see that the IPs you got aren't the same.
In QMI mode, the LTE interface is directly connected to your system, the 10.... IP you got is the one assigned by your carrier (I am not sure where the initial 169.254.231.106 IP it had before you established the connection came from) and your machine is directly reachable from the Internet.
In ECM mode, you are given a fake Ethernet interface that's connected to a router and NAT gateway ran by the LTE module itself; this is also where that web interface comes from. In this mode, the module's router part is the one exposed to the internet and acts as a router/firewall/NAT gateway - your computer is not directly reachable unless you use that web interface to forward a port. Given the usual "quality" of these (as you witnessed by looking at the UI) I would not be surprised if there are exploits or bugs in there.
There should also be a third mode, where the modem exposes one or more serial ports that accept AT commands and you can establish a connection over it using wvdial/pppd just like good old dial-up. This is worse than QMI mode but preferable to ECM mode as you still get your system directly connected to the internet without relying on the modem's router/gateway part.
In general, I would recommend QMI mode - it is a breath of fresh air compared to the old school way of the modem exposing multiple serial ports and having to do dial-up on it. It does require some work but it's a solved problem - for desktop Linux look into ModemManager, or alternatively look into how OpenWRT deals with it for a potentially lighter alternative.
Maybe from the OS on the Raspberry Pi. I don't think it comes from the modem. Generelly I would recommend using MBIM if possible.
Some Huawei devices also had NCM, which is even better in terms of handling. You only have to send a command on the serial port to connect, then use a DHCP client on the ethernet interface and you'll get the external IP on that. Also bridge mode is way easier that way.
I would like to figure it out though, because it seems they'd be the more plug-and-play solution if they can work with the modem out of the box.
As an aside: You should pull out the SIM from your mobile phone anyway, and use a VPN hotspot instead (and connect your phone to that with Wi-Fi). The carriers are monitoring all of your unencrypted DNS traffic (and your unencrypted SNI traffic that starts all of your TLS/SSL connections that says to which hostname you're connecting) and selling it, alongside your phone number and name. Use a hotspot that supports a VPN so you're only sending VPN traffic across a mobile carrier network.
(This is why T-Mobile in the US is now blocking Apple's "Private Relay" encrypted service and are complaining loudly - it's costing them sniffing-data-resale revenue.)
Same goes for the Pi: make sure you run a VPN client so you're only sending encrypted traffic across the wwan0 interface. Verify with 'tcpdump -i wwan0'.
iOS is not a privacy-focused OS.
Having a discrete device means you can also connect multiple mobile devices via Wi-Fi on the same SIM, and do your own monitoring/filtering of traffic (such as blocking all unencrypted DNS, and installing DoH profiles on devices, or running an unencrypted-dns-to-DoH forwarding proxy).
But after rereading the parent comment, I think they could have a misconfiguration with their VPN. I just ran a few DNS leak tests with my WireGuard setup on iOS and they all go over the VPN to my personal resolver.
Or maybe only portions of the OS is leaking around the VPN (i.e not safari but system components).
No, the idea is that you use your phone without a SIM card at all: no mobile network, wi-fi to the VPN router only.
I'm fairly sure this was shown to be false. From memory, they were shown only to be blocking it on their "family monitoring" opt in plan - i.e where they sniff and intercept at the behest of the owner. As far as I know, no US mobile carriers are blocking it beyond this. Do you have evidence of this?
> The carriers are monitoring all of your unencrypted DNS traffic (and your unencrypted SNI traffic that starts all of your TLS/SSL connections that says to which hostname you're connecting) and selling it, alongside your phone number and name.
You're probably right they're collecting it, but adding PII and selling it doesn't pass the Litmus test for me. Do you have any evidence or sources of this?
Whilst the advice you provide is sound, it appears to be built on unfounded falsehoods.
I just plug them in my laptop first, use the browser based config site to set it up and then put it on the pi.
> A mini PCI Express 4G LTE modem. I used a Quectel EC25-A, but there are some from Sierra Wireless I've heard recommended too.
This is actually just a mini-pci-e adapter board that exposes the USB pins to the USB type A connector seen in the photo. It's not an actual pci-e bus device LTE modem. As shown in the example, it appears to the kernel as a USB device.
Seems like there would be no reason to upgrade to PCIe, since (I assume) it would be a pretty expensive update for no meaningful change in theoretical max speed.
Didn't have any reconnects though but I had a script running checking the connection and if it failed it would re dial