A (largely) open-source cellular router!

[tj90241]

Some years ago, I built my parents (who are full-timing) a Peplink/Crandlepoint alternative due to the higher cost and closed-source nature of those solutions. What I ended up going with was based on a commercial-off-the-shelf router, the Turris Omnia, which I stuck a 4G LTE (Telit LM940) modem in. All said and done, I spent somewhere around 40-50% less in cost than a comparable Peplink. For the first few years, it worked well... but it's been getting especially bad lately -- both speed-wise and in terms of connection stability.

I wrote it off as the carrier (Verizon) moving to 5G and went to look at what options were available mid last year. The Peplink BR1 Pro 5G with X62 modem is/was going for $999.99 before tax/shipping, which still felt a bit steep today for what you get. It was also disappointing to see that Peplink is now using Quectel (Chinese) modems rather than something designed in the West -- the former BR1 Pro 5G modem used a Sierra Wireless X55 modem IIUC. Mix in with the fact that they do not seem to support upgrading the modem, it does not have the software support I want (Wireguard), etc... no thanks!

So I did something similar this year and went full-bore custom. Ended up building a system using:
* Mini-Box M300 case with 80W PicoPSU (DC/DC converter)
* Supermicro A3SEV-4C-LN4 motherboard (Intel Atom x6425E quad-core processor, up to 3GHz turbo)
* Sierra Wireless EM9291 PCI-e (Qualcomm Snapdragon X62)
* Qualcomm WCN6855 PCI-e (2x2 WiFi-6E)

Hardware-wise, the only thing I regret is not having a better solution to mount the modem onto the motherboard. However, it's a small price to pay -- everything about the hardware is standard! SATA ports, replaceable DIMMs, PCI-e lots (2xM.2 and a fat PCI-e), etc. The Atom processor is even capable of in-band ECC for the DRAM!

In particular, I can now just upgrade the modem, and only the modem, when it inevitably becomes obsolete/EOL and keep the rest of the hardware unchanged. Here's a side-profile view of the case with 2 SSDs mounted on top showing that it's still fairly compact:

All loaded with 1 SSD, the WiFi 6E and 5G radios, 16GB RAM, etc. - it pulls just under 9W at idle from an AC brick.

Moreover, it's running a bog-standard Linux distribution (Debian GNU/Linux "bookworm" with a 6.6 LTS kernel)! By day, I'm a cloud engineer, so having something that's readily administrable and not behind some proprietary UI/CLI is a huge win for me. e.g. due to the genecity of the platform, I was able to spin up something on my parents laptops that dumps a backup of their files periodically to the SSD on the router, and from there I periodically sync the router's backup to an offsite one. Using the router as an intermediary data tank works well as I can do the offsite backup pull off-hours and their laptops don't have to be running for it to all work.

It's also nice knowing that the majority of the software in the router will receive software/security updates essentially for perpetuity and not subject to a vendor EOL'ing the platform.

For the antenna, we went with a Panorama White MAKO Dome Antenna (LP-IN2443-W) run to the roof, which I understand is fairly common here.

All said and done, it did require a good bit of technical know-how to get going -- and was rougher than what I expected... in particular:
* Patches to the Linux kernel to add support for the EM9291 (X62) modem. As of now, only EM9191 (X55) is supported.
* A connection manager to manage the data sessions for the modem and maintain the kernel's routing tables. It's perhaps possible to use the already existing modem-manager software, but I wanted something a little more tightly integrated with the modem than what its feature set offered, and Sierra Wireless (the modem manufactuer) provides a QMI SDK with the modem, so why not...
* A lot of time banging my head on iptables (Linux firewalling)...

The real challenges I faced were the (still yet) dropped connections from Verizon. After much hair pulling, I found out the hard way that Verizon will drop your entire cellular data session if you route them any packets with a source address other than that for which the modem is assigned. I had a SNAT setup so that, in theory, should not have happened... but in some cases, packets without a connection tracking state would sidestep the NAT due to an oversight on my part. As a result, Verizon will get a packet whose source address was one behind the NAT and they'd kill the connection. Something to check if you are getting unexpectedly dropped connections from your provider.

All said and done, I've got it just about to the point where we get maintain the same IPv4/v6 address for >24 hours and there are no drops (fingers crossed...). I'm so tickled that it just works! Short of the modem/BIOS firmwares and QMI SDK/libraries, everything is open-source at its core! I did a speedtest run earlier today and clocked 90Mbps downlink at the park we're currently staying at whereas (using the same provider/data plan), a Pixel 4a 5G cellphone inside the rig was struggling to pull 1Mbps on the same speedtest.

I'm hoping to be able to share all the configuration/software involved if anyone has the necessary background to pick it up and make it work -- let me know. It should be possible to do a cost-reduced version.

 

[JRS]

I'm one of the Parents involved with the post above. Trust me.... it's a sweet setup. Speed tests as high as 90+mbps vs my cell phone (while in the rig) getting less than 5mbps

 

[redbaron]

Some years ago, I built my parents (who are full-timing) a Peplink/Crandlepoint alternative due to the higher cost and closed-source nature of those solutions. What I ended up going with was based on a commercial-off-the-shelf router, the Turris Omnia, which I stuck a 4G LTE (Telit LM940) modem in. All said and done, I spent somewhere around 40-50% less in cost than a comparable Peplink. For the first few years, it worked well... but it's been getting especially bad lately -- both speed-wise and in terms of connection stability.

I wrote it off as the carrier (Verizon) moving to 5G and went to look at what options were available mid last year. The Peplink BR1 Pro 5G with X62 modem is/was going for $999.99 before tax/shipping, which still felt a bit steep today for what you get. It was also disappointing to see that Peplink is now using Quectel (Chinese) modems rather than something designed in the West -- the former BR1 Pro 5G modem used a Sierra Wireless X55 modem IIUC. Mix in with the fact that they do not seem to support upgrading the modem, it does not have the software support I want (Wireguard), etc... no thanks!

So I did something similar this year and went full-bore custom. Ended up building a system using:
* Mini-Box M300 case with 80W PicoPSU (DC/DC converter)
* Supermicro A3SEV-4C-LN4 motherboard (Intel Atom x6425E quad-core processor, up to 3GHz turbo)
* Sierra Wireless EM9291 PCI-e (Qualcomm Snapdragon X62)
* Qualcomm WCN6855 PCI-e (2x2 WiFi-6E)

Hardware-wise, the only thing I regret is not having a better solution to mount the modem onto the motherboard. However, it's a small price to pay -- everything about the hardware is standard! SATA ports, replaceable DIMMs, PCI-e lots (2xM.2 and a fat PCI-e), etc. The Atom processor is even capable of in-band ECC for the DRAM!

In particular, I can now just upgrade the modem, and only the modem, when it inevitably becomes obsolete/EOL and keep the rest of the hardware unchanged. Here's a side-profile view of the case with 2 SSDs mounted on top showing that it's still fairly compact:
View attachment 24823

All loaded with 1 SSD, the WiFi 6E and 5G radios, 16GB RAM, etc. - it pulls just under 9W at idle from an AC brick.

Moreover, it's running a bog-standard Linux distribution (Debian GNU/Linux "bookworm" with a 6.6 LTS kernel)! By day, I'm a cloud engineer, so having something that's readily administrable and not behind some proprietary UI/CLI is a huge win for me. e.g. due to the genecity of the platform, I was able to spin up something on my parents laptops that dumps a backup of their files periodically to the SSD on the router, and from there I periodically sync the router's backup to an offsite one. Using the router as an intermediary data tank works well as I can do the offsite backup pull off-hours and their laptops don't have to be running for it to all work.

It's also nice knowing that the majority of the software in the router will receive software/security updates essentially for perpetuity and not subject to a vendor EOL'ing the platform.

For the antenna, we went with a Panorama White MAKO Dome Antenna (LP-IN2443-W) run to the roof, which I understand is fairly common here.
View attachment 24824

All said and done, it did require a good bit of technical know-how to get going -- and was rougher than what I expected... in particular:
* Patches to the Linux kernel to add support for the EM9291 (X62) modem. As of now, only EM9191 (X55) is supported.
* A connection manager to manage the data sessions for the modem and maintain the kernel's routing tables. It's perhaps possible to use the already existing modem-manager software, but I wanted something a little more tightly integrated with the modem than what its feature set offered, and Sierra Wireless (the modem manufactuer) provides a QMI SDK with the modem, so why not...
* A lot of time banging my head on iptables (Linux firewalling)...

The real challenges I faced were the (still yet) dropped connections from Verizon. After much hair pulling, I found out the hard way that Verizon will drop your entire cellular data session if you route them any packets with a source address other than that for which the modem is assigned. I had a SNAT setup so that, in theory, should not have happened... but in some cases, packets without a connection tracking state would sidestep the NAT due to an oversight on my part. As a result, Verizon will get a packet whose source address was one behind the NAT and they'd kill the connection. Something to check if you are getting unexpectedly dropped connections from your provider.

All said and done, I've got it just about to the point where we get maintain the same IPv4/v6 address for >24 hours and there are no drops (fingers crossed...). I'm so tickled that it just works! Short of the modem/BIOS firmwares and QMI SDK/libraries, everything is open-source at its core! I did a speedtest run earlier today and clocked 90Mbps downlink at the park we're currently staying at whereas (using the same provider/data plan), a Pixel 4a 5G cellphone inside the rig was struggling to pull 1Mbps on the same speedtest.

I'm hoping to be able to share all the configuration/software involved if anyone has the necessary background to pick it up and make it work -- let me know. It should be possible to do a cost-reduced version.

I am very interested in this setup, please share the details

 

[Kevin D Pem]

Have

I am very interested in this setup, please share the details

You considered an open source project? I would predict in less than a month, thousands of qualified professionals would be onboard, and shortly after your creation would find it's way into many, many homes world wide!!!

 

[tj90241]

If you're going to dabble with this, I highly suggest doing so with Sierra Wireless modems. They are one of the two vendors I've seen who readily provide the necessary tooling and firmware for the modems without requiring an NDA.

Before you order anything, click here and click "sign up" in the top right. Register an account with them and ensure you can then download the files on that page, and that they have firmware for the carrier you intend to use the modem with. I don't think my account is special in any way, though it would be good to confirm.

In order to get started, you need to procure this enclosure along with a modem of your choice. The enclosure is important because it holds down pins on the modem when it boots that forces the modem to operate in a "USB only" mode (even though they're driven through PCI-e primarily). I was only ever able to get the modem flashing utility to work when it was in this mode. The modems do not always come pre-flashed with carrier-specific firmware (or outdated firmware at that). This adapter is more expensive than ones that look similar on Amazon, but it is the only one I found that have interoperability with higher end modems.

Next, select a modem. Any of the Sierra ones should do really. I went with a modem based on Qualcomm's X62 chipset, the EM9291. Digikey also sells an X65 variant, the EM9293 which should work all the same -- though the only thing it really gets you is mmWave... the frequency band that they use in dense, urban cities that can barely penetrate signal through a tree...

You should also be able to do this with the EM7565 (4G) modem if you want to prove it all out without investing as much, though I haven't tested it so you may also hit some snags in doing so.

---

Beyond this, it's just a matter of running some of the Sierra Wireless tools to flash the modem with the carrier firmware if your choice -- I'll add some steps on how to do that in the coming days.

The nice part about the enclosure is that you can actually use it to develop/test/iterate a solution that works for you -- everything works fine in this USB-only mode. Both the open-source modem-manager software and the software that I wrote will work with it.

 

[tj90241]

Have

You considered an open source project? I would predict in less than a month, thousands of qualified professionals would be onboard, and shortly after your creation would find it's way into many, many homes world wide!!!

One already exists largely for this purpose: ModemManager. I cannot comment on it as I have not used it personally, though I think this is what OpenWRT is using nowadays.

Beyond that, an open-source project takes a ton of energy/time -- something I'm short on these days...

 

[redbaron]

Did you write your own modem connection manager?

 

[tj90241]

Did you write your own modem connection manager?

Yep! The Turris Omnia I was formerly using used an OpenWrt derivative, which itself used some kind of shell script with a bunch of qmicli/uqmi invocations to configure the modem. It left a bit to be desired (e.g., it would not listen for any indications from the modem stack that the call dropped... so you'd have to go whack the router when that happened).

My parents are not technical and I do not want to be goofing with the thing, so I wrote a connection manager in straight C which leverages Sierra's QMI/QMUX libraries. It does the bare minimum in terms of QMI calls right to get the thing up and running and plugs in the address/routes gleaned through QMI calls to the modem into the kernel using netlink. Once it's up, it just sits in a loop and watches for indications from the modem that a call dropped/something exceptional, and just bounces the modem's data sessions and programs the interface/routing table as needed.

I can throw the source code for it up on GitHub later. I will unfortunately have to remove the Sierra libraries/header files from the repo and provide instructions on how to obtain those externally as they're only available under Sierra's SLA/licensing (this is why you need to have an account with them). There are also some things which would have to be cleaned up for general use -- e.g., I hard coded the APN/3GPP profile ID for Verizon, I hard coded the addition of some static routes that are populated when the modem goes up for my VPN needs, etc. It's not too intrusive and those could easily be cut out, though.

 

[redbaron]

Very interesting. I am going to look at the sierra wireless apis.

I do agree with your comments on OpenWRT not being stable for cellular.

I like the idea of a raw Debian, shore wall, WireGuard router. The only thing I want to figure out is how to also add a layer for cellular as wan and easy enough for the non-it person to change.

Peplink has this done very well. I am not aware of an existing open source project for this but will do some searching.

 

[tj90241]

Sorry for the long delay -- here's a link the the source:

GitHub - tj90241/modem-monitor: WWAN modem control and monitoring service

WWAN modem control and monitoring service. Contribute to tj90241/modem-monitor development by creating an account on GitHub.

github.com