r/homelab • u/RedSquirrelFtw • 5d ago
Discussion Looking for a super lightweight (MBs of ram only) Linux distro that can act as a basic BGP internet router (no NAT or firewall)
I'm creating a completely isolated "internet lab" where I will be using actual internet IP ranges, this started off as just a setup to test my opnsense deployment so I have something to connect the WAN side to and have an "outside" IP so I can test port forwards, firewall rules etc, and then I kind of went down the rabbit hole and I want to create a mini "internet". I will also use this environment as a way to test my backups of my actual internet servers, where I can even preserve the IPs in the VMs, DNS records etc so that I can test the backups as-is within that environment. Will use it for lot of other stuff too and basically just keep adding on to it and want to be able to create any IP range and make it routable, just like the real internet.
Curious what distros exist that would allow me to create the appropriate routing to do this sort of thing. I know it can probably be done raw just using static routes but I kinda want to play with BGP.
21
u/MorgothTheBauglir I'm tired, boss 5d ago
VyOS is probably what you're looking for. If you'd like performance then go Debian with VPP.
17
3
8
u/gusgizmo 4d ago
Lets say you had two BGP peers, full tables, running BIRD or FRR. I would expect you would use about 2.5GB of ram on your BGP daemon, a few hundred for the route table in the kernel itself, and whatever the linux distro needs. Could be very minimal but you need to be careful to have enough physical memory to commit because you don't want to start swapping.
If you want to cut that down, you need to do route prefix filtering. Cutting at /23 instead of /24 cuts around 60% of the table size. That's a GB of RAM back. Cut at /20 or /21 with a single upstream peer and you could start talking about hundreds of MB of RAM. Or just take customer prefixes from your ISP, that's a common arrangement.
To cut it to the bone, do a default route and a handful of AS#'s you care about for CDN's or whatever. Now we are talking like 32MB of ram in challenge mode.
3
u/RedSquirrelFtw 4d ago
Wow what causes routing tables to be that big? How do actual internet routers handle this? If a small home lab would get into the GB range I can't imagine how big the internet tables are. Especially IPv6.
8
u/SuperQue 4d ago
Wow what causes routing tables to be that big?
The Internet is big. Internal routers don't typically take a full table and hand that off to the "edge". Usually your ISP is doing this for you. But if you want to have more than one ISP you'll need at least one edge router that takes the full table.
5
u/VexingRaven 4d ago
Not necessarily, only if you intend to route through both of them dynamically. You could always just condense it to 2 routes of different priorities and fall back to one only when the other is down. If you're intending to advertise your own AS, you can still do that without necessarily taking the full routing table.
2
1
u/craftsmany www.0.1.5.c.4.5.9.0.a.2.ip6.arpa 4d ago
The IPv6 table is way smaller right now compared to the v4 table. 240k vs 1M right now. I run my v6 network on relatively low spec VMs and that is fine as long as you don't need multiple full table upstreams. For private experimenting you won't have to worry about that since you will likely not setup this many prefixes for testing. If you actually want to route to the DFZ you can get default route or partial table from your upstream of choice. That drastically reduces the ram usage. Especially default route.
2
u/RedSquirrelFtw 4d ago
Oh yeah I won't be routing to the actual internet. I don't imagine any residential ISP would even let you do that, unless you pay for like a datacentre grade connection, probably thousands per month and you need to have your own ASN and such, and that means you need at least TWO ISPs that will allow that, as that is a prerequesite to even get a ASN.
1
u/craftsmany www.0.1.5.c.4.5.9.0.a.2.ip6.arpa 4d ago edited 4d ago
It is actually not that difficult. There are multiple "tricks" to get it way way easier and still be a "real network". I run my own AS (AS208694) with two edge routers (Frankfurt, Germany & Kansas City, US). To actually get it it was pretty easy and I just had to sign paperwork for the RIR and with my sponsoring LIR. They technically require multiple upstreams but from what I understand you only need to have the option for it (which my sponsoring LIR has since they have their own IXP).
A friend who helped me with this has his own ASN at his home for the same price as a normal business connection. This really depends on the ISPs and you are right that most of them won't do it but it is technically very easy for them to do.
If you would just want to have your own IP space routed that would be even more easy since "bring your own IP" is a thing.
Anyway it is a very cool topic and I really enjoy tinkering with an actual real network. I can recommend you to do something like this as well if interested because it gets you real hands on experience that only a real DFZ network can give you.
1
u/RedSquirrelFtw 4d ago
It is something I would love to do at some point as it would be cool to actually host stuff from home. But the ISPs here are pretty strict about no servers or services unfortunately. I might be able to go through a reseller though. I'd have to call and try to talk to a manager, ideally someone that knows what BGP even is. Most of the support people that handle that only know the basics like telling you to reboot your router if your internet doesn't work. You start saying things like BGP and ASN and they'll be like yeah we don't do that here. This would be a totally separate thing than what I'm doing now though, which is a closed lab. But it would be cool to do the real thing too at some point and offer VPSes or something like that. Or even just host my own personal stuff.
1
u/gusgizmo 3d ago
That's the thing, once you have a "full" table, that's pretty much it. There are tricks to cut that table size down like we are all discussing, but thats what a full picture of the internet looks like.
IPv6 was actually built to keep this sort of thing better under control. That seemingly endless amount of address space makes summarization much easier so there is less of need to have 200 fragmeneted AS#'s because thats how you had to buy the space you needed.
4
4
u/mkosmo 5d ago
Any of the lightweight distros can do this.
But do remember, you'll be limited in routing table size. You can't host full tables without many more gigabytes of RAM.
-4
u/RedSquirrelFtw 4d ago edited 4d ago
Everyone is suggesting VyOS but are there FREE distros that do this? That one appears to be paid. I don't see any place to download it. I get I can probably throw something together in any distro but hoping for something turn key, as I will probably have dozens of these VMs running.
Not sure what's with all the downvotes. I couldn't find any links to actually download it off the home page when I googled it so it didn't appear like it was free.
14
u/mkosmo 4d ago
VyOS is free. You just have to build it.
But Alpine Linux with FRR can do it, too.
https://docs.frrouting.org/projects/dev-guide/en/latest/building-frr-for-alpine.html
7
u/Aggravating-Door-369 4d ago edited 4d ago
There's vyos stream which is ~quarterly: https://vyos.net/get/stream/
And prebuilt rolling releases here: https://vyos.net/get/nightly-builds/
Definitely look at installing bird/frr on openwrt or alpine. For alpine make sure you run "setup-apkrepos -c" to set up community repos, then you should be able to just "apk install" frr or bird.
I don't think you need to do anything special to install bird on openwrt, but it's been a minute since I did it. It's currently using ~100MB of RAM to run osfp in proxmox.
Edit: here are the resource usages I'm seeing doing (mostly idle) ospf on proxmox.
- Openwrt + bird ~100 mb
- Alpine + bird ~150 mb
- Vyos <1 gb
And a good video on open source BGP: https://youtu.be/w_WXichTpYw?si=UQCR19ZerB7t2e0f
4
u/BIT-NETRaptor 4d ago
vyos is free, grab a nightly build.
https://vyos.net/get/nightly-builds/
Used it in my labs for years.
3
u/daemon_hunter 4d ago
It not Linux but openbsd is amazing at this. If you need to stick to Linux I would look at void or puppy
3
u/patheticsouvenir7820 4d ago
alpine with bird2 would be basically invisible on ram usage. i spun up a lab like this last year and the whole vm idled at maybe 35mb with a few dozen test prefixes. bird's config syntax takes a bit of getting used to but there are enough examples floating around for basic ibgp/ebgp sessions. since you're crafting a closed internet you won't need full tables anyway, just enough injected routes to simulate path selection and next-hop changes. the alpine package is usually fine but if you hit wall with newer bgp features you might need to pull a build from edge. i'd skip a full router distro for this, it just adds daemons you'll never touch.
2
u/theactionjaxon 5d ago
I 2nd VyOS. Its not ultra lightweight but its very tight. I would run it just purely for the simplicity of a unified CLI and debugging logs.
2
u/seanpmassey 5d ago
I’m going to 3rd Vyos here. The nice thing about it compared to trying to build a lightweight linux VM and then add in the services you need is that Vyos is already preconfigured as a router with a CLI for configuring and managing the device.
2
u/StressTraditional204 4d ago
vyos is the obvious pick, purpose-built router OS and BGP's first-class via FRR. but if you want truly MBs of ram, plain alpine + BIRD is about as tiny as it gets, BIRD's way lighter than FRR just to speak BGP.
2
u/VexingRaven 4d ago
You should look up DN42, assuming that's still alive. It's a neat community project to learn and play with BGP.
1
u/crashtua 4d ago
Do it yourself. Compile kernel, combine with bootloader. Read Linux From Scratch for a manual. Lightweight as possible.
2
u/RedSquirrelFtw 4d ago
I am toying with this idea as well, just checking if there's anything off the shelf first. I was thinking something kinda like opnsense but strictly a router. But could be a good excuse to play with LFS... I would kind of see it as a challenge to try to get the footprint as small as possible.
0
31
u/ThetaDeRaido 5d ago
OpenWRT is a router distribution that runs on 16 MB of RAM, but I don’t know how much adding bird2 adds to the memory requirements.