Friday, April 28, 2017

Bringing up 802.11ac on FreeBSD

I've been chipping away at bringing up 802.11ac on FreeBSD. I've been meaning to write this post for a while, but you know, life got in the way.

net80211 has reasonably good 802.11n support, but no 802.11ac support. I decided a while ago to start adding basic 802.11ac support. It was a good exercise in figuring out what the minimum set of required features are and another excuse to go find some of the broken corner cases in net80211 that needed addressing. I'll talk about a few here.

802.11ac introduces a few new concepts that the stack needs to understand. I decided to use the QCA 802.11ac parts because (a) I know the firmware and general chip stuff from the first generation 11ac parts well, and (b) I know that it does a bunch of stuff (like rate control, packet scheduling, etc) so I don't have to do it. If I chose, say, the Intel 11ac parts then I'd have to implement a lot more of the fiddly stuff to get good behaviour.

So, the first step was to survey what I needed to at bring up a very small subset of functionality. The typical first target is "monitor" mode, since that requires channel configuration to work, but doesn't require the rest of the stack to know about anything (ie, no 11ac aware IE negotiation.)

Step one - adding VHT channels. I decided in the shorter term to cheat and just add VHT channels to the already very large ieee80211_channel map. The linux way of there being a channel context rather than hundreds of static channels to choose from is better in the long run, but I wanted to get things up and running. So, that's what I did first - I added VHT flags for 20, 40, 80, 80+80 and 160MHz operating modes and I did the bare work required to populate the channel lists with VHT channels as well.

This required adding support to lib80211 and ifconfig to support loading VHT channels from the regulatory domain (regdomain.xml.) And yes, this let me get VHT channels statically configured.

Then I needed to add VHT capabilities to the driver structure (ieee80211com) and the virtual interface structure (ieee80211vap.) This is so that drivers can be told that yes, we can do VHT things.

Then I needed to glue it into an 11ac driver. My ath10k port was far enough along to attempt this, so I added enough glue to say "I support VHT" to the ic_caps field and propagated it to the driver for monitor mode configuration. And yes, after a bit of dancing, I managed to get a VHT channel to show up in ath10k in monitor mode and could capture 80MHz wide packets. Success!

So at this point I took a big of a segue and decided to ensure that the receive packet flags made sense for VHT operation. I did a bunch of work to tidy this up so I could mark frames as 20/40/80/etc and had all of that mostly done in a few days.

Next up was station operation. So this required a few things to get right:
  • Channel promotion (ie, going from 11a -> 11n -> 11ac)
  • Knowing about VHT IEs, both for transmit and receive
  • Negotiating VHT capabilities to announce to the AP during association request/response and then telling the driver about what's going on.
By far the most fiddly was getting channel promotion to work. net80211 supports the concept of dumb NICs (like atheros 11abgn parts) very well, where you can have multiple virtual interfaces but the "driver" view of the right configuration is what's programmed into the hardware. For firmware NICs which do this themselves (like basically everything sold today) this isn't exactly all that helpful. So, for now, it's limited to a single VAP, and the VAP configuration is partially derived from the global state and partially derived from the negotiated state. It's annoying, but it is adding to the list of things I will have to fix later.

Sorry, I went off on a bit of a tangent. So one of the things net80211 does is it looks at the list of desired channels for each VAP and determines what the "NIC" channel should be in order to support it. So yes, ic_curchan is being used and it's what gets updated. the vap->iv_des_chan is the "I'd like to be this channel" but the active channel is actually vap->iv_bss->ni_chan (yes, the BSS node current channel.) That's what gets promoted. I need to uhm, "fix" the per-VAP state to have a "current channel" that gets updated with the BSS channel - so that's high on the list of things to fix.

Then the next part for channel promotion is how it's done. IT's not done via a call to "promote channel" - it instead was done by a call to net80211's 802.11n routines which just parse HT (11n) IEs. Yes, parsing an 11n IE also upgraded the channel. So I spent a bunch of time refactoring that, and now the bits that parse IEs are separate from channel promotion. Fun times.

So, once channel promotion worked, I associated fine as an open mode 11ac station. All of the negotiated pieces were wrong, so I spent a few days looking at packet captures to ensure I negotiated VHT capabilities correctly, but it seems to work fine.

Next was adding in support for things that stopped me doing encrypted data. The TL;DR here is - net80211's crypto key handling model needed overhauling.

Firstly the key management code assumes everything is synchronous and can't sleep. For firmware based devices this really isn't true. So, a lot of drivers defer key management into taskqueues to run things. This sounds fine, but then you realise that once the key addition succeeds, net80211 then will transmit frames expecting them to be encrypted. Userland too has this problem - it will do an ioctl() to set a key, then when it returns OK, it will queue further key negotiation frames. So, I had to do some uhm, clever things to try and work around this, and I'll cover how I need to fix it properly later.

Next up - the QCA chips/firmware do 802.11 crypto offload. They actually pretend that there's no key - you don't include the IV, you don't include padding, or anything. You send commands to set the crypto keys and then you send unencrypted 802.11 frames (or 802.3 frames if you want to do ethernet only.) This means that I had to teach net80211 a few things:

  • frames decrypted by the hardware needed to have a "I'm decrypted" bit set, because the 802.11 header field saying "I'm decrypted!" is cleared
  • frames encrypted don't have the "i'm encrypted" bit set
  • frames encrypted/decrypted have no padding, so I needed to teach the input path and crypto paths to not validate those if the hardware said "we offload it all."
So yeah, that's now done.

(Now, there's a whole separate discussion about hostap that I'll defer for later. Suffice to say - at this point I got hostap mostly working.)

Now comes the hard bit of fixing the shortcomings before I can commit the driver. There are .. lots.

The first one is the global state. The ath10k firmware allows what they call 'vdevs' (virtual devices) - for example, multiple SSID/BSSID support is implemented with multiple vdevs. STA+WDS is implemented with vdevs. STA+P2P is implemented with vdevs. So, technically speaking I should go and find all of the global state that should really be per-vdev and make it per-vdev. This is tricky though, because a lot of the state isn't kept per-VAP even though it should be.

So, I need to fix the following to be per-vdev rather than global:
  • Slot time configuration
  • ERP configuration (ie, 11g BSS overlap with 11b)
  • Frame protection configuration
  • QoS/WMM configuration
  • 20/40 channel width configuration - which needs extending to 11ac channel widths too
Then there are some things which are just plainly not implemented. A-MPDU offload for example is fine, but A-MSDU offload pretends that there are multiple 802.11 frames with the same sequence number / crypto information. This is problematic for the sequence number de-duplication code and also A-MPDU offload - in both cases, all but one A-MSDU frames are just tossed. This drastically kills performance. I'm almost done implementing A-MSDU in A-MPDU offload support, and when that's done, I'll implement A-MSDU offload de-duplication awareness and performance should drastically rise.

Another thing as mentioned is channel width, as well as other 802.11n things that need extending for 802.11ac. A bit one that I'd like to get right is SMPS (spatial multiplexing power save.) If the chip supports it in firmware then we just need to allow it to be configured and controlled, but that isn't even an option right now. I'd like to implement SMPS for Atheros 11n parts in ath(4), but that's a lot more work.

Anyway, so far so good. I need to do some of the above and land it in FreeBSD-HEAD so I can finish off the ath10k port and commit what I have to FreeBSD. There's a lot of stuff coming - including all of the wave-2 stuff (like multiuser MIMO / MU-MIMO) which I just plainly haven't talked about yet.

Viva la FreeBSD wireless! 

Thursday, April 27, 2017

(finally) investigating how to get dynamic WDS (DWDS) working in FreeBSD!

I sat down recently to figure out how to get dynamic WDS working on FreeBSD-HEAD. It's been in FreeBSD since forever, and it in theory should actually have just worked, but it's extremely not documented in any useful way. It's basically the same technology in earlier Apple Airports (before it grew into what the wireless tech world calls "Proxy-STA") and is what the "extender" technology on Qualcomm Atheros chipsets implement.

A common question I get from people is "why can't I bridge multiple virtual machines on my laptop and have them show up over wifi? It works on ethernet!" And my response is "when I make dynamic WDS work, you can just make this work on FreeBSD devices - but for now, use NAT." That always makes people sad.

So what is it?

With normal station / access point setups, wireless frames have up to three addresses. In the header it's "address 1", "address 2", and "address 3".

Depending upon the packet type, these can be a variety of addresses:

  • SA - source address - source of the packet (eg the STA address)
  • TA - transmitter address - STA/AP that transmitted the frame
  • RA - receiver address - immediate destination of the packet 
  • DA - finally recipient of the data
  • BSSID - BSS ID (ie, AP mac address)
There are a lot of addresses. There are, in fact, more than the number of address fields in a normal 802.11 frame.

Now, if you want to understand when each of these are used in which frames, you can totally find blog posts from people which describe things (eg will fill you in. But the TL;DR for normal AP/STA traffic is:

  • From an AP, the frame has BSSID, SA of the MAC (eg ethernet behind the AP bridge) sending data, and DA is the STA MAC address
  • From a STA, the frame has BSSID, TA is the STA that transmits, and DA is the final destination of the frame (eg ethernet behind the AP bridge.)
The big note here is that there's not enough MAC addresses to say "please send this frame to a station MAC address, but then have them forward it to another MAC address attached behind it in a bridge." That would require 4 mac addresses in the 802.11 header, which we don't get.

.. except we do. There's a separate address format where from-DS and to-DS bits in the header set to 1, which means "this frame is coming from distribution system to a distribution system", and it has four mac addresses. The RA is then the AP you're sending to, and then a fourth field indicates the eventual destination address that may be an ethernet device connected behind said STA.

If you don't configure up WDS, then when you send frames from a station from a MAC address that isn't actually your 802.11 interface MAC address, the system would be confused. The STA wouldn't be able to transmit it easily, and the AP wouldn't know how to get back to your bridged ethernet addresses.

Ok, so how does this work with WDS? The above from/to-DS mode is actually the technical hilarity behind "Wireless Distribution System",  which is a fancy way of saying "an AP connects to another AP and can relay frames for you." It's what was used for extending wireless networks before true meshing solutions came into existence.

The original WDS was a statically configured thing. You'd configure up a particular device as a WDS extender, and both sides would need configuring:

  • The central AP would need to know the MAC address of a WDS master, so it would know that frames from/to that particular AP needed the four-address frame format, and
  • The extender AP would need to be configured to talk to the central AP to act as a WDS master - it would then associate as a station to that central AP, and would use 4-address frames to relay traffic to it.
 So for static configurations, this works great. You'd associate your extender AP as a station of the central AP, it'd use wpa_supplicant to setup encryption, then anything between that central AP and that extender AP (as a station) would be encrypted as normal station traffic (but, 4-address frame format.)

But that's not very convenient. You have to statically configure everything, including telling your central AP about all of your satellite extender APs. If you want to replace your central AP, you have to reprogram all of your extenders to use the new MAC addresses.

So, Sam Leffler came up with "dynamic WDS" - where you don't have to explicitly state the list of central/satellite APs. Instead, you configure a central AP as "dynamic WDS", and when a 4-address frame shows up from an associated station, it "promotes" it to a WDS peer for you. On the satellite AP, it will just find an AP to communicate to, and then assume it'll do WDS and start using 4-address frames. It's still a bit clunky (there's no beacon, probe request, etc IEs that say "I do dynamic WDS!" so you'd better make ALL your central APs a different SSID!) but it certainly is better than what we had.

(Yes, one of the things I'll be doing to FreeBSD now that this works is adding that concept of "I'm a DWDS primary!" node concept so satellites can just "find" a DWDS primary enabled AP to associate to. Baby steps..)

But, I tried it - and ... let's just say, the documentation didn't say very much. So I couldn't really get it to work.

Then a friend pointed out he figured it out. (Thankyou Edward!)

Firstly, there are scripts in src/tools/tools/net80211/ - setup.wdsmain and setup.wdsrelay. These scripts are .. well, the almost complete documentation on a dynamic WDS setup. The manpage doesn't go into anywhere near enough information.

So I dug into it. It turns out that dynamic WDS uses a helper daemon - 'wlanwds' -  which listens for dynamic WDS configuration changes and will do things for you. This is what runs on the central AP side. Then it started making sense!
  • For dynamic WDS, there are no WDS interface types created by default
  • net80211 will post a routing socket message if a 4-address frame shows up on a "dwds" enabled interface, which is the signal to userland to plumb up a DWDS interface for that particular peer
  • wlanwds is then responsible for creating that virtual interface with the right configuration
  • Then it runs a shell script that you provide which lets you do things like assign it to a bridge group so it can bridge traffic
  • Finally, if the station goes away, wlanwds will get another notification from net80211 saying the station has gone, and wlanwds will destroy the virtual interface for that peer.
So far, so good. I followed that script, modified it a bit to use encryption, and .. well, it half worked. Association worked fine, but no traffic was passing.

A little tcpdump'ing later showed what was going on!
  • 4-address frames from the extender side was successfully being encrypted and transmitted to the central AP
  • 4-address frames from the central AP were being send, but unencrypted
  • .. so the station dropped them as well, unencrypted when they should've been encrypted.
A little more digging showed the actual problem - the dynamic WDS example scripts are for an open/unencrypted network. If you are using an encrypted network, the central AP side needs to enable privacy on the virtual interfaces so traffic gets encrypted with the parent interface encryption keys. So adding this:

ifconfig $DEV wepmode mixed

.. to the shell script for when an interface was created made everything work.

Now, I've only done enough testing to show that indeed it is working. I haven't done anything like pass lots of traffic via iperf, or have a mix of DWDS and normal STA peers, nor actually run it for longer than 5 minutes. I'm sure there will be issues to fix. However - I do need it at home, as I can't see the home AP from the upstairs room (and now you see why I care about DWDS!) and so when I start using it daily I'll fix whatever hilarity ensues.

Thursday, August 4, 2016

FreeBSD on a tiny system; what's missing

Now that I'm trying to use more userland services, there are some obvious shortcomings which need addressing.

The first is a lack of real service management. FreeBSD doesn't have a service management daemon - the framework assumes that daemons implement their own background and monitoring. It would be much nicer if init or something similar to init could manage services and start/restart them where appropriate. Yes, this is one of those arguments for systemd. Eg, maybe I want to only start telnetd or dropbear/sshd whenever a connection comes in. But I'd also like to be able to add services for monitoring, such as dnsmasq and hostapd.

The next is a lack of suitable syslog daemon. Yes, I'd like to be able to log some messages locally - even if it's only a couple hundred kilobytes of messages. I'd also like to be able to push messages to a remote service. Unfortunately the FreeBSD syslog daemon doesn't do log rotation or maximum log file sizes itself - it's done by "newsyslog" which runs out of cron. This isn't any good for real embedded systems with limited storage.

Then yes, there's a lack of a cron service. It'd be nice to have that integrated into the service management framework so things could be easily added/removed. I may just use cron, but that means cron is also always running which adds memory footprint (~1.3 megabytes) for something that is almost never actually active. When you have 32MB of RAM, that's quite a bit of wasted memory.

Finally, there's a lack of some message bus and notification mechanism for device changes. For example, openvpn-client creates a tunnel device - ok, so what should then check to see if a NAT configuration needs updating? Or updated firewall rules? It can be done with shell scripts (which I'll write tomorrow) but ideally there'd be something like dbus (a dirty word, I know) where these systems could push updates to and events could be triggered from them. I'd like to be able to run ntpdate whenever an interface comes up, because yes, there is no RTC on this hardware.

With all of the above in mind, I'll start working on some of it tomorrow. Hopefully I can automate the openvpn NAT configuration a little bit more so I can optionally have wifi NAT or openvpn NAT, depending upon the current requirement. Fixing ntpdate to run out of dhclient as part of the 'up' script may be helpful. I'll see what else I can do to tidy things up before I start the process of merging all of this back into freebsd-wifi-build.

At least this year I can now use the defcon wireless with all of my devices.

Wednesday, August 3, 2016

Musings on bringing up services on freebsd-wifi MIPS devices, or "why cross compiling is important"

FreeBSD has run on these MIPS routers for quite some time, but it was limited to what ships in base. There's not been any cross-built packages as part of the image building, which means we can't easily have third-party functionality.

Now, some of this third-party functionality is pretty important these days. Relying on telnet sucks; I'd like to have dropbear as an SSH server so we at least have SSH. Not having a DNS relay or DHCP server also sucks; dnsmasq would solve this problem. I'd also like some VPN services, so openvpn would be nice.

So, I eventually snapped a few months ago and started integrating some external toolchain compiler use with the freebsd-wifi-build scripts. bapt@freebsd did a whole lot of work to build ports of cross-compilers to be used by the FreeBSD ports and base system so I'm leveraging that for doing MIPS cross compiling. A bit of hacking later, and I'm cross compiling dnsmasq, dropbear, openvpn and lua.

Then I needed to integrate things. I wrote up a bunch of simple startup script wrappers to generate suitable config files for these services. Everything except the openvpn server/client configuration is in the rc.conf file, which will eventually make it much easier to turn into a configuration database.

OpenVPN was the most amusing. It cross compiled fine, save needing liblzo for compression (so that's disabled for now.) However, FreeBSD's openvpn package is version 2.3 but the easyrsa component is actually from 3.0 - which means all the documentation is out of date.

I used this for the OpenVPN config:

And this for easyvpn:

And digitalocean have a writeup for how to convert the config file into a combined config file and certification bundle:

A few things tripped me up:
  • as mentioned before - the lack of freebsd openvpn documentation that works with easyvpn 3.0 made things frustrating;
  • openvpn really wants valid system time, so I am going to have to run ntpdate when the WAN comes up;
  • there's no RTC on many of these router boards, making time keeping very much reliant on NTP;
  • kernel NAT works pretty well, but it needs interfaces to be up before you can add them. I'll have to add some scripts to openvpn-client to setup the NAT state once the link comes up so this stops being a problem;
I'll go into a little more detail about the details in a future post. But, hi from being behind an openvpn-client LAN gateway!

Saturday, June 18, 2016

Debugging TDMA on the AR9380

So, it turns out that TDMA didn't work on the AR9380. I started digging into it a bit more with AR9380's in 5GHz mode and found that indeed no, it was just transmitting whenever the heck it wanted to.

The first thing I looked at was the transmit packet timing. Yes, they were going out at arbitrary times, rather than after the beacon. So I dug into the AR9380 HAL code and found the TX queue setup code just didn't know how to setup arbitrary TX queues to be beacon-gated. The CABQ does this by default, and the HAL just hard-codes that for the CAB queue, but it wasn't generic for all queues. So, I fixed that and tried again. Now, packets were exchanged, but I couldn't get more than around 1mbit of transmit throughput. The packets were correctly being beacon gated, but they were going out at very long intervals (one every 25ms or so.)

After a whole lot of digging and asking around, I found out what's going on. It turns out that the new TX DMA engine in the AR9380 treats queue gating slightly different versus previous chips. In previous chips you would see it transmit whatever it could, and then be gated until the next time it could transmit. As long as you kept poking the AR_TXE bit to re-start queue DMA it would indeed continue along transmitting whenver it could. But, the AR9380 TX DMA FIFO works differently.

Each queue has 8 TX FIFO descriptors, which can contain a list of frames or a single frame. For the CABQ I just added the whole list of frames in one hit and that works fine. But for the normal data paths it would push one frame into a TX DMA FIFO slot. If it's an A-MPDU aggregate then yes, it'd be a whole list of frames, but still a single PPDU. But for non-aggregate traffic it'd push a single frame in.

With this in mind, the TX DMA gating now works on FIFO slots, not just descriptor lists. That is, if you have the queue setup to gate on something (say a timer firing, like the beacon timer) then that un-gating is for a single FIFO slot only. If that FIFO slot has one PPDU in it then indeed it'll only burst out a single frame and then the rest of the channel burst time is ignored. It won't go to the next FIFO slot until the burst time expires and the queue is re-gated again. This is why I was only seeing one frame every 25ms - that's the beacon interval for two devices in a TDMA setup. It didn't matter that the queue had more data available - it ran out of data servicing a single TX FIFO slot and that was that.

So I did some local hacks to push more data into each TX FIFO slot. When I buffered things and only leaked out 32 frames at a time (which is roughly the whole slot time worth of large frames) then it indeed behaved roughly at the expected throughput. But there are bugs and it broke non-TDMA traffic. I won't commit it all to FreeBSD-HEAD until I figure out what's going on

There's also something else I noticed - there was some situation where it would push in a new frame and that would cause the next frame to go out immediately. I think it's actually just scheduling for the next gated burst (ie, it isn't doing multiple frames in a single burst window, but one every beacon interval) but I need to dig into it a bit more to see what's going on.

In any case, I'm getting closer to working TDMA on the AR9380 and later chips.

Oh, and it turns out that TDMA mode doesn't add some of the IEs to the beacon announcements - notably, no atheros fast-frames announcement. This means A-MSDUs or fast-frames aren't sent. I was hoping to leverage A-MSDU aggregation in its present state to improve things, even if it's just two frames at a time. Hopefully that'd double the throughput - I'm currently seeing 30mbit TX and 30mbit RX without it, so hopefully 60mbit with it.)

Friday, May 27, 2016

Updating the broadcom driver part #2

In Part 1, I described updating the FreeBSD bwn(4) driver and adding some support for the PHY-N driver from b43. It's GPL, but it works, and it gets me over the initial hump of getting support for updated NICs and initial 5GHz operation.

In this part, I'll describe what I did to tidy up RSSI handling and bring up the BCM4322 support.

To recap - I ported over PHY-N support from b43, updated the SPROM handling in the bus glue (siba(4)), and made 11a OFDM transmission work. I was lucky - I chose the first 11n, non-MIMO NIC that Broadcom made which behaved sufficiently similarly to the previous 11abg generation. It was non-MIMO and I could run non-MIMO microcode, which already shipped with the existing firmware FreeBSD builds. But, the BCM4322 is a 2x2 MIMO device, and requires updated firmware, which brought over a whole new firmware API.

Now, bwn(4) handles the earlier two firmware interfaces, but not the newer one that b43 also supports. I chose BCM4321 because it didn't require firmware API changes and anything in the Broadcom siba(4) bus layer, so I could focus on porting the PHY-N code and updating the MAC driver to work. This neatly compartmentalised the problem so I wouldn't be trying to make a completely changed thing work and spending days chasing down obscure bugs.

The BCM4322 is a bit of a different beast. It uses PHY-N, which is good. It requires the transmit path setup the PLCP header bits for OFDM to work (ie, 11a, 11g) which I had to do for BCM4321, so that's good. But, it required firmware API changes, and it required siba(4) changes. I decided to tackle the firmware changes first, so I could at least get the NIC loaded and ready.

So, I first fixed up the RX descriptor handling, and found that we were missing a whole lot of RSSI calculation math. I dutifully wrote it down on paper and reimplemented it from b43. That provided some much better looking RSSI values, which made the NIC behave much better. The existing bwn(4) driver just didn't decode the RSSI values in any sensible way and so some Very Poor Decisions were made about which AP to associate to.

Next up, the firmware API. I finished adding the new structure definitions and updating the descriptor sizes/offsets. There were a couple of new things I had to handle for later chip revision devices, and the transmit/receive descriptor layout changed. That took most of a weekend in Palm Springs (my first non-working holiday in .. well, since Atheros, really) and I had the thing up and doing DMA. But, I wasn't seeing any packets.

So, I next decided to finish implementing the siba(4) bus pieces. The 4322 uses a newer generation power management unit (PMU) with some changes in how clocking is configured. I did that, verified I was mostly doing the right thing, and fired that up - but it didn't show anything in the scan list. Now, I was wondering whether the PMU/clock configuration was wrong and not enabling the PHY, so I found some PHY reset code that bwn(4) was doing wrong, and I fixed that. Nope, still no scan results. I wondered if the thing was set up to clock right (since if we fed the PHY the wrong clock, I bet it wouldn't configure the radio with the right clock, and we'd tune to the wrong frequency) which was complete conjecture on my part - but, I couldn't see anything there I was missing.

Next up, I decided to debug the PHY-N code. It's a different PHY revision and chip revision - and the PHY code does check these to do different things. I first found that some of the PHY table programming was completely wrong, so after some digging I found I had used the wrong SPROM offsets in the siba(4) code I had added. It didn't matter for the BCM4321 because the PHY-N revision was early enough that these SPROM values weren't used. But they were used on the BCM4322. But, it didn't come up.

Then I decided to check the init path in more detail. I added some debug prints to the various radio programming functions to see what's being called in what order, and I found that none of them were being called. That sounded a bit odd, so I went digging to see what was supposed to call them.

The first thing it does when it changes channel is to call the rfkill method with the "on" flag set on, so it should program on the RF side of things. It turns out that, hilariously, the BCM4322 PHY revision has a slightly different code path, which checks the value of 'rfon' in the driver state. And, for reasons I don't yet understand, it's set to '1' in the PHY init path and never set to '0' before we start calling PHY code. So, the PHY-N code thought the radio was already up and didn't need reprogramming.


I commented out that check, and just had it program the radio each time. Voila! It came up.

So, next on the list (as I do it) is adding PHY-HT support, and starting the path of supporting the newer bus (bhnd(4)) NICs. Landon Fuller is writing the bhnd(4) support and we're targeting the BCM943225 as the first bcma bus device. I'll write something once that's up and working!

Thursday, May 19, 2016

Updating the broadcom softmac driver (bwn), or "damnit, I said I'd never do this!"

If you're watching the FreeBSD commit lists, you may have noticed that I .. kinda sprayed a lot of changes into the broadcom softmac driver.

Firstly, I swore I'd never touch this stuff. But, we use Broadcom (fullmac!) parts at work, so in order to get a peek under the hood to see how they work, I decided fixing up bwn(4) was vaguely work related. Yes, I did the work outside of work; no, it's not sponsored by my employer.

I found a small cache of broadcom 43xx cards that I have and I plugged one in. Nope, didn't work. Tried another. Nope, didn't work. Oh wait - I need to make sure the right firmware module is loaded for it to continue. That was the first hiccup.

Then I set up the interface and connected it to my home AP. It worked .. for about 30 seconds. Then, 100% packet loss. It only worked when I was right up against my AP. I could receive packets fine, but transmits were failing. So, off I went to read the transmit completion path code.

Here's the first fun bit - there's no TX completion descriptor that's checked. There is in the v3 firmware driver (bwi), but not in the v4 firmware. Instead, it reads a pair shared memory registers to get completion status for each packet. This is where I learnt my first fun bits about the hardware API - it's a mix of PIO/DMA, firmware, descriptors and shared memory mailboxes. Those completion registers? Reading them advances the internal firmware state to read the next descriptor completion. You can't just read them for fun, or you'll miss transmit completions.

So, yes, we were transmitting, and we were failing them. The retry count was 7, and the ACK bit was 0. Ok, so it failed. It's using the net80211 rate control code, so I turned on rate control debugging (wlandebug +rate) and watched the hilarity.

The rate control code was never seeing any failures, so it just thought everything was hunky dory and kept pushing the rate up to 54mbit. Which was the exact wrong thing to do. It turns out the rate control code was only called if ack=1, which meant it was only notified if packets succeeded. I fixed up (through some revisions) the rate control notification path to be called always, error and success, and it began behaving better.

Now, bwn(4) was useful. But, it needs updating to support any of the 11n chipsets, and it certainly didn't do 5GHz operation on anything. So, off I went to investigate that.

There are, thankfully, three major sources of broadcom softmac information:
  • Linux b43
  • Linux brcmsmac
The linux folk did a huge reverse engineering effort on the binary broadcom driver (wl) over many years, and generated a specification document with which they implemented b43 (and bcm-v3 for b43legacy.) It's .. pretty amazing, to be honest. So, armed with that, I went off to attempt to implement support for the first 11n chip, the BCM4321.

Now, there's some architectural things to know about these chips. Firstly, the broadcom hardware is structured (like all chips, really) with a bunch of cores on-die with an interconnect, and then some host bus glue. So, the hardware design can just reuse the same internals but a different host bus (USB, PCI, SDIO, etc) and reuse 90% of the chip design. That's a huge win. But, most of the other chips out there lie to you about the internal layout so you don't care - they map the internal cores into one big register window space so it looks like one device.

The broadcom parts don't. They expose each of the cores internally on a bus, and then you need to switch the cores on/off and also map them into the MMIO register window to access them.

Yes, that's right. There's not one big register window that it maps things to, PCI style. If you want to speak to a core, you have to unmap the existing core, map in the core you want, and do register access.

Secondly, the 802.11 core exposes MAC and PHY registers, but you can't have them both on at once. You switch on/off the MAC register window before you poke at the PHY.

Armed with this, I now understand why you need 'sys/dev/siba' (siba(4)) before you can use bwn(4). The siba driver provides the interface to PCI (and MIPS for an older Broadcom part) to probe/attach a SIBA bus, then enumerate all of the cores, then attach drivers to each. There's typically a PCI/PCIe core, then an 802.11 core, then a chipcommon core for the clock/power management, and then other things as needed (memory, USB, PCMCIA, etc.) bwn(4) doesn't attach to the PCI device, it sits on the siba bus as a child device.

So, to add support for a new chip, I needed to do a few things.

  • The device needs to probe/attach to siba(4);
  • The SPROM parsing is done by siba(4), so new fields have to be added there;
  • The 802.11 core revision is what's probe/attached by bwn(4), so add it there;
  • Then I needed to ensure the right microcode and radio initvals are added in bwn(4);
  • Then, new PHY code is needed. For the BCM4321, it's PHY-N.
There are two open PHY-N implementations - brcmfmac is BSD licenced, and b43's is GPL licenced. I looked at the brcmfmac one, which includes full 11n support, but I decided the interface was too different for me to do a first port with. The b43 PHY-N code is smaller, simpler and the API matched what was in the bcm-4 specification. And, importantly, bwn(4) was written from the same specification, so it's naturally in alignment.

This meant that I would be adding GPLv2'ed code to bwn(4). So, I decided to dump it in sys/gnu/dev/bwn so it's away from the main driver, and make compiling it in non-standard. At some point yes, I'd like to port the brcmfmac PHYs to FreeBSD, but I wanted to get familiar with the chips and make sure the driver worked fine. Debugging /all/ broken and new pieces didn't sound like fun to me.

So after a few days, I got PHY-N compiling and I fired it up. I needed to add SPROM field parsing too, so I did that too. Then, the moment of truth - I fired it up, and it connected. It scanned on both 2G and 5G, and it worked almost first time! But, two things were broken:
  • 5GHz operation just failed entirely for transmit, and
  • 2GHz operation failed transmitting all OFDM frames, but CCK was fine.
Since probing, association and authentication in 2GHz did it at the lowest rate (CCK), this worked fine. Data packets at OFDM rates failed with a PHY error of 0x80 (which isn't documented anywhere, so god knows what that means!) but CCK was fine. So, off I went to b43 and the brcmfmac driver to see what the missing pieces were.

There were two. Well, three, but two that broke everything.

Firstly, there's a "I'm 5GHz!" flag in the tx descriptor. I set that for 5GHz operation - but nothing.

Secondly, the driver tries a fallback rate if the primary rate fails. Those are hardcoded, same as the RTS/CTS rates. It turns out the fallback rate for 6MB OFDM is 11MB CCK, which is invalid for 5GHz. I fixed that, but I haven't yet fixed the 1MB CCK RTS/CTS rates. I'll go do that soon. (I also submitted a patch to Linux b43 to fix that!)

Thirdly, and this was the kicker - the PHY-N and later PHYs require more detailed TX setup. We were completely missing initializing some descriptor fields. It turns out it's also required for PHY-LP (which we support) but somehow the PHY was okay with that. Once I added those fields in, OFDM transmit worked fine.

So, a week after I started, I had a stable driver on 11bg chips, as well as 5GHz operation on the PHY-N BCM4321 NIC. No 11n yet, obviously, that'll have to wait.

In the next post I'll cover fixing up the RX RSSI calculations and then what I needed to do for the BCM94322MC, which is also a PHY-N chip, but is a much later core, and required new microcode with a new descriptor interface.