Jekyll2022-08-24T09:17:23+00:00https://edoput.github.io/openwispgsoc/feed.xmlAirOS backend for OpenWISPThis is the report for my 2017 GSOC project
It is fine if your work is not merged2019-05-19T00:00:00+00:002019-05-19T00:00:00+00:00https://edoput.github.io/openwispgsoc/netjsonconfig/2019/05/19/we-did-not-merge<p>Hello there, I’m assuming you are either a GSOC student who found this blog
or somebody looking for info about integrating NetJSON and OpenWISP. Don’t worry
I have good news for both of you.</p>
<p>If you are a GSOC student and you are worried about your work not being
able to be merged in the project I’m telling you, don’t worry too much.
There is time after GSOC and the train is not leaving the station yet. You made
good friends, you did a lot of work, if you continue showing up there is always
another merge window.</p>
<p>Actually it’s also good if you don’t merge. That’s what happened with me at least
and everybody was still happy, and I was too.</p>
<p>You see after the GSOC was done I had an intuition that I could not just dump
thousands of lines of code on somebody’s lap and call it a day, after all I was
the most prominent authority on the intersection of NetJSON and AirOS (I had to become) who can
debug the thing I wrote if he’s not me?</p>
<p>So my work was not merged in netjsonconfig and instead I focused on
adding a nifty feature that would make the [very next best thing happen. <a href="https://groups.google.com/g/openwisp/c/Qehtop4YJpA/m/Q2ghvv2QCQAJ">I added
a plugin awareness to netjsonconfig and made my work a plugin</a>.</p>
<p>You see it could not be merged because the bar of quality needed to be high
but this won’t stop people now from experimenting with it because they are
not bound to my version of netjsonconfig, they can intall and uninstall
my work without loosing the tools they have.</p>
<p>So if you are still interested in trying out my plugin for AirOS
and see if it can fit your use case I suggest you install both netjsonconfig
and <a href="https://github.com/edoput/netjsonconfig-airos">netjsonconfig-airos</a>.</p>
<p>For those interested in trying out this I suggest you get in contact with
me. I don’t have access anymore to AirOS hardware to try and I can provide
guidance on how to test your configuration and how to find/work with bugs
and I am a friendly mantainer that is willing to merge patches from you (yes you!).</p>Hello there, I’m assuming you are either a GSOC student who found this blog or somebody looking for info about integrating NetJSON and OpenWISP. Don’t worry I have good news for both of you.10K2017-08-04T00:00:00+00:002017-08-04T00:00:00+00:00https://edoput.github.io/openwispgsoc/netjsonconfig/2017/08/04/10k<p>The last month began extremely weel as I had finally created my monster. One part of the configuration is obviously dedicated to the radio interface and is very device dependent. A device with two antennas, e.g. one for 2GHz and one for 5GHz, cannot be configured with my single antenna configuration. But I also had to ship something that would work on mine!</p>
<p>In the end we decided to include the radio configuration as a blob and update just what was necessary, maybe in the future we will have something better but it worked on my device. And it was glourious! I finally didn’t have to copy and paste someting before uploading it!</p>
<p>I also participated at the ICFP contest with the LILIK team, it took all the weekend and was a lot of fun to play, I look forward for the next one!</p>
<p>At this point time was running out and I still had something to work on, feel the rush?</p>
<p>But there was no rush! I could configure the antenna in bridge mode with wathever combination of network and authentication I could think of and I had the initial support for router mode with my mentors ready to help with situations such as “It happened again, I can’t get in when it’s in router mode”.</p>
<p>But this thing has ended without this support for router mode, it’s a shame but the configuration closely match the one I can get from the device and still won’t work when applied. I did various tests and now I have an hypothesis for other to work on.</p>
<p>Possibly there is some “hard” switch, a procedure I still haven’t found, to be executed before switching from bridge to router mode (and viceversa) that leaves the antenna in a broken middle ground state.</p>
<p>You can find</p>The last month began extremely weel as I had finally created my monster. One part of the configuration is obviously dedicated to the radio interface and is very device dependent. A device with two antennas, e.g. one for 2GHz and one for 5GHz, cannot be configured with my single antenna configuration. But I also had to ship something that would work on mine!5K2017-07-07T00:00:00+00:002017-07-07T00:00:00+00:00https://edoput.github.io/openwispgsoc/netjsonconfig/2017/07/07/5k<p>The second month was all about reaching feature parity with the web interface and adding tests. Unfortunately the update from AirOS 8.1.4 to 8.3.1 changed many things and I had to scrap much of my tests and code.</p>
<p>Now that I write about it I get to see that I did a lot of work during this month. Eventually everything I had to work on fit in what I could see was required, seems like a miracle.</p>
<p>During the second half my mentors asked me to begin working on the pull request <a href="https://github.com/openwisp/netjsonconfig/pull/91">#91</a> to include my work upstream.</p>
<p>I spent much of the third week just browsing the comments on the pull request searching for what to change to satisfy the reviews.</p>
<p>This was a hell of a work as my code approached 3000 lines and had a lot of stylistic problems. I had a lot to fight with flake and other tools for finding common pitfalls in Python but eventually I had the time to make them happier.</p>
<p>Moreover I had a <a href="https://github.com/openwisp/netjsonconfig/issues/93">misterious bug</a> that I could reproduce but didn’t make much sense. This was about the schema, and object that is used for validation, and how we updated it to fit into the AirOS backend. My mentor came up with a workaround but it’s still there.</p>
<p>During the last week I had to work more on the schema and provide all the useful information that would go in the netjsonconfig web editor, another OpenWISP project. It has been a refreshing task as every change had a visual meaning and most important, fast feedback!</p>
<p>One last thing for this month was the great splitting. I began splitting all I could think of into modules, interface helpers, configuration defaults, update functions and refactoring what couldn’t be separated from it’s converter.</p>The second month was all about reaching feature parity with the web interface and adding tests. Unfortunately the update from AirOS 8.1.4 to 8.3.1 changed many things and I had to scrap much of my tests and code.First run2017-06-09T00:00:00+00:002017-06-09T00:00:00+00:00https://edoput.github.io/openwispgsoc/netjsonconfig/2017/06/09/first-run<p>With the summer of code first month officialy started I got to work on more pressing (for me) problems, such as error reporting in netjsonconfig command line interface.</p>
<p>I decided to throw some time at it because I really didn’t know anything about networks and this was a problem when writing NetJSON configuration. I couldn’t get much but the simpler case to work and when it didn’t work it was like a misterious murder investigation.</p>
<p>Eventually I got it working and the prepared a pull request <a href="https://github.com/openwisp/netjsonconfig/pull/84">#84</a> for upstream. This has changed much of the experience as now I had a fully cooperating tool to work with. In retrospect without this I couldn’t have done this much.</p>
<p>Then with all the yak shawed on the netjsonconfig front I had one thing to set my mind on, hardware testing.</p>
<p>The Ninux group here in Firenze agreed to let me have one device to work on and during this month they tried teaching me useful things such as the reset procedure, having different profiles on my laptop, setting up connections and more.</p>
<p>Eventually I got all this working and begun learning how that thing worked, how it can update the antenna configuration from a file, what commands are important and what do they do.</p>
<p>This period was also about poking the web interface, trying different configuration and downloading the associated file, diffing what files I had to see what would change between two states and so on.</p>
<p>In the end I got more “scientific” and came up with a reproducible protocol for how to do things.</p>
<ol>
<li>Reset the antenna</li>
<li>Put it in the desired state</li>
<li>Download the configuration</li>
<li>Write a netjson for the very same configuration</li>
<li>Generate the file with netjsonconfig</li>
<li>Save the diff between the two</li>
</ol>
<p>Even with this protocol things got ugly because I didn’t get consistent results and the device doesn’t like to experience a sequence of reset.</p>With the summer of code first month officialy started I got to work on more pressing (for me) problems, such as error reporting in netjsonconfig command line interface.Baby steps2017-05-12T00:00:00+00:002017-05-12T00:00:00+00:00https://edoput.github.io/openwispgsoc/netjsonconfig/2017/05/12/baby-steps<p>Before the summer of code there is a month for community bonding. I spent much of that reading the code for netjsonconfig and working up a mental model of how things should work.</p>
<p>Obviously I had some opinion on how something should be documented and I opened some pull request, <a href="https://github.com/openwisp/netjsonconfig/pull/67">#67</a>, <a href="https://github.com/openwisp/netjsonconfig/pull/69">#69</a>, <a href="https://github.com/openwisp/netjsonconfig/pull/71">#71</a>.</p>
<p>During this month my work was included in <a href="https://github.com/openwisp/netjsonconfig/releases/tag/0.5.6">netjsonconfig v0.5.6</a></p>
<p>Another improvement that didn’t make into master was <a href="https://github.com/EdoPut/netjsonconfig/tree/backend-for-dummies">backend for dummies</a> I didn’t invest much time in that and in the end never made a pull request.</p>
<p>During this month I had a lot of time to think about how to represent the AirOS configuration and in the end I settled on a tree like structure that I documented <a href="http://netjsonconfig.openwisp.org/en/airos/backends/intermediate.html">here</a>.</p>
<p>Othere than that I began my daily ritual of lurking the IRC channel for user support and writing a report to the mailing list every Friday</p>Before the summer of code there is a month for community bonding. I spent much of that reading the code for netjsonconfig and working up a mental model of how things should work.The beginning2017-05-04T00:00:00+00:002017-05-04T00:00:00+00:00https://edoput.github.io/openwispgsoc/netjsonconfig/2017/05/04/welcome-to-jekyll<p>This year I got interested in a community network project called <a href="http://www.firenze.ninux.org/">Ninux</a> in Firenze. A community network is a networking project owned and run by a group of individuals instead of a company or association. The project goal is to be able to run networked applications (as the world wide web) on your network, extend the capabilities of software and make new friends.</p>
<p>The ideas behind are those of free software and the only way to realize them is by running your own hardware as there is no ISP that allows the kind of transparency that we are striving for.</p>
<p>So sometimes we climb on a roof and we try to install a ninux-node to get connectivity to Firenze’s ninux island to some other participant. Using a mix of off the shelves hardware and open source tools we can bring connectivity as fast as 1Gbs, not bad!</p>
<p>Anyway this is how I got in touch with <a href="https://github.com/gabri94">Gabriele</a> who later convinced me to apply for a GSOC project with OpenWISP.</p>
<p>I had two projects I could choose from and I got interested more in adding a backend to <a href="http://netjsonconfig.openwisp.org/en/stable/">netjsonconfig</a>.</p>
<p>Netjsonconfig provides a library and a command line interface to transform <a href="https://netjson.org/rfc.html">NetJSON</a> into other format such as UCI blocks for OpenWRT/LEDE.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"interfaces"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"eth0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"addresses"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"192.168.1.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"mask"</span><span class="p">:</span><span class="w"> </span><span class="mi">24</span><span class="p">,</span><span class="w">
</span><span class="nl">"family"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipv4"</span><span class="p">,</span><span class="w">
</span><span class="nl">"proto"</span><span class="p">:</span><span class="w"> </span><span class="s2">"static"</span><span class="p">,</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>So eventually this is how I got to work during summer on this project to add the first AirOS backend. But what is AirOS?</p>
<p>AirOS is the firmware distributed by Ubiquity on their long range WiFI devices and it comes with a web interface to configure things such as WiFi authentication, VLAN, networking and other services.</p>
<p>Is it also possible to upload a file containing the configuration from the web interface and this is where things become interesting.</p>
<p>This configuration file is easily readable and with some networking knowledges is possible to infer what state the device is in.</p>
<p>Here is an example to set the <code class="language-plaintext highlighter-rouge">eth0</code> interface to the static ip <code class="language-plaintext highlighter-rouge">192.168.1.1/24</code></p>
<figure class="highlight"><pre><code class="language-ini" data-lang="ini"><span class="py">netconf.1.devname</span><span class="p">=</span><span class="s">eth0</span>
<span class="py">netconf.1.ip</span><span class="p">=</span><span class="s">192.168.1.1</span>
<span class="py">netconf.1.mask</span><span class="p">=</span><span class="s">255.255.255.0</span></code></pre></figure>
<p>The project goal is to be able to output this configuration file from an initial NetJSON configuration and then upload it to the device skipping the web interface.</p>
<p>This is important because community network projects rely on this kind of hardware to connect distant users, being able to control not only the routers but also the antennas would greatly simplify the management of this kind of network where users (and owners) have different level of abilities.</p>
<p>With this in mind the journey started and I had to find a device to test and sample configurations.</p>This year I got interested in a community network project called Ninux in Firenze. A community network is a networking project owned and run by a group of individuals instead of a company or association. The project goal is to be able to run networked applications (as the world wide web) on your network, extend the capabilities of software and make new friends.