AdamClementsλJekyll2015-05-07T23:37:37+01:00/Adam Clements/adam.clements@gmail.com Clojure!]]>/articles/clarity-keyboard-uses-clojure2015-05-07T00:00:00+01:002015-05-07T00:00:00+01:00Adam Clementsadam.clements@gmail.com<p><img style="width:250px;float:right;shape-outside:circle();" src="/images/clarity_logo.png" alt="Clarity logo" /></p>
<p>Last week here at <a href="http://swiftkey.com">SwiftKey</a>, we released <a href="https://play.google.com/store/apps/details?id=com.swiftkey.clarity.keyboard&referrer=utm_source%3Dadamblog%26utm_medium%3Dblog%26utm_content%3Dprogrammingpost">Clarity Keyboard Beta</a> - an experimental keyboard that we’ll be using to test out some new keyboard concepts over the coming few weeks and months. It’s been built from the ground up and uses a host of bleeding-edge technologies as well as an interesting and unusual (for mobile) choice of programming language: Clojure.</p>
<p>This gave us an unprecedented degree of freedom and agility. When you can safely redefine whole chunks of your program on the fly without even restarting, it changes the way you think about designing your product and gives you the freedom to genuinely <strong>play</strong> with new features.</p>
<h2 id="what-is-clojure">What is Clojure?</h2>
<p><img style="width:250px;float:right;shape-outside:circle();" src="/images/250px-Clojure-Logo.png" alt="Clojure logo" /></p>
<p><a href="http://clojure.org/">Clojure</a> is a functional programming language that, like Java, targets the JVM.</p>
<p>For the uninitiated a language is ‘functional’ because it encourages the use of ‘pure’ functions (no side-effects - i.e. no state) and immutable data. Contrast this with an imperative language like Java in which the use of member variables (state) that can be updated many times during a program’s execution are the norm. One of the biggest benefits of this approach is that multi-threaded programming is much easier and safer if you don’t have to worry about mutable state (i.e. variables being updated by multiple threads, potentially simultaneously and/or unpredictably). It also makes chunks of code easier to reason about without having to understand all of the context in which it is used. </p>
<p>I won’t list the many other benefits of using Clojure here - you can read a more extensive <a href="http://clojure.org/rationale">rationale</a> on the Clojure website. Not everything is rosy though, there are some big drawbacks to using Clojure too. The main one being that it has historically had a very long initial load time. The time to load a Clojure application is often counted in seconds! As a result of this, Clojure is most often used to build server infrastructure where a long initial load time is irrelevant for the long running process. It has rarely been used to build consumer mobile applications.</p>
<h2 id="clojure-on-android">Clojure on Android</h2>
<p>As you might expect, the issue of the long load time was the biggest barrier to using Clojure to build a mobile application. How long would you wait for an application to load on your mobile phone before thinking it’s crashed (especially when in some cases it had… sorry people on 0.3.2 with a Samsung phone)? However with the help of the key contributors to some of the open-source libraries we’re using we’ve reached a point where using Clojure to build a mobile application in production is absolutely feasible. Clarity Keyboard’s initial loading time is around 1.5-2 seconds and there are still a number of improvements we plan to make to get this down well below a second.</p>
<p>This load time was made possible by <a href="https://github.com/alexander-yakushev">Alexander Yakushev’s</a> fork of Clojure called <a href="http://clojure-android.info/skummet/">Skummet</a>, with standard upstream Clojure the load time is still around 10 seconds, which is obviously unacceptable. Skummet takes away some of Clojure’s dynamicism in return for some extra speed. This means we accept the long load time when developing and get REPL development, then when we’re ready for launch hit the Skummet button for a free speed boost. Long term we’d really like to see some lean compilation options added to the official version of Clojure so that we don’t have to rely on an unofficial fork. Hopefully now we’ve shown that Clojure on Android is not only possible, but desirable, this will become more of a priority for upcoming versions.</p>
<h3 id="development">Development</h3>
<p>We’re using a fairly typical Clojure development setup here - nothing Android-specific and pretty standard stuff for Clojure developers: Emacs + cider.</p>
<p>One of the most powerful tools this setups offers us though - which for us was a revelation when we first tried this on Android a year or so ago - was the Clojure REPL <strong>on Android</strong>!</p>
<h3 id="the-clojure-repl">The Clojure REPL</h3>
<p>REPL stands for Read, Evaluate, Print, Loop - an environment in which you can tweak your program on the fly without re-compiling and re-installing (a process that can take several minutes). The Clojure REPL has been especially valuable for us while working on Clarity Keyboard, in particular when making subtle changes to the user experience, and has made a big difference to our development speed.</p>
<p>We no longer need to re-compile and re-install our application to test our changes, we simply re-evaluate our changes on the device and a moment later the keyboard is running our new code. In a lot of cases we can make these changes without even refreshing the keyboard on screen: A user testing our debug build might spot something wrong with the keyboard layout and we can tweak it before they’ve even finished their sentence.</p>
<h3 id="build-tools">Build tools</h3>
<p>Unfortunately, as the use of Clojure on Android is a relatively recent innovation all of the Android build tools assume that you are using standard Java, in a standard IDE, and that you won’t wander too far from the beaten path.</p>
<p>Luckily for us <a href="https://github.com/sattvik">Daniel Solano Gómez</a>, <a href="https://github.com/oakes">Zach Oakes</a> and <a href="https://github.com/alexander-yakushev">Alexander Yakushev</a> among others have done a lot of the hard work in improving Clojure support for Android as part of the <a href="http://clojure-android.info">Clojure-Android</a> group on GitHub. This includes lein-droid, a plug-in for the leiningen build system which deals with turning Clojure code into an APK, packaging resources etc. and a special build of Clojure which can actually generate Android Dalvik byte code at runtime, allowing for the dynamic behaviour mentioned earlier.</p>
<p>Having said this, when looking to make a production-quality app, there were some features which simply hadn’t been built yet and so tool support was missing for things like version code generation, bundling of native libraries, Proguard optimisation and Multi-dex (which allows you to have more than 65,000 method references). However, lein-droid is open source and its contributors have been really supportive and responsive to our proposed changes so we’ve been able to propose the missing features and have them merged into upstream quickly and they are now part of the official version.</p>
<h2 id="design-decisions-influenced-by-clojure">Design decisions influenced by Clojure</h2>
<h3 id="event-driven">Event Driven</h3>
<p>In Clarity, everything that happens in the system is an “event” expressed as pure data, namely a Clojure map. No modules can do or react to anything if it isn’t broadcast as data in an event.</p>
<div class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">{</span><span class="ss">:type</span> <span class="ss">:start-input</span> <span class="ss">:editor</span> <span class="p">{</span><span class="ss">:type</span> <span class="ss">:plain</span> <span class="ss">:action</span> <span class="ss">:send</span><span class="p">}}</span>
<span class="p">{</span><span class="ss">:type</span> <span class="ss">:touch</span> <span class="ss">:touches</span> <span class="p">[[</span><span class="mf">0.12</span> <span class="mf">0.43</span> <span class="mi">12332</span><span class="p">]]</span> <span class="ss">:action</span> <span class="ss">:down</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:type</span> <span class="ss">:touch</span> <span class="ss">:touches</span> <span class="p">[[</span><span class="mf">0.12</span> <span class="mf">0.43</span> <span class="mi">12332</span><span class="p">]</span> <span class="p">[</span><span class="mf">0.13</span> <span class="mf">0.43</span> <span class="mi">12333</span><span class="p">]]</span> <span class="ss">:action</span> <span class="ss">:up</span><span class="p">}</span></code></pre></div>
<p>The fact that everything is data means that we can really easily see everything that’s happening in the system when debugging - simply print out the event stream as things pass through it. It also means that if we want to test a system in isolation, it’s trivial to spin up only that system and simply inject the events that it expects. All application configuration is simply done by appending a <code>:persist :true</code> key to <strong>any</strong> event which simply indicates that the settings module should store that particular event and replay it on application startup.</p>
<p>This is all made possible by persistent immutable data structures in Clojure. You can happily send the events off to as many receivers as you want and know that the operation of one cannot change the copy that another sees. It also means that we can broadcast for example every time a new sample is added to a touch trace, <strong>without</strong> creating an entirely new list, it will share structure with the previously emitted touch event but without changing it for any modules that are still processing it.</p>
<p>In practice this is really powerful and leads to excellent modularity.</p>
<h3 id="modular">Modular</h3>
<p>Because all subsystems only care about events in/out, and don’t care at all about where those events come from or where they go to, it’s absolutely trivial to replace whole subsystems or add new ones, without breaking what’s already there. It also means practically no conflicts when merging feature branches, because each new feature can be implemented in isolation. This has the added benefit that if you’re not sure about a feature, you can simply keep the old feature code but not activate it (and it will get stripped out in the build step). By making sure everything considers the fact that there may be multiple receivers for their events, it’s easy and encouraged to keep everything nice and generic, so you don’t end up with all your modules depending on each other in a spaghetti code mess.</p>
<p><img src="/images/pseudo-architecture-clarity.png" alt="Clarity architecture diagram" /></p>
<p>The fact that everything is nice and modular makes things very very easy to test. You can simply fire events in one side of a module and check that the appropriate events come out the other side. Or spin up a couple of modules and fire events in to check that they interact with one another as expected. This combined with <a href="https://github.com/clojure/test.check">test.check</a> to generate all sorts of unforeseen properties of the incoming events and check that various properties hold true have made testing a breeze. We plan to post again in the future about our testing setup.</p>
<h3 id="totally-asynchronous">Totally Asynchronous</h3>
<p>Given that we have immutable, persistent data and many largely independent modules, it makes sense to make everything asynchronous by default. <a href="https://github.com/clojure/core.async">core.async</a> was an excellent fit for us as it allowed us to set up our central stream of events totally asynchronously but in such a way that we can guarantee ordering of events (i.e. if a <code>:change-layout</code> event happens followed by a <code>:touch</code> event they are guaranteed to arrive that way round, even if some other module is being really slow processing <code>:change-layout</code> events).</p>
<p>While it’s really powerful, and beats using threading primitives hands down, it is still asynchronous code, and asynchronous code is always hard to reason about and even harder to test. In practice, our unit tests tend not to spin up modules using core.async, but instead feed lists of events through the module using <a href="http://clojure.org/transducers">transducers</a> where possible, and get a simple list out the other side.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We’ll be publishing more detailed posts about our experience building mobile applications in Clojure on Android; technologies we’ve used such as <a href="https://github.com/clojure/test.check">test.check</a>, <a href="http://clojure-android.info/">Clojure on Android</a> and <a href="https://github.com/clojure/core.async">core.async</a>; REPL-driven development; Clojure itself and the Clarity project over the next few weeks and months. Let us know in the comments if there’s anything in particular you’d like us to cover in more depth.</p>
<p><a href="https://play.google.com/store/apps/details?id=com.swiftkey.clarity.keyboard&referrer=utm_source%3Dadamblog%26utm_medium%3Dblog%26utm_content%3Dprogrammingpost">
<img alt="Get it on Google Play" style="width:150px" src="https://developer.android.com/images/brand/en_generic_rgb_wo_45.png" />
</a></p>
<p>Clarity Keyboard Beta is available on the Google Play store now - try it out and let us know what you think.</p>
<p><a href="http://adamclements.github.io">Adam Clements</a> (Tech Lead) and <a href="http://benleavett.github.io">Ben Leavett</a> (Product Manager)</p>
<p><a href="http://swiftkey.com/en/blog/what-makes-clarity-keyboard-tick-clojure/">(Reposted from the SwiftKey Tech Blog)</a></p>
<p><a href="/articles/clarity-keyboard-uses-clojure">What Makes Clarity Keyboard Tick? Clojure!</a> was originally published by Adam Clements at <a href="">AdamClements</a> on May 07, 2015.</p>/articles/core-async-runtime-dependencies2015-01-24T00:00:00+00:002015-01-24T00:00:00+00:00Adam Clementsadam.clements@gmail.com<h2 id="problem">Problem</h2>
<p>I have been working on an android application which uses a core.async based event bus at its core (a subject for a future post). This works fantastically, but takes an absolute age to load on first run - in the order of 20 seconds. This week I’ve spent some time trying to work out why that is and do something about it.</p>
<p>One of the main issues I came across was that when requiring <code>core.async</code> pulls in the massive clojure.tools.analyzer, even if it has been AOT (Ahead Of Time) compiled. This shouldn’t be necessary, because the analyzer is only used in macros and none of that code should be necessary at runtime as macroexpansion should already have taken place.</p>
<p>The reason it loads is that Clojure doesn’t have any separation between runtime code and macros when it comes to the usual method of declaring dependencies - the ns form. When <code>clojure.core.async</code> is required, which contains the go macro and friends, <code>clojure.core.async.impl.ioc-macros</code> needs to be pulled in so that it can work at compile time and as a result all the same namespaces get imported at runtime as well.</p>
<h2 id="solution">Solution</h2>
<p>My initial solution was to set <code>ioc-macros</code> to <code>^{:skip-aot true}</code> in the namespace form, but its dependencies on tools.analyzer were still loading. Then I tried moving the require to compile time so that it wouldn’t resolve at run time at all:</p>
<div class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="kd">defmacro </span><span class="nv">ignore-me-if-aotd</span> <span class="p">[]</span>
<span class="p">(</span><span class="nf">require</span> <span class="o">'</span><span class="p">[</span><span class="nv">clojure.tools.analyzer</span> <span class="ss">:as</span> <span class="nv">an</span><span class="p">]</span>
<span class="o">'</span><span class="p">[</span><span class="nv">clojure.tools.analyzer.ast</span> <span class="ss">:as</span> <span class="nv">ast</span><span class="p">]</span>
<span class="o">'</span><span class="p">[</span><span class="nv">clojure.tools.analyzer.jvm</span> <span class="ss">:as</span> <span class="nv">an-jvm</span><span class="p">])</span>
<span class="nv">nil</span><span class="p">)</span>
<span class="p">(</span><span class="nf">ignore-me-if-aotd</span><span class="p">)</span></code></pre></div>
<p>And this did the trick. The code compiles and runs without loading the tools.analyzer namespaces. As an added benefit, it doesn’t even generate class files for tools.analyzer and its transitive dependencies, cutting down on package size.</p>
<h2 id="issues">Issues</h2>
<p>There is one remaining problem with this, and that’s REPL support. If you now attach a repl and try to evaluate a go block or any of the macros depending on this, it will fail. For the moment I have simply made two builds - one core.async.runtime and one core.async which I use for development and bear the startup time. Longer term however, I could potentially see a more solid solution where the require form is moved inside the <code>(defmacro go ...)</code> itself, allowing the dependencies to be packaged but not loaded unless needed. This would incur a little extra cost at compilation time and incur some complexity overhead as there might be more than one entry point to this code and backtracking what dependencies would be required isn’t totally trivial.</p>
<h2 id="future-enhancements">Future enhancements</h2>
<p>Clojurescript doesn’t have this problem because its macros are separated by necessity, not having runtime code evaluation at all.</p>
<p>Ideally I would like to see something like Clojurescript’s <code>(:require-macros ...)</code> form in clojure, whose implementation simply added to a list of namespaces to require if anything tries to evaluate a macro and otherwise does nothing. There’s no reason I can think of that this couldn’t be totally compatible with Clojurescript’s require-macros form so that the same code could work in both situations without need for feature expressions.</p>
<p><a href="/articles/core-async-runtime-dependencies">Quicker core.async bootstrapping</a> was originally published by Adam Clements at <a href="">AdamClements</a> on January 24, 2015.</p>