Looking back at the FullStack Fest

It was a couple of months or so ago when I came across this conference called FullStack Fest, skimming through the agenda, I was immediately intrigued and thought “I’ve got to check this out”. The coolest bit? The conference was taking part in the beautiful city of Barcelona.

September finally came around, and just as the Berlin air was getting chilly and bringing signs of the impending winter, I was flying off to the warmth of Spain. I got there a bit early and spent a nice Sunday roaming around on the streets, admiring the architecture and the history. The next day began the Backend bit of the FullStack Fest. It was interesting to step into the intricate world of the architecture of buildings one day, and after admiring it, to step into the equally intricate world of Software Architecture the next.

Sun baked and refreshed, I went to the conference, all set with a notebook and a bag of goodies from the organisers. One must collect stickers for the laptop after all.

The backend days of the conference were abstractly divided into “DevOps” and “Architecture” with the topic being “Problems of Today, Wonders from the Future”. To describe the theme of the conference in a single word, I would say “Distributed Systems”.

Day 1:  DevOps

The first talk was by Karissa McKelvey (@okdistribute). She talked about a project which would allow people to share their scientific work without the consumers having to pay for it. A common problem in research is getting access to the required data, journals, publications etc. This is so because a lot of bureaucracy, censorship and corporate licenses get in the way of open sourcing knowledge. Karissa and her team have worked on something called the Dat Project. This creates a distributed network of many data hosts (mostly universities), through which you can upload your files and download any file through a little identifier. You can access Karissa’s presentation from the conference using this little identifier (dat://353c5107716987682f7b9092e594b567cfd0357f66730603e17b9866f1a892d8) once you install the dat tool on your machine. Though this is still vulnerable to being used as an illegal file hosting service, it’s a good step towards making data and knowledge more reachable and transparent.

Following up on this was an interesting introduction to Ethereum as a way to enter ‘contracts’ without trusting a third party such as a notary, this is done by distributing the idea of trust amongst many participants. As Luca Marchesini (@xbill82) said in his talk:

“The machine is everywhere.. The machine is nowhere”.

With the beautiful underlying power of the Nakamoto consensus protocol that powers the blockchain and the added flexibility of Turing complete capabilities, allowing you to express the intent of your contract and its fulfilment in terms of an actual computer program, you can have the word of truth floating around in the world, verifiable and undeniable.

With the buzz words “microservices” and “serverless” applications going around, one would of course be expecting a talk on these topics. Marcia Villalba (@mavi888uy) gave a great talk on what “serverless” really means…and no, it does not mean there is no server (of course). The idea of a serverless application is to utilise the cloud and write self contained functions to do simple tasks. Some highlights from the talk worth remembering are:

  • Functions in the cloud are triggered by events, they do not have state.
  • Pay as you go, scale automatically.
  • Create a proof of concept and then optimise your solution to take advantage of the cloud.
  • Automate: your CI pipeline and your testing.
  • Reduce latency by being at the edge of the cloud.

Next we stepped into the world of cyber security with Dr. Jessica Barker (@drjessicabarker), who talked about tackling vulnerabilities, specifically those introduced by negligence on the part of an end user. She talked about educating users on security instead of treating them as the weakest link in the chain and ‘dumbing things down’. She made her case in light of the Pygmalion Effect, according to which higher expectations lead to better performance. A common problem when building human friendly security guidelines is that the user is treated as a dumb entity and that leads to the user acting like a dumb entity.

Frank Lyaruu (@lyaruu) then came in with an anecdote about how he wanted a swiss army knife that did everything when he was a child, and ended up with an utterly useless one. It was quite easy to see the analogy here… we have all faced feature bloat, we’ve all wanted a framework to do everything and then been frustrated with the abstractions that make customisations a nightmare. Frank introduced the concept of ‘fullstack databases’. The key idea? Identify your use case and use the right database for it. While SQL databases may work for one scenario, GraphQL would be much better in another. The take away:

“Your thinking influences your choice of tools and your choice of tools influences your thinking.”

A representative from Booking.com, Sahil Dua (@sahildua2305) , then told us how Booking.com handles their deep learning models in production. The problem they need to solve is that different data scientists need access to an independent environment for training. They have their training script in a container, and a container runs on every needed server. The load of containers is managed by Kubernetes. This indeed was a lesson in how to manage different containers and independent environments with very high performance needs.

As Software Engineers, we know one thing for sure, and that is that things will, at some point, fail.

“There are two kinds of systems, those which have fails and those which will.”

Aishraj Dahal (@aishraj) walked us through chaos management. Some useful principles that he talked about were to:

  • Automate what you can and to have a framework for dealing with incidents.
  • Define what a “minor” and “major” incident means..
  • Define business failures in terms of business metrics, for example, the amount of revenue lost per hour of down time..
  • Single Responsibility Principle: One person should be responsible for one task in an incident, if everyone is combing through the git history looking for the last stable commit, its redundant work..
  • Never hesitate to escalate.
  • You need an incident commander, this person is the one who orchestrates the efforts to get back on track.

Day 2: Architecture

The second day of the FullStack Fest began with an exciting talk by John Graham Cumming (@jgrahamc) on the Internet of Things as the vector for DDoS attacks. He showed how vulnerable IoT devices are, with simple lapses like having telnet open on port 23. These devices are exploited by sending small http requests to a server, and sending A LOT of them, demanding a large response targeted towards a victim. As an employee of Cloudflare he could shed some light on how network patterns are used to discern legitimate and other requests. Some ways to protect yourself against DDoS attacks are to install something to do rate limiting, block every entry point that you do not need and use DDoS protection tools from a vendor such as Cloudflare.

One of my favourite talks from Day 2 included James Burns’ (@1mentat) introduction to chaos engineering and distributing tracing. He began by defining a practical distributed system as one that is observable and resilient. Observability comes with tracing whereas resilience can be tested through Chaos Engineering i.e. intentionally causing a system to fail as a “drill” and having the engineers on board try to fix it without knowing the cause of the problem or even what the problem is. If you have many such drills, when real chaos hits the team will be well prepared to tackle it.

Chris Ford (@ctford) took the stage and talked about a hipster programming language called Idris which can be used to specify distributed protocols. In Ford’s words, his 10th rule of microservices is:

“Any sufficiently complicated microservice architecture contains an ad-hoc, informally-specified, bug-ridden, slow implementation of a distributed protocol.”

A distributed protocol’s specification can be tricky to get right. With a language like Idris, whose compiler checks the types, where functions are value and even types are values, the level of strictness when specifying a protocol is greatly increased and the chances of runtime bugs reduced as the compiler is smart enough to capture protocol violations. A protocol can be thought of as a finite state machine and is so specified in the Idris programming language. Be forewarned though, this is still ongoing research and definitely not production ready!

We then dove into philosophy, the nature of order and structure preserving transformations with Jerome Scheuring(@aethyrics). He talked about identifying the core of the application and then building transformations around it. The key being that the structure of your application remains the same when more layers are added onto it. He hinted at functors as a tool for achieving such transformations of architecture.

After some lightning talks and a tutorial on ‘hacking’ into systems that only exist for a few milliseconds (lambdas which are only alive for the scope of a simple execution) and then on how to defend such systems, the backend bit of the conference came to a close.

The conference was a pretty cool look into research topics meeting with the software industry and creating some innovative solutions to existing problems. Though I haven’t listed all the talks here, you can check them out on youtube: https://www.youtube.com/watch?v=_V4tGx85hUA&t=536s.

I left Barcelona having felt that I have gazed into the future of technology and seen the wheels set in motion for many advancements set to come in the next few years. Though the conference could have been even better if it had some more topics related more explicitly to everyday software development, I feel that I walked out a more knowledgeable person than before.

Screen Shot 2017-10-04 at 13.45.21

Broadening one’s horizons, beyond the scope of their job description is not only intellectually stimulating but also makes for a more content and productive mind. Small Improvements, by sponsoring this trip (and many others for their employees’ learning and development) is creating a happier and smarter workplace. I am yet again in Berlin, at my desk, ready to tackle more challenges and apply the numerous things I gleaned from the FullStack Fest. Looking forward to next conference!

Using Haskell to Find Unused Spring MVC Code

Screen Shot 2016-12-02 at 14.43.09.png

Not into reading text? Click here for the code.

Like a lot of people at Small Improvements I’m fascinated by functional programming. After coming back from our company trip in San Francisco I had trouble beating jet lag due to spending the evenings reading about monad transformers, I’m not kidding, it actually kept me awake.

For a while I’ve been thinking about cleaning up a little in our codebase, mainly the backend which is written in Java. I have known for ages that Haskell is really good with abstract syntax trees (ASTs) and was playing with the thought of creating a Haskell tool that would help me with this. However, to not completely violate the “do not reinvent the wheel” rule I first had a quick look at what’s already out there.

Finding An Existing Tool or Building My Own

Most of the developers at work use IDEA (for editing Java) which has built in tools for finding unused code and do all different kinds of code analysis. I tried using it for finding unused code a couple of times with different settings but didn’t manage to get acceptable results. The number of false positives was way too high for it to be useful, in addition to this it was incredibly slow. I also tried Findbugs without satisfying results.

I’m sure it’s possible to configure some existing software, but rather than spending more time finding a COTS-tool I figured I might just code it myself. I was thinking that if it’s specific to our project it shouldn’t be so hard. I quickly realized regular expressions wouldn’t be enough or would be very tricky to use and limit my flexibility. This left me with the choice of writing a custom parser or building a proper AST and work with that.

I have bad experience of working with ASTs in Java, but Haskell is another story, traversing a tree is a piece of cake. I had a quick look at Hackage and noticed that someone already has written a parser for Java in Haskell, so it was settled, I was starting Small Improvements’ first, albeit small, Haskell project. Finally I got to use Haskell at work!

My Solution For Finding Unused Code

It is actually quite simple to find unused Java code. Let’s have a look at my solution. In essence I’m reading all the .java-files in a folder, building an AST using language-java and then traversing the AST to collect information that can later be used to decide if a file is used or not.

The main information I’m looking for is whether any other file imports a file. However, since Java does not require an import statement if the dependency is within the same package I also look for other things such as method calls. After this I’m using the information to actually find unused files.

To find unused files I’m building a graph. Nodes are files and an edge means that a file is used by another file. So the challenge here is to actually add an edge every time a file is used. An obvious thing to do is to add an edge for every import statement.

To improve the result further I’m adding edges for references within a package, eg. used classes or methods within the package. However, this is not enough since Spring MVC has a powerful dependency injection system. It supports injecting dependencies and still only relying on interfaces. You can get all classes of a type (interface) injected or one specific instance but still only depending on its interface.

When harvesting the AST I also collected autowired classes and superclasses. Using this I filtered out files that are autowired, either directly or via an interface. The result is not 100% perfect, but with a small blacklist of classes and some other trivial filtering I managed to make it good enough for it to be very useful. Everything I get from the AST is modeled using the following data structure:

data Result = Result { fileName :: String
                     , imports :: [String]
                     , references :: [String]
                     , topLevelAnnotations :: [String]
                     , methodAnnotations :: [String]
                     , implements :: [String]
                     , autowired :: [Autowiring]
                     } deriving (Show)

Have a look at the code and try it on your own Spring MVC project. Feel free to comment here if you need help or have suggestions of improvements. Let’s now compare coding Haskell with Java / JavaScript that we normally do at Small Improvements.

Reflection of Development With Haskell

I’m a big fan of Haskell and have been for ages. One of the first things I noticed is the wonderful support you get from the compiler. When the compiler blesses your code it is very likely to just work. Once you have established that your code works, that it behaves correctly, then it is really difficult to accidentally change its behavior when refactoring. You might break it, as in making it not compile, but once it compiles again it is very likely to behave like before.

Composition is just beautiful. It strongly promotes breaking your program into trivial pieces and then glueing them together. Types are excellent documentation, the type signature together with the function name often makes it easy to guess exactly what the function does. It’s easy to write relatively clean code in Haskell. I think that the pureness and composition of small functions almost automatically makes it happen.

Actually, in Haskell it is a bit difficult to write functions that are hundreds of lines of code doing many different things. In Java or JavaScript that is what many people begin doing, and something they only unlearn as they become more skilled. I think that it is possible to produce nice code in all languages, but Haskell does help you quite a lot to keep your code nice, not to mention hlint. Haskell does not guarantee that you produce good code though, let’s look at some of my learnings from this project.

Learnings From This Project

One thing I learned is that type aliases are very useful, you should use them whenever it makes your code more readable. Comments are in general not needed if the type signature and function name is good.

Naming your code increases readability, for example extracting out small pieces of code to the where clause of a function or simply making them top-level functions in the module. Putting too many functions that are relatively complex in the where clause is a bad idea, because you lose the explicit type signature (you should always specify it for top-level functions) which makes it difficult to directly understand when they can be used and how they can be combined. A small example of a nice usage of the where clause is:

transformToEdges :: Result -> Node
transformToEdges r = (r, fileName r, outgoingEdges)
  where outgoingEdges = references r ++ imports r ++ implements r

Note the increased readability in the top level expression. The where-clause is used to hide the messy details of what outgoing edges are behind a simple name. By using where it is often possible to make the top level expression very easy to read.

Curried functions are just awesome, they make it possible to compose almost any function. A good way to design them is to think of functions as being configured and getting what they operate on as the final argument.

Lazy evaluation is powerful, I still need to practice how to leverage it fully, but it is important to be aware of it. For example in my case I ran into problem when reading all files lazily. This caused my program to have too many open file handles. It was easily solved though, by hacking a bit to force the complete file to be read directly:

readFileStrict :: FilePath -> IO String
readFileStrict path = do
  file <- readFile path
  _ <- evaluate $ length file
  return file

Recursion further promotes clean code (small functions) and is quite easy to work with when you think of it in terms of base-case and induction/normal case. An interesting thing is that a lot of principles and ideas can be transferred to other languages.

Transferable Knowledge

One example of a transferable idea is solving problems through composition of many small functions, this can be used in JavaScript (eg. using Lodash-fp or Ramda) quite easily. Composition promotes having many small functions solving simple subproblems, and does often result in cleaner code.

It doesn’t end here, Hindley-Milner type signatures might be worth to use in JavaScript as well, even if they aren’t used for more than documentation. Without them all the functions you end up with can be quite difficult to read.

Currying is easy to use in JavaScript (eg. with Lodash-fp or Ramda). I think I would go as far as to say that composition is not especially useful without curried functions.

It is important to be aware of differences between Haskell and other languages though. For example lazy evaluation is a quite unique feature of Haskell, another feature is tail call optimization, which means that you can use recursion without constantly worrying about your stack blowing up. I think there are a lot of other transferable learnings, but they are a bit deeper and you simply have to code Haskell to learn them. If you don’t want to walk the path via Haskell, for JavaScript you might find Professor Frisby’s Mostly Adequate Guide to Functional Programming useful. 

Final Words

I would like to encourage every programmer to experiment with different languages and concepts. It is easy to just use what is immediately required for your daily job. But you miss out on a lot of ideas from other languages and risk getting caught in a small bubble, hindering you from developing as a developer.

At Small Improvements we get to spend around 20% of our time doing other things such as fixing pet peeves and working on side-projects (for example this one). In addition to this we have hackathons and ship-it weeks. I would recommend every company to introduce these kind of events, because I don’t think I’m the only developer who would agree with that programming is way more fun when you keep learning new things and growing as a developer.

To be a good developer you need to keep learning and don’t be afraid of not being instantly awesome when picking up something new. Keep exploring the beautiful world of coding!

More than just slinging code

I like to think of myself as a coder of convenience. I often tell myself this work is just a means to an end – I am only coding till I can afford to spend the rest of my days lying in a hammock, drinking out of a coconut. But that’s not really true. If it were, I would be pursuing elite consulting gigs, or jumping from start-up to start-up on the hunt for unicorn stock options. Instead, after 15 years in the industry, I am working at a small, customer-focused company that turns a respectable profit building unpretentious B2B software.

The other day the owner, Per, asked me why I chose to apply for a job at Small Improvements. There were several reasons – not least of which was actually being able to talk about this sort of thing with my CEO – but I think my main motivation was because I wanted to help people again.

Continue reading

Creating The Right Knobs

I recently participated in Softwareskills’ Liar’s Dice competition, and since people have expressed interest to hear about how I managed to win, I decided to summarize the process and results.


As I prize I got 500SEK at Teknikmagasinet (Swedish store), a USB memory and this nice piece of paper 🙂

Liar’s Dice and the Competition

Liar’s dice is a game where each player starts with six dice. One player starts by announcing a bet. A bet consists of a number and a face, for instance four fives. The next player can then either challenge the bet or raise the bet. You can read about the details at https://en.wikipedia.org/wiki/Liar%27s_dice. Softwareskills’ made a competition about writing the best AI. I noticed that it is not clear to everyone what an AI is in this context, and how to get started writing one. So let’s discuss that briefly.

Simplified View of an AI

An AI (actually Intelligent Agent [IA], but I’ll continue to call it AI) in its simplest form can be seen as a program that given a state responds with a valid action. A good AI must, in addition to producing valid actions, produce the best valid action given a state. What best means is the real challenge which typically requires mathematical analysis and creativity to figure out. In some cases you might calculate what the best action is, but it might not be easy given that you don’t know how your opponents behave. In the case of Liar’s dice you don’t know your opponents’ dice and they might behave differently depending on what dice they have. Let’s have a look at how I approached this challenge.

The Development of My AI

First off, I’m not developing AIs on a day to day basis. I had some experience from before, but not much. I started by drawing the “Simplified View of an AI” on a piece of paper, this made it easier for me to break the problem down into the main components:

  1. Generate all valid actions given a state
  2. Put a score on all valid actions
  3. Pick the valid action with the highest score

Let’s look at each problem separately, starting with how to generate valid actions.

Action Generation

To generate actions you need to create a function that given the current state and the rules of the game can produce all valid actions. This is pretty boring work that I wanted to get through as quickly as possible to move on to what could actually get me a good place on the top list. Softwareskills’ provided a model with a track that has 27 positions. A die on a position corresponds to a specific bet with the face of the die (some positions require the bet to be a star, where star = six). Now, either the bet can be challenged or raised. A raise can either be of a “higher” face on the same position or any face at a later position. So to generate valid actions I separated the problem into the following sub problems:

  1. Find the position of the current bet on the “track”
  2. Given the position and face:
    1. Get all bets with a higher face on the same position
    2. For all subsequent positions, generate bets with all possible faces

This allowed me to quite quickly generate all possible actions. Now, to value all the possible bets, provide them with a score, I needed to revisit some basic statistics.


When I first saw the competition I thought it was just a matter of calculating the optimal action using statistics. I asked myself whether I was more likely to win if I challenged the previous bet or if I raised the bet, and if I raise, how much should I raise the bet. Based on these questions I created new questions which I could directly “solve”, namely:

  1. Given my known dice, and the number of unknown dice in game, what is the probability of the previous bet to be true?
  2. Given my known dice, and the number of unknown dice in game, what is the probability of my raised bet to be true?

Simple was my first thought before I actually got started. I was easy to come up with a solution, and it could even work decently. The only problem was that it was wrong. And to reach the top of the toplist the number of afforded mistakes is pretty low. It was not easy to figure out whether the statistics was properly calculated or not. When I was stuck at a score far from the best, I challenged my code by doing static analysis, which in this case meant verifying the mathematics. I realized, by reasoning, that it was wrong. This led me to reach for my old book from school (Introduction to Probability and Statistics: Principles and Applications for Engineering and the Computing Sciences by J. Susan Milton, Jesse Arnold) as well as to look at a couple of lectures at Khan Academy (https://www.khanacademy.org/math/probability). After a couple of hours studying combinations, permutations and the good old “number of favorable events / number of total outcomes” as well as experimenting with simple examples, I managed to get it right. I jumped to the top but not to the first place. Something was still missing.

Getting Desperate

I wanted to go for a clean solution based on simple mathematics. But analysing the games made it clear that my AI was often put into bad situations, where no action was particularly good, and that it challenged the bet too often. It was not really playing the game, it was still just following the rules but a bit more intelligently. I started to add a lot of knobs now. For instance weighted averages and factors and terms of various kinds. I created an massive amount of mathematical functions whose graphs looked approximately as I wanted. Many linear (y=kx+m) as well as some polynomials of a higher degree. I experimented by taking into account everything I could think of, such as if I have a larger or smaller part of the game than the previous player or next player, if the game just started, the previous bets of other players and many other things. The factors could influence the score of an action. But I didn’t know how much they should do so. So I created knobs, factors that I could adjust to make the impact higher or lower. Then I tried to find the optimal values for the knobs.

Function used for deciding how much to trust other opponent’s bets. Eg. the second bet is trusted to 100%. Note that this function was combined with other functions to calculate the final trust factor.

I basically created a big optimization problem which I only could try to solve empirically. I did this in iterations. I created a lot of knobs, and realized that I couldn’t grasp what I actually did. Then I removed all of them and tried to create a new set of knobs. Repeated this a couple of times until I finally overtook the throne, meaning I finally conquered first place on the toplist. Even if I wouldn’t have won, I feel that this kind of competition is very valuable. To share why I think so I will briefly mention some things I learned from participating.

Lessons Learned

When I started the competition I didn’t even know there was a prize. This was just a nice bonus. I entered the competition for the challenge. I was sure that I would learn something. Afterwards I think my biggest gain is refreshing statistics skills. It’s quite fascinating how easy it was to do so despite not having worked with it for a long time. I also learnt that it is important to carefully measure the results when you have knobs, or parameters, that you can adjust. With the risk of using too big words, I would say that it taught me to be a slightly better scientist by clearly showing the necessity of careful measurements when empirically looking for optimal parameters. From a software engineering point of view it made it even clearer that correctness of the core is very important, and just because something seems correct it might still have bugs in it. Proper testing of the basic units (function in this case) is very valuable. And just because your solution is based on mathematics it doesn’t mean it will be right or wrong. It might still be slightly wrong.

Furthermore, I learnt the value of keeping the code in a at least decent state at all time, to not hinder oneself from experimenting due to the effort being too high. And finally, I learnt the value of to kill your darlings over and over. Even if you are at second place, when you’re stuck, start over and just keep the generic core which lets you build working solutions quickly. Don’t stick to your good solution just because you feel that you invested so much time in it. Challenge it! You managed to come up with it once, and probably learnt from it, so the next time you might come up with something even better.