A Facelift for the Font Family

lucasarundell Design , , , , , ,

Today we’re excited to announce a new makeover for the Small Improvements application; a new font family! Please welcome “Avenir Next”!

Avenir Next - our new font

newfont1
Sample Objective within Small Improvements

This is all part of our mission to create a more enjoyable, engaging and enticing experience for  Small Improvements users. The Design Team @ Small Improvements has had a big year; going responsive, updating colours, icons and badges, and now for the final Christmas treat, we’ve introduced the smart, the elegant, the distinguished style of Avenir Next.

Watching our Weight

Part of the new font release has also included making better use of font weights (eg. Regular vs. Bold) – giving you a clearer view of what’s important on the page, and making it easier to scan the page with your eyes to find the right things.

Keeping it Uniform

In the past, the default SI font varied across devices. That’s because we used ‘system fonts’ only, so some users had Helvetica, some Arial, some had which ever default ‘sans-serif’ font their device had.

Now all users will share the same experience across all devices.

screen-shot-2016-12-12-at-15-55-35
Sample Message within Small Improvements – with new typography

 

 

Looking Back at GOTO 2016

Peter Crona Conferences

By Peter Crona and Michael Ruhwedel

goto

First of all, it was an amazing conference as always. None of us presented this year, but look for us in the future. Many of us at Small Improvements tend to go to more specific conferences, such as React Europe, DockerCon or JSUnconf. GOTO is more of a generic software engineering conference, focusing on issues such as architecture, security and new trends in the field. It doesn’t go as deep into the topics as the specialized conferences, but it serves well to give an overview and an introduction to interesting topics. Some of the most interesting and most popular topics were, as expected, microservices, data science, security and ethics. Let’s start with microservices.

Microservices are the Future

Something interesting about the future is that it is also always in the present, just initially hiding a bit in the corners. A clear message from Mary Poppendieck was that microservices are the future. Regardless of whether we want it or not, we need to learn it and will eventually use it.

Susanne Kaiser from Just Software talked about their ongoing journey from a monolith to microservices. She warned us from doing too much at once, but concluded that going from a monolith to microservices was worth it in the end. She also told us about the importance to not underestimate the effort required to do so. Later on Ilya Dmitrichenko walked us through Socks Shop, a demo application to show how an application built with microservices can look. He also showed us how a microservice-based application is deployed.

I urge you to read up on microservices if you haven’t. It is truly fascinating how convenient the configuration is nowadays, and if you’ve been around for awhile, you will find it interesting to compare with how we did it in the good old days. Have a look at this configuration for example, lovely, isn’t it? Let’s move on to another topic, which I have a very strong interest for, namely data science.

Seeing into the Future

It is truly fascinating how quickly data science has become popular and advanced. One of the first talks I went to was “Applied data science and engineering for local weather forecasts” by Nikhil Podduturi from Meteogroup. He took us through how they started using machine learning, running everything on their own laptops and then moving into the cloud. He showed us a bit of their architecture that process more than a terabyte daily. I enjoyed his talk very much and had a chat afterwards, in which he pointed out that, when getting started with data science, it is sensible to start with the basics, learning/repeating the mathematics and then move on to hot techniques such as deep learning. This will make it easier to develop an intuition for which technique to use when and how to find the best parameters. He recommended using Python since it has a very mature ecosystem for machine learning.

Robert Kubis from Google tutored us in Tensorflow, by working the Hello World of machine learning, namely classification of handwritten digits. He pushed the success rate of a neural network, up to an impressive 98%, while touching the basics of the Python API. This was a very interesting and hand-on talk, showing you how to use Tensorflow and giving you an introduction to deep learning.

How to find insights without using machine learning, was the topic of Michael Hunger from Neo Technology talk. He demonstrated how data can be modelled and queried using a graph. His talk focused on how Neo4J was used by journalists to analyze the panama papers.

Even your code repository is a datasource in itself that can be mined. This concept was presented by Dr. Elmar Juergens. By coloring new additions of code and test-coverages of functional tests, he clearly demonstrated that a dev- and a test-department at one of his clients had a serious communication problem: There was little overlap in what was tested and what was newly implemented.

The last two talks about data science were focusing a bit more on possibilities, philosophy and ethics. “Deep Stupidity: What Neural Networks Can and Cannot do …” by Prof J. Mark Bishop discussed about whether we can build general intelligence or not. “Consequences of an Insightful Algorithm” by Carina C. Zona focused on the importance of thinking through the ethical aspects when developing algorithms and using them. We are giving a lot of power to algorithms, and algorithms tend to reinforce prejudices and do not necessarily care about what is right, but are still used to make decisions that affect people’s lives. Let’s now have a look at the security talks.

A Secure Internet

When you learn a new concept, such as microservices, it is important to read up on security. It is easy to make mistakes that introduce vulnerabilities when you are new to technologies. Phil Winder talked about how to make your microservices secure. He was very practical and showed us common mistakes people do, such as running as root in containers and not setting up a sensible network policy. Dr. Jutta Steiner introduced us to Blockchain technology. She pointed out how we can use techniques from safety critical systems development, such as N-version programming, to securely implement it and minimizing the risk of bugs. The talk was unfortunately not going into implementation details of blockchain technology itself, but she made it clear that the technology can be used for much more than just a currency such as Bitcoin. Finally, let’s have a look at the ethics focused talks.

Ethics in Technology

The great thing about goto is, that it’s not got the latest technology topics covered, but also how to better get along with your fellow human beings.

Jamie Dobson encouraged us to think beyond capitalism in his inspiring “Postcapitalism” talk. It’s possible that the  power of 3D printing small and large can bring back the capital and onshore work in developed countries again.

Beginning with a short meditation Jeffery Hackert build a compelling argument for giving our full presence. With a full awareness of ourselves and our workplace come better informed observations, decisions and implementations. After all if you’re ever involved in a trolley problem, it would be really unfortunate if you’d be focused on your cellphone and not the lever.

If you’ve been exhausted by office politics Kate Gray and Chris Young can help you. Their great talk “How to Win Hearts and Minds” is about how the finesse of real world politics were used to push a blocked IT project to success.

Talks ranging from microservices to ethics shows you the great variety offered at GOTO, the conference really has a lot to offer.

Something for Everyone

Let’s end with some words about the conference itself. GOTO has five different tracks and the mix is very good, covering important and trending topics such as architecture (in particular microservices), security, data science and much more. In addition to this you find plenty of interesting people there to share ideas and pain points with. My only disappointment was that there was not a single talk about functional programming. But hey, you can’t fit everything into one conference.

Using Haskell to Find Unused Spring MVC Code

Peter Crona Haskell, Programming

Screen Shot 2016-12-02 at 14.43.09.png

Not into reading text? Click here for the code.

Like a lot of people at Small Improvements I’m fascinated by functional programming. After coming back from our company trip in San Francisco I had trouble beating jet lag due to spending the evenings reading about monad transformers, I’m not kidding, it actually kept me awake.

For a while I’ve been thinking about cleaning up a little in our codebase, mainly the backend which is written in Java. I have known for ages that Haskell is really good with abstract syntax trees (ASTs) and was playing with the thought of creating a Haskell tool that would help me with this. However, to not completely violate the “do not reinvent the wheel” rule I first had a quick look at what’s already out there.

Finding An Existing Tool or Building My Own

Most of the developers at work use IDEA (for editing Java) which has built in tools for finding unused code and do all different kinds of code analysis. I tried using it for finding unused code a couple of times with different settings but didn’t manage to get acceptable results. The number of false positives was way too high for it to be useful, in addition to this it was incredibly slow. I also tried Findbugs without satisfying results.

I’m sure it’s possible to configure some existing software, but rather than spending more time finding a COTS-tool I figured I might just code it myself. I was thinking that if it’s specific to our project it shouldn’t be so hard. I quickly realized regular expressions wouldn’t be enough or would be very tricky to use and limit my flexibility. This left me with the choice of writing a custom parser or building a proper AST and work with that.

I have bad experience of working with ASTs in Java, but Haskell is another story, traversing a tree is a piece of cake. I had a quick look at Hackage and noticed that someone already has written a parser for Java in Haskell, so it was settled, I was starting Small Improvements’ first, albeit small, Haskell project. Finally I got to use Haskell at work!

My Solution For Finding Unused Code

It is actually quite simple to find unused Java code. Let’s have a look at my solution. In essence I’m reading all the .java-files in a folder, building an AST using language-java and then traversing the AST to collect information that can later be used to decide if a file is used or not.

The main information I’m looking for is whether any other file imports a file. However, since Java does not require an import statement if the dependency is within the same package I also look for other things such as method calls. After this I’m using the information to actually find unused files.

To find unused files I’m building a graph. Nodes are files and an edge means that a file is used by another file. So the challenge here is to actually add an edge every time a file is used. An obvious thing to do is to add an edge for every import statement.

To improve the result further I’m adding edges for references within a package, eg. used classes or methods within the package. However, this is not enough since Spring MVC has a powerful dependency injection system. It supports injecting dependencies and still only relying on interfaces. You can get all classes of a type (interface) injected or one specific instance but still only depending on its interface.

When harvesting the AST I also collected autowired classes and superclasses. Using this I filtered out files that are autowired, either directly or via an interface. The result is not 100% perfect, but with a small blacklist of classes and some other trivial filtering I managed to make it good enough for it to be very useful. Everything I get from the AST is modeled using the following data structure:

data Result = Result { fileName :: String
                     , imports :: [String]
                     , references :: [String]
                     , topLevelAnnotations :: [String]
                     , methodAnnotations :: [String]
                     , implements :: [String]
                     , autowired :: [Autowiring]
                     } deriving (Show)

Have a look at the code and try it on your own Spring MVC project. Feel free to comment here if you need help or have suggestions of improvements. Let’s now compare coding Haskell with Java / JavaScript that we normally do at Small Improvements.

Reflection of Development With Haskell

I’m a big fan of Haskell and have been for ages. One of the first things I noticed is the wonderful support you get from the compiler. When the compiler blesses your code it is very likely to just work. Once you have established that your code works, that it behaves correctly, then it is really difficult to accidentally change its behavior when refactoring. You might break it, as in making it not compile, but once it compiles again it is very likely to behave like before.

Composition is just beautiful. It strongly promotes breaking your program into trivial pieces and then glueing them together. Types are excellent documentation, the type signature together with the function name often makes it easy to guess exactly what the function does. It’s easy to write relatively clean code in Haskell. I think that the pureness and composition of small functions almost automatically makes it happen.

Actually, in Haskell it is a bit difficult to write functions that are hundreds of lines of code doing many different things. In Java or JavaScript that is what many people begin doing, and something they only unlearn as they become more skilled. I think that it is possible to produce nice code in all languages, but Haskell does help you quite a lot to keep your code nice, not to mention hlint. Haskell does not guarantee that you produce good code though, let’s look at some of my learnings from this project.

Learnings From This Project

One thing I learned is that type aliases are very useful, you should use them whenever it makes your code more readable. Comments are in general not needed if the type signature and function name is good.

Naming your code increases readability, for example extracting out small pieces of code to the where clause of a function or simply making them top-level functions in the module. Putting too many functions that are relatively complex in the where clause is a bad idea, because you lose the explicit type signature (you should always specify it for top-level functions) which makes it difficult to directly understand when they can be used and how they can be combined. A small example of a nice usage of the where clause is:

transformToEdges :: Result -> Node
transformToEdges r = (r, fileName r, outgoingEdges)
  where outgoingEdges = references r ++ imports r ++ implements r

Note the increased readability in the top level expression. The where-clause is used to hide the messy details of what outgoing edges are behind a simple name. By using where it is often possible to make the top level expression very easy to read.

Curried functions are just awesome, they make it possible to compose almost any function. A good way to design them is to think of functions as being configured and getting what they operate on as the final argument.

Lazy evaluation is powerful, I still need to practice how to leverage it fully, but it is important to be aware of it. For example in my case I ran into problem when reading all files lazily. This caused my program to have too many open file handles. It was easily solved though, by hacking a bit to force the complete file to be read directly:

readFileStrict :: FilePath -> IO String
readFileStrict path = do
  file <- readFile path
  _ <- evaluate $ length file
  return file

Recursion further promotes clean code (small functions) and is quite easy to work with when you think of it in terms of base-case and induction/normal case. An interesting thing is that a lot of principles and ideas can be transferred to other languages.

Transferable Knowledge

One example of a transferable idea is solving problems through composition of many small functions, this can be used in JavaScript (eg. using Lodash-fp or Ramda) quite easily. Composition promotes having many small functions solving simple subproblems, and does often result in cleaner code.

It doesn’t end here, Hindley-Milner type signatures might be worth to use in JavaScript as well, even if they aren’t used for more than documentation. Without them all the functions you end up with can be quite difficult to read.

Currying is easy to use in JavaScript (eg. with Lodash-fp or Ramda). I think I would go as far as to say that composition is not especially useful without curried functions.

It is important to be aware of differences between Haskell and other languages though. For example lazy evaluation is a quite unique feature of Haskell, another feature is tail call optimization, which means that you can use recursion without constantly worrying about your stack blowing up. I think there are a lot of other transferable learnings, but they are a bit deeper and you simply have to code Haskell to learn them. If you don’t want to walk the path via Haskell, for JavaScript you might find Professor Frisby’s Mostly Adequate Guide to Functional Programming useful. 

Final Words

I would like to encourage every programmer to experiment with different languages and concepts. It is easy to just use what is immediately required for your daily job. But you miss out on a lot of ideas from other languages and risk getting caught in a small bubble, hindering you from developing as a developer.

At Small Improvements we get to spend around 20% of our time doing other things such as fixing pet peeves and working on side-projects (for example this one). In addition to this we have hackathons and ship-it weeks. I would recommend every company to introduce these kind of events, because I don’t think I’m the only developer who would agree with that programming is way more fun when you keep learning new things and growing as a developer.

To be a good developer you need to keep learning and don’t be afraid of not being instantly awesome when picking up something new. Keep exploring the beautiful world of coding!

Onboarding your Team to React + Redux

Robin Wieruch JavaScript

react redux team

Eventually the time will come when your team wants to use React + Redux for their frontend stack. We made that commitment some time ago at Small Improvements – we never had to regret it. As we come from an Angular 1.x frontend application, we needed to decide between React (+ ecosystem) and Angular 2. Staying with Angular 1.x was no option for our three teams. We saw too many benefits in other solutions like React e.g. to embrace functional programming. In the end we decided to go all the way with React + Redux, since most of our developers used it already in their side projects.

The article should give other teams or even companies some learnings and insights to have a smooth onboarding to the React + Redux ecosystem. While some learnings are applicable to React + Redux, others might be general insights about migrating to another technology.

React before Redux

In the beginning not everyone is familiar with React and its ecosystem. Give people time to understand, to experiment and to exchange thoughts. Introduce React without a state management library. Make use of the React lifecycle methods and internal state management (setState) to teach React itself. After a while you will want to introduce a state management library. It’s easy to get around an external library in smaller side projects, but it isn’t when you are contributing to a larger code base. In general don’t introduce a state management library when you don’t know the problem it solves for you.

Introducing Redux

In Angular state management got messy for us. Sharing state between components, watching state changes of components and services, storing server data – it was pretty soon a chaos. We knew the flaws of state management in Angular when the Flux pattern evolved. Perhaps that’s the cause why so many developers at Small Improvements got hooked by React + Redux eventually.

Don’t overengineer Redux

Once you started using Redux as state management library, don’t overengineer it. In the beginning the Redux ecosystem itself can be overwhelming. Moreover the way you deal with state management in Redux is different than what a lot of people are used to from the past. Again, like in React, give your team a chance to understand Redux. Not everyone will be already familiar with the overarching functional programming principals behind it.

Already one little library like redux-actions masks the excellence of Redux to stick to vanilla JavaScript. In redux-actions people might never experience plain actions and action creators. The functionality createAction camouflages some basic usage of Redux. Same goes for the handleActions helper function. People might never experience a plain Redux reducer which itself is plain JavaScript reducer function.

We love to use these little enhancements in Redux, but we made the mistake to introduce them too early. Nowadays it’s easy to npm install a package. Speak with your team about new packages. Don’t take from them the opportunity to learn it themselves.

Understand it, before you use it

Learn React before using Redux. In react-redux you use the connect functionality to literally connect the Redux store to your React components. The Provider component at the root level makes sure that the store is passed as context to the underlying components. Thus you can retrieve the store state in mapStateToProps and pass dispatchable actions on the store to your components in mapDispatchToProps.

But what is the connect doing there? It’s too easy to simply call it magic. When learning React before using Redux, you can make sure that everyone in your team understands the concept of higher order components (HOC). After that everyone has the chance to reproduce the underlying mechanics of connect in react-redux. Eventually everyone is aware of the hidden Redux store in the React context.

Asynchronous Actions

We decided to use redux-saga as we had to introduce asynchronous actions. In our case some people already used it before and felt that it is a great match. After some time using it, we don’t regret to handle our side effects with generators.

But what is the best approach to introduce asynchronous actions? Not everyone is aware of generators after all and it might add yet another level of complexity. Redux-thunk is a great way to begin with asynchronous actions. It allows you to dispatch delayed actions. When the whole team feels comfortable with asynchronous actions, you should decide whether you want to experiment with another solution. The way to deal with asynchronous actions should be a recurring discussion topic until you make the final decision. Otherwise you will delay the decision and end up with bigger refactorings of your asynchronous layer.

Normalize your data?

We don’t normalize our data, even if we were aware of the aspect to keep a flat state in Redux. It was no unconscious decision to keep a deep nested state. Since we already have a large Angular 1.x application, we are used to most of the data structures. In our case it would expand the gap between the two worlds, because we would have to get used to two different data structures. Once we introduce new data structures, we keep the state flat from the beginning. We are not sure yet whether it was a bad (unconscious) decision. Still we feel comfortable to keep our deep nested state immutable by using ES6 spread operators. Moreover reducer tests with deep-freeze help to ensure immutability.

Feature Folders

In the beginning we had a technical folder separation. Everyone is used to it from React + Redux tutorials. We had folders for reducers, actions, components, constants etc. Very early we noticed that the approach would never scale with independent teams. We decided to have feature folders. Now we have packages with clear boundaries. Take for instance a Table component package:

--Table/
----index.js
----components/
------Table/
--------index.js
--------container.js
--------presenter.js
--------style.less
------Cell/
--------index.js
--------presenter.js
--------spec.js
------Row/
--------index.js
--------presenter.js
--------spec.js
----ducks/
------index.js
------filter
--------index.js
--------spec.js
------sort
--------index.js
--------spec.js
------select
--------index.js
--------spec.js

One index file gives an entry point to each package. The ducks index file still exposes all necessary action creators and reducers. When another package wants to dispatch a Table action, it has to import it from “Table” and not from “Table/ducks”. The package has clear boundaries.

Ducks Everywhere

“I saw you are using ducks?” Yes. We decided to use them when we introduced feature folders. The advantage is to have everything in one place. But once you introduce ducks, you should decide on best practices to keep the duck files tidy. Standardize your naming for reducer and action creator functions to distinguish them.

Moreover we noticed that ducks don’t scale very well for us. The lines of code grew very fast. That’s why we decided to split up the ducks responsibilities in smaller domains, like you can see in the ducks folder for the Table in the example above.

What about boundaries to legacy frameworks?

It’s silly that we call Angular 1.x already legacy, but that’s JavaScript today. Still we had to figure out how to connect both worlds.

ReactDOM.render() is all you need to have a React component tree in Angular. Moreover you can simply use the react-redux Provider component to pass the imported Redux singleton store as context to the component tree. You can dispatch actions on the store and get the state from everywhere, since you only need to import the store in your non React world.

The other way around we use a helper to render Angular components in React. Once you have a large code base with complex non React components, you can’t easily rewrite them all at once. That approach ensures us a stable migration from Angular 1.x to React. We can still reuse Angular components. Once we refactor one component from Angular to React, we can easily exchange the component in one place.

What about a synced cache to the legacy framework?

In the beginning we experimented with Relay to facilitate caching of our backend data. Even more we had attempts to make Relay independent of React to use it in Angular as well. But very soon Relay felt like a foreign object in React to us. We stopped the experiment to use Relay + GraphQL and remained with using our RESTful solution.

Still we had to figure out how to cache the server side data in our single page React + Angular application. Since we already used an own store architecture in Angular, we synced the stores to the Redux store. Everything we implement in the future uses the Redux store, but our old Angular pages still get the cached data from our store architecture.

Moreover we introduced ladda to cache requests to our API. It’s an in-house solution by one of our developers, which will get open sourced properly. Ladda introduces a JavaScript data fetching layer with caching without dependencies. You can easily make requests in Angular or React.

Hack & Tell

You read a lot about giving people time to understand the ecosystem properly. Your whole team is sitting in the same boat when introducing something novel. Everyone tries to accomplish a scalable and maintainable code base in the new ecosystem. At Small Improvements we are having weekly Hack & Tells to exchange our recent gatherings. We share learnings to get a mutual understanding of doing things in React + Redux. In general those Hack & Tells don’t apply necessarily to one technology.

Knowledge you could exchange in a weekly Hack & Tell:

  • best practices
  • patterns
  • decisions like naming, folder structure etc.
  • reusable components / feature packages
  • new npm modules which solve a real problem in your code base
  • recent pull requests

Perhaps once a week isn’t even enough to exchange knowledge in a whole new ecosystem. Our code base is scaling well, even though we feel that we could refactor all the time. We don’t regret the step to migrate from Angular to React and its ecosystem.

Building the Activity Stream. Part 1: Product Design

Per Fragemann How we work, UX

We recently launched our new Activity Stream. What began as a small idea turned into quite the large engineering undertaking. The results are astonishing, but it was a lot of hard work. In this part we’ll share how the design took shape.

 

Whiteboard and paper drawings

Each larger project at Small Improvements begins with a rough written draft, and a kickoff meeting that involves PM, developers, designers and a customer-team person. User stories are extracted, prioritised and then broken down into manageable MVP plus one or two “expansion releases” so it’s clear they are not part of the original release. Then the product design work starts (and if needed, we’ll conduct  some code exploration too to rule out major obstacles)

We use whiteboards and paper drawings to try out ideas, discuss and iterate quickly. Here are some of the many initial ideas for how to represent complex concepts like our cycles in a changelog:

as-paper.jpg

Wireframes

Lots of future problems can be avoided by coming up with good quality wireframes. So after the paper scribbles were done, we went to paint proper wireframes. Here are two examples of how we described the overall architecture of the activity stream, then followed by an actual sample wireframe.

wireframe 1.png

wireframe 2.png

 

We don’t take wireframes too seriously of course. Wireframes are a plan, not a committment to the future product, and everything is still up for change.

Also, quite often the real challenge lies in the user flow, not just in screen design. Some flows can be simulated at the whiteboard or by moving paper pieces around, but we typically move to InVision quickly. Using a web-based tool allows us to create actual clickable prototypes, and get insight from other (possibly remote) staff easily.

 

Mockups

A high fidelity image can raise unrealistic expectations (“oh look we’re almost done”) or draw attention to the wrong details (“why are there 20px margin, shouldn’t it be 30px?”). But on the other hand, the more realistic our mockups look like, the more and better feedback we get from outside the core team.

It’s not about pixel perfection at this stage, everything can still get adjusted. But it is about making things feel “real” sooner than later, so that staff feel encouraged to give feedback about the flows, and so that early access customers feel they are actually commenting on something we are serious about. So we quite like our high fidelity mockups!

 

mockup-comment.png

(The above is just a screenshot, but we typically make mockups clickable so it’s clearer what action takes you where)

Code level prototypes

We prefer to not write lots of “real” code before there has been some agreement on the flows between screens. Yes we’re agile, and yes we can make changes down the track. But starting to code user flows without first sketching them has failed us too often.

So once we start coding the frontend, we have a somewhat good grasp of a feature, and this makes coding quicker, and styling is also more fun when you don’t have to fear everything changes totally next week.

As designers we love to get down and dirty at the UI level too. But it’s not scaleable to be positioning every button ourselves. We invest time teaching design principles to frontend developers, and make everyone follows our living style guide, so we can focus on the advanced stuff, not on placing basic buttons.

 

User testing

Once a feature has been developed to alpha stage, it’s time for user testing. We aim for conducting some some 5 to 10 “successful” user tests for larger features or changes. If user tests show there are major problems in our designs, then we reset the counter, keep adjusting the design, and schedule further user tests.

We prefer a combination of tests. On one hand we like watching experienced customers explore their existing content with new features enabled. On the other hand we like scenario-driven tests where we present users with concrete tasks, using a synthetic demo content setup (so people aren’t afraid to break things). Both have their place, but usertesting is such a complex beast that we’ll blog about it separately. 🙂

 

Launch

When the tests confirm that a feature works, it is time for the polishing phase, and we’ll gradually enable features to our beta program customers. After another round of feedback, answering questions and polishing, it’s time for a wider rollout to clients who didn’t sign up for the beta. Now proper documentation needs to be written, blogged, and announced in-app using Intercom or home-made hints.

And then it’s time to roll out for real and celebrate!

celebrate.jpg

 

Running our App Engine Application in the Flexible Environment (Java 8)

Michael Ruhwedel Google App Engine, Java

It’s no secret that we at Small Improvements love to use cutting edge technologies for our application. On the client side, there’s no limit, that’s why we’re rapidly transitioning to React. In the backend, we’re pushing the limits too, but we’re currently bound by what the App Engine has to offer. The main grievance for us is that we’re still using Java 7.

There are hints that Google will bring Java 8 to the App Engine, but during our recent Ship-It week, we decided to take matters into our own hands and run Small Improvements on a Java 8 Flexible Runtime, aka Flexible Environment or Managed VM, the name changes frequently ;).

If you never heard of the Flexible Runtime, it’s basically a Docker container that will run your App Engine application. To get started quickly, you can use Google’s Java 8 / Jetty 9.3 Compat Runtime container without touching (or even seeing) any Dockerfile.

While Google provides a couple of Hello World examples, this won’t help you much when your app won’t start and you can’t figure out why.

If you’re like us and prefer to use the Cloud SDK to deploy over Maven, please read on and I’ll show you how we managed to get our app running.

Caveat: It’ll work, but it’s definitely not quite production ready. We wouldn’t recommend it for your main app, but if you have a non-mission critical service,  you could give it a shot.

Bye bye XML! Hello YAML!

XML was quite nifty when it was introduced 20 years ago. But YAML is so much easier on the eyes.

Lucky for us, the Flexible Runtimes are configured by YAML files. You can generate them from your exploded App Engine project using appcfg.sh which is included in the Java SDK:

appengine-java-sdk/bin/bin/appcfg.sh stage\
   your-exploded-app stage-directory

Have a look at the generated YAML files: Cron, Dispatch, Dos, Index and Queue. They should all be deployable and contain the exact same configuration as their XML counterparts.

To get app.yaml into production it requires some additional steps …..

App.yaml and its gotchas

Static files … or not?

Our generated app.yaml was a bit crude and yours might be too. For us, the static files and their expiration settings were very verbose:

- url: (/resources/.*\.jpg)
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  login: optional
  secure: always
  expiration: 5d
- url: (/remote_api/.*\.jpg)
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  login: optional
  secure: optional
  expiration: 21d
- url: (/api/tasks/.*\.jpg)
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  login: admin
  secure: optional
  expiration: 21d
...

You’ll notice a lot of duplications. In our case so many, that the deploy failed since there is a hard limit for the number of entries :D. But no worries the handler syntax supports regular expressions.

So for example, you can configure the serving/caching of your and fonts and images in a single handler:

- url: (/.*\.(ttf|eot|svg|woff|gif|jpg|png|ico))
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  secure: always
  expiration: 21d

It’s unclear if Google will support serving static files automagically in the Flexible Environment. Currently, they suggest that you upload the files to a cloud storage bucket and serve them from there.

Our hope is that this is only an intermediate step. Who knows, they are not so forthcoming with their roadmap 😉

What we’ve gathered by monitoring the logs of our deployed app: Currently Flexible Environment deployments ignore the static_files handlers. So whatever you write in the handlers your application will still serve the files.

Security Constraints

If you have security constraints for your Servlets/Resources, you’ve expressed them so far in web.xml:

  <security-constraint>
      <web-resource-collection>
          <url-pattern>/admin/tasks/*</url-pattern>
      </web-resource-collection>

      <auth-constraint>
          <role-name>admin</role-name>
      </auth-constraint>
  </security-constraint>

This won’t work in a Flexible Runtime. For us, it closed all the responses of the server unexpectedly. You can safely remove the constraints from this file and express them in app.yaml.

Here’s how the example from above looks in the app.yaml:

- url: /api/tasks/.*
  script: unused
  login: admin

Selecting the Runtime

The last missing piece is to actually configure your app to run in a Flexible Environment:

vm: true
runtime: java
runtime_config:
   jdk: openjdk8
   server: jetty9
threadsafe: True
resources:
  cpu: 4

Line 1 is the big switch that will let your app run in the Flexible Environment.
Line 2 will upgrade you to Java 8.
Lines 3-5 are optional, just in case you’d like to try different Java/Jetty combinations.
Lines 7-8 are specifying how powerful your compute engine machine is – and how expensive.

Check out Google’s documentation to  learn what other settings you can play with.

Cleaning up

Remove the XML configurations

Now that all your YAML files are ready, take off the training wheels and delete the following the XML configurations:

  • cron.xml
  • queue.xml
  • datastore-indexes.xml
  • dos.xml

Bonus (almost) get rid of application-web.xml

Whatever you’ve got in application-web.xml you can configure it in app.yaml now. Here are the only settings you’ll need to keep in there:

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <vm>true</vm>
    <threadsafe>true</threadsafe>
    <sessions-enabled>true</sessions-enabled>
</appengine-web-app>

Test Run

The Cloud SDK brings its own App Engine development server dev_appserver.py. You can use it to test your upgraded application in a Flexible Environment locally:

# install the dev_appserver.py
gcloud components update app-engine-python
dev_appserver.py stage-directory/app.yaml

If everything worked as expected, you’ll be able to access the development server.

Deployment (Fingers Crossed)

For our deployment, we choose not to use the Maven plugin from Google’s examples (who would after getting rid of so many XML files 😉 ).

You can elegantly use gcloud from the Cloud SDK to deploy:

cd stage-directory
gcloud\
  app deploy\
    --no-promote\
    --version=any-version\
    --project=your-project\
    app.yaml\
    cron.yaml\
    dispatch.yaml\
    dos.yaml\
    index.yaml\
    queue.yaml

Congratulations you’ve upgraded your application to Java 8 and a modern Jetty!

So I can use Flexible Runtimes, or what?

We’ve encountered a lot of errors before the deployment worked.

Sometimes the cloud build timed out. Or the generated app.yaml file broke the gcloud deploy. (Google support helped us patch the Python executable: Big thanks!)

The main problem we have is, that the deployment of our application – composed of two modules – is taking 15~18 minutes in the Flexible Environment. To put this in perspective: The regular re-build and deploy of our application is well below 10 minutes.

Also from the development perspective, we’re not ready to forgo the convenience of firing up a development server in IntelliJ Idea. The development server from the Cloud SDK is cool, but it would need some more tweaking to develop locally without a lot of restarts (read: too many 😉 ).

Conclusion

All in all, it was a fun and interesting project for us. It’s good to see that our application can run on the latest stable Java version.

The Flexible Environment is still in beta and NOT production ready. It’s NOT covered by any SLA.

We decided to let a lesser important microservice run in the Flexible Environment. It doesn’t require many redeploys and has been happily serving for two months. So far it only had the forgivable quirk of logging to standard error instead of the request log.

Nevertheless, don’t be discouraged. If our instructions worked for you, you’ll be ready when Google finally ships the Flexible Runtime … we know we are 🙂

Hack It, Ship It!

Robin Wieruch How we work

Small Improvements conducts Hackathons every few months which usually involves two days of hacking on an experimental project. Hacking doesn’t imply that it’s a “developers-only” affair either; other departments at Small Improvements like Customer Success and Marketing also get a chance to get experimental too. Hackathons usually give us a chance to really get creative about novel concepts and ideas for the product and company. It is about coming together as people to grow as a team – something that we at Small Improvements value as an important part of internal culture.

From Hackathon to Ship It Week

Being creative with innovative ideas is great, but how would we actually Ship It? We introduced the Ship It Week as new concept to extend our occasional Hackathons from 2 days of hacking to 5 days more serious hacking! In the first two days we had our regular experimental Hackathon, but afterwards had to decide which projects were feasible to “ship” until the end of the week. In the context of a feature in the Small Improvements feature, ship would mean: going live for customers!

Continue reading →