Running our App Engine Application in the Flexible Environment (Java 8)

It’s no secret that we at Small Improvements love to use cutting edge technologies for our application. On the client side, there’s no limit, that’s why we’re rapidly transitioning to React. In the backend, we’re pushing the limits too, but we’re currently bound by what the App Engine has to offer. The main grievance for us is that we’re still using Java 7.

There are hints that Google will bring Java 8 to the App Engine, but during our recent Ship-It week, we decided to take matters into our own hands and run Small Improvements on a Java 8 Flexible Runtime, aka Flexible Environment or Managed VM, the name changes frequently😉.

If you never heard of the Flexible Runtime, it’s basically a Docker container that will run your App Engine application. To get started quickly, you can use Google’s Java 8 / Jetty 9.3 Compat Runtime container without touching (or even seeing) any Dockerfile.

While Google provides a couple of Hello World examples, this won’t help you much when your app won’t start and you can’t figure out why.

If you’re like us and prefer to use the Cloud SDK to deploy over Maven, please read on and I’ll show you how we managed to get our app running.

Caveat: It’ll work, but it’s definitely not quite production ready. We wouldn’t recommend it for your main app, but if you have a non-mission critical service,  you could give it a shot.

Bye bye XML! Hello YAML!

XML was quite nifty when it was introduced 20 years ago. But YAML is so much easier on the eyes.

Lucky for us, the Flexible Runtimes are configured by YAML files. You can generate them from your exploded App Engine project using appcfg.sh which is included in the Java SDK:

appengine-java-sdk/bin/bin/appcfg.sh stage\
   your-exploded-app stage-directory

Have a look at the generated YAML files: Cron, Dispatch, Dos, Index and Queue. They should all be deployable and contain the exact same configuration as their XML counterparts.

To get app.yaml into production it requires some additional steps …..

App.yaml and its gotchas

Static files … or not?

Our generated app.yaml was a bit crude and yours might be too. For us, the static files and their expiration settings were very verbose:

- url: (/resources/.*\.jpg)
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  login: optional
  secure: always
  expiration: 5d
- url: (/remote_api/.*\.jpg)
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  login: optional
  secure: optional
  expiration: 21d
- url: (/api/tasks/.*\.jpg)
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  login: admin
  secure: optional
  expiration: 21d
...

You’ll notice a lot of duplications. In our case so many, that the deploy failed since there is a hard limit for the number of entries😀. But no worries the handler syntax supports regular expressions.

So for example, you can configure the serving/caching of your and fonts and images in a single handler:

- url: (/.*\.(ttf|eot|svg|woff|gif|jpg|png|ico))
  static_files: __static__\1
  upload: __NOT_USED__
  require_matching_file: True
  secure: always
  expiration: 21d

It’s unclear if Google will support serving static files automagically in the Flexible Environment. Currently, they suggest that you upload the files to a cloud storage bucket and serve them from there.

Our hope is that this is only an intermediate step. Who knows, they are not so forthcoming with their roadmap😉

What we’ve gathered by monitoring the logs of our deployed app: Currently Flexible Environment deployments ignore the static_files handlers. So whatever you write in the handlers your application will still serve the files.

Security Constraints

If you have security constraints for your Servlets/Resources, you’ve expressed them so far in web.xml:

  <security-constraint>
      <web-resource-collection>
          <url-pattern>/admin/tasks/*</url-pattern>
      </web-resource-collection>

      <auth-constraint>
          <role-name>admin</role-name>
      </auth-constraint>
  </security-constraint>

This won’t work in a Flexible Runtime. For us, it closed all the responses of the server unexpectedly. You can safely remove the constraints from this file and express them in app.yaml.

Here’s how the example from above looks in the app.yaml:

- url: /api/tasks/.*
  script: unused
  login: admin

Selecting the Runtime

The last missing piece is to actually configure your app to run in a Flexible Environment:

vm: true
runtime: java
runtime_config:
   jdk: openjdk8
   server: jetty9
threadsafe: True
resources:
  cpu: 4

Line 1 is the big switch that will let your app run in the Flexible Environment.
Line 2 will upgrade you to Java 8.
Lines 3-5 are optional, just in case you’d like to try different Java/Jetty combinations.
Lines 7-8 are specifying how powerful your compute engine machine is – and how expensive.

Check out Google’s documentation to  learn what other settings you can play with.

Cleaning up

Remove the XML configurations

Now that all your YAML files are ready, take off the training wheels and delete the following the XML configurations:

  • cron.xml
  • queue.xml
  • datastore-indexes.xml
  • dos.xml

Bonus (almost) get rid of application-web.xml

Whatever you’ve got in application-web.xml you can configure it in app.yaml now. Here are the only settings you’ll need to keep in there:

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <vm>true</vm>
    <threadsafe>true</threadsafe>
    <sessions-enabled>true</sessions-enabled>
</appengine-web-app>

Test Run

The Cloud SDK brings its own App Engine development server dev_appserver.py. You can use it to test your upgraded application in a Flexible Environment locally:

# install the dev_appserver.py
gcloud components update app-engine-python
dev_appserver.py stage-directory/app.yaml

If everything worked as expected, you’ll be able to access the development server.

Deployment (Fingers Crossed)

For our deployment, we choose not to use the Maven plugin from Google’s examples (who would after getting rid of so many XML files😉 ).

You can elegantly use gcloud from the Cloud SDK to deploy:

cd stage-directory
gcloud\
  app deploy\
    --no-promote\
    --version=any-version\
    --project=your-project\
    app.yaml\
    cron.yaml\
    dispatch.yaml\
    dos.yaml\
    index.yaml\
    queue.yaml

Congratulations you’ve upgraded your application to Java 8 and a modern Jetty!

So I can use Flexible Runtimes, or what?

We’ve encountered a lot of errors before the deployment worked.

Sometimes the cloud build timed out. Or the generated app.yaml file broke the gcloud deploy. (Google support helped us patch the Python executable: Big thanks!)

The main problem we have is, that the deployment of our application – composed of two modules – is taking 15~18 minutes in the Flexible Environment. To put this in perspective: The regular re-build and deploy of our application is well below 10 minutes.

Also from the development perspective, we’re not ready to forgo the convenience of firing up a development server in IntelliJ Idea. The development server from the Cloud SDK is cool, but it would need some more tweaking to develop locally without a lot of restarts (read: too many😉 ).

Conclusion

All in all, it was a fun and interesting project for us. It’s good to see that our application can run on the latest stable Java version.

The Flexible Environment is still in beta and NOT production ready. It’s NOT covered by any SLA.

We decided to let a lesser important microservice run in the Flexible Environment. It doesn’t require many redeploys and has been happily serving for two months. So far it only had the forgivable quirk of logging to standard error instead of the request log.

Nevertheless, don’t be discouraged. If our instructions worked for you, you’ll be ready when Google finally ships the Flexible Runtime … we know we are🙂

Hack It, Ship It!

Small Improvements conducts Hackathons every few months which usually involves two days of hacking on an experimental project. Hacking doesn’t imply that it’s a “developers-only” affair either; other departments at Small Improvements like Customer Success and Marketing also get a chance to get experimental too. Hackathons usually give us a chance to really get creative about novel concepts and ideas for the product and company. It is about coming together as people to grow as a team – something that we at Small Improvements value as an important part of internal culture.

From Hackathon to Ship It Week

Being creative with innovative ideas is great, but how would we actually Ship It? We introduced the Ship It Week as new concept to extend our occasional Hackathons from 2 days of hacking to 5 days more serious hacking! In the first two days we had our regular experimental Hackathon, but afterwards had to decide which projects were feasible to “ship” until the end of the week. In the context of a feature in the Small Improvements feature, ship would mean: going live for customers!

Continue reading

More than just slinging code

I like to think of myself as a coder of convenience. I often tell myself this work is just a means to an end – I am only coding till I can afford to spend the rest of my days lying in a hammock, drinking out of a coconut. But that’s not really true. If it were, I would be pursuing elite consulting gigs, or jumping from start-up to start-up on the hunt for unicorn stock options. Instead, after 15 years in the industry, I am working at a small, customer-focused company that turns a respectable profit building unpretentious B2B software.

The other day the owner, Per, asked me why I chose to apply for a job at Small Improvements. There were several reasons – not least of which was actually being able to talk about this sort of thing with my CEO – but I think my main motivation was because I wanted to help people again.

Continue reading

Creating a SI Style Guide

After a significant project involving making Small Improvements responsive, we came up with some UI ‘rules’ that we in the design team or (‘UI Taskforce’) agreed upon. The longer we worked on this refactoring and ‘cleaning up’ of the app’s style, the more we realised the importance of (finally) having a Style Guide.

IMG_20160526_152845.jpg
The UI Taskforce! Adrienne, Kristof, Kolja and Timur. (I took the photo).

At Small Improvements, we often have the front end developers implementing UIs faster than we (the design team) can keep up, so it’s important that they can access this guide.

Continue reading

React Europe 2016

In the past, we at Small Improvements have sponsored various Angular Conferences and Meetups. So how did our developers end up at React Europe? Let’s backpedal to understand the story behind that.

Small Improvements and React!?

At Small Improvements we decided in early 2016 to shift towards React instead of working to migrate to Angular 2. Internally a lot of our developers had had great experiences with React and its ecosystem in their side projects. We wanted to keep up with evolving technologies rather than being stuck with Angular 1. Even though we already have a large frontend codebase in Angular, in the past 6 months we’ve also been able to implement new components and pages in React. We are confident that it was the right choice for the future. We also experimented with Relay + GraphQL, but it felt too premature to use these technologies at this point.

DSC03847

Continue reading

SI Process Hacks – The Internal Product Changelog

Development is hard. But communication is even harder – especially across departments and continents. Our SI Process Hacks series will highlight a few simple hacks  we wish we had known right from the start!

The problem

We deploy many times a day –  bugfixes, feature improvements, new options, changes in user flows etc. As a developer, you know when you committed, and you can see your commit go live in real-time:

Screen Shot 2016-07-03 at 13.23.56.png

But as the developer on another subteam, or especially as a non-developer, git-hashes are not really your preferred way of looking at things. It’s tricky to know when what change goes live on what day. Knowledge of changes is crucial, you don’t want to demo the product to a potential client and get caught off guard by that new button on a core screen.

We tried using Slack to keep people in the loop, we considered using a special Trello board, or including even tiny changes into our Release Notes. But everything was just complicated and never quite flexible either.

Continue reading

Introducing Development Team Leads

The challenge: Scaling

Our dev team has reached a quite impressive size these days! Unfortunately I just don’t scale to this size. I can’t possibly conduct 1:1s, listen to feedback, give feedback and help people grow in a flat team of 12 people. While I don’t code anymore these days, and have assigned most project management tasks to Feature Coordinators, lots of new responsibilities of a growing company are now on my plate. I just have to lean on others so that every dev team member gets the attention they deserve.

That’s why we introduced the concept of Team Lead two weeks ago.

Continue reading