In the spirit of Pamela Fox’s post about her year at Coursera, I thought it would be interesting to talk about a large project I worked on for the past year (2013).

The specifics are under NDA, but there are still a lot of interesting details I can talk about, the journey, people, technology.

The Project

The project is dead.

I can’t say I’m surprised, the project was already in cold storage when I joined and most of the efforts were put into justifying its existence. That’s usually a bad sign.

Despite that, I had the opportunity to work with very smart people, enjoy a huge personal growth thanks to the situations and technologies I got to work with.

Situations such as dealing with internal client politics, problematic inefficient teams, but also meeting good people, go through a ton of user testing - not everybody likes it, but I love observing people and got to confirm and disprove a lot of assumptions, travel a bit and meet very colorful people.

In the end, it was a very enriching experience.

The project, btw, consisted of a mobile client created by a single team, backed by back-end services and web apps developed by multiple teams. Pretty generic description I’m afraid.

The Experience

It was amazing. Despite its ill fate, I hope I’ll have this much freedom, voice, experience when it comes to technology and design anytime soon.

Design

Experience design all-round. Although I didn’t got to design the whole product experience and vision as I would like to*. To a certain extent perhaps due to the immense friction the project faced.

• the project vision wasn’t years ahead of what’s out there but had the potential to become something very interesting.

Because design isn’t visual: the tools, deployment workflow can also be included in here, it was a joy designing them.

I got to design parts of the product and experience - consumption/content creation, architecture the client and back-end services, write a full API Documentation for a mixed REST & real-time platform with scaling considerations and intended to be used by thousands of people.

Designing the social component and web presence of a product was fun. Taking the partners into consideration so they know exactly what they’re getting out of providing you with data, how will the web presence help a mobile-first product gain awareness à la Instagram, and creating not a social layer, but actually provide value to the user by interweaving the social interactions with the soul of the product.

That’s all I can share design-wise.

Technology

Architecting

Both the Mobile Web App (MWA) and Back-end services benefited a lot from proper architecting.

The MWA was highly modularized, each module had a ‘public’ event-based API, mediated by the Application - the glue.

This allowed us to do drop-in replacement of data sources, mock modules. Helped with integration tests by plugging into the internal events and simulating real-time events, user actions, etc.

I’m a big believer in the UNIX philosophy and appreciate a lot the fact that Substack, Isaacs and many others in the Node community follow and promote it.

All of the server code I was responsible for was done in Node because it was the most appropriate tool to deal with the problems we needed to solve.

Services were composed of small libraries, we used Streams when appropriate, text as our interface. We kept it simple, scalable, SOLID.

We also architected a few REST and Realtime services, we naturally made it follow standards for encoding, dates, RESTful URL design, explicitness, discoverability, performance. Pagination, search were also included. (Documented using simple markdown, App.net’s documentation was a good inspiration)

Modularity and nature (web) of the app brought us an interesting advantage.

We needed to allow content curators to preview their work*, the presentation logic was relatively complex, re-implementation by a different team on a short schedule didn’t seem ideal.

• ideally, curation shouldn’t fall far apart from presentation, but I had no word in that and the team working on it was part of a traditional compartmentalized organization where developers are just that.

Fortunately, thanks to modularity and abstraction, we were able to leverage the implementation in the existing mobile web app.

We basically took the consumption screen and a few essential modules and configured a minimal version of the client that could be seamlessly inserted into the also web-based editor tool.

Continuous Deployment

The web client deployment was automated so that whenever we committed to one of the environment branches on GitHub (dev/prod/staging), the web client would also be updated with varying levels of checks - development was always deployed, staging and production only if the tests check out OK.

There were also special builds for certain occasions such as demo clients that would be deployed with configuration files pointing to mock services and a cherry-picked list of features.

The native iPad client was initially built with Phonegap Build, but we later switched to Travis-CI for added flexibility - an OSX machine using the Testflight Upload API.

The list of software/services involved include: Travis-CI, GitHub, Standard UNIX Tools (sed, curl, etc), Ruby, PhantomJS, CasperJS, sskeychain, S3, NodeJS, Testflight.

Monitoring and Alerting

Initially, Pingdom provided for some quick uptime monitoring, eventually we needed to access a higher level of detail and overview of the system status, so we ended up whipping up a Status Page for all the critical systems, segmented by environment and feature, modeled after iCloud’s status page.

Mobile Web Application JS errors were logged to BugSense, Objective-C to TestFlight - which was also used to distribute the builds. NodeJS errors were logged to Loggly.

Analytics

While we initially started with Mixpanel only, we eventually evolved to take advantage of an internal event firehose and proxied that to segment.io.

This firehose provided to its consumers a raw feed of ALL events occurring in the client apps. The event schema was similar to an Activity Streams event and included: device info (UUID, connectivity type, OS, App version), event payload and name, currently logged in user.

The information collected is a delicate subject (UUID, etc), but remember this was still in testing and development.

Tooling

This was required to ease the pain of creating new builds, managing configuration files, etc.

It was mostly composed of dependency-free, robust, portable shell scripts (unlike, say: Grunt), with a few NodeJS cli tools.

Phonegap

Since the client was a mobile web app, we used Phonegap to provide access to features such as power management, native storage which the client would use instead of local storage when running in development/browser environments.

We followed the approach of mocking all the external services and native apis whenever possible, so that we could develop the client in the browser, regardless of network issues and partner downtimes, and then had the mock modules replaced by real ones that would talk to real device APIs, services.

We also updated and created Phonegap plugins as required.

Objective-C

Features such as the In-App browser required customization. We also needed to use Objective-C to create new plugins e.g.: Phonegap Social plugin using the Social Framework.

NodeJS

An additional note on NodeJS: it was a joy to work with it: elegant and performant. Used libraries include: Winston, Resourceful, Socket.io, Express, Lodash, Consolidate, Handlebars, Vows, more.

CouchDB, Redis

For the services we developed, these were the best tools for the job. Again: keeping it simple and performant enough for the development phase of the product.

CouchDB is super awesome for prototyping (not only), its replication features were very helpful.

Redis was great for situations where a lot of dynamic content needed to be generated and served as quickly as possible. Again: simple (well…) and fast.

Nodejitsu

And Amazon. All of our prototype services were hosted on Nodejitsu, with static assets (web clients, media) being served off of S3.

We bumped into some growing pains on the Nodejitsu and IrisCouch side, but the team was always quick to fix.

It kinda led me to spin up and provision a server on a certain occasion, which was great ‘cos you don’t want to get rusty on that, gives you a skill set that can be used when hacking with the Raspberry Pi for e.g, and it’s close enough to also be useful to the daily shell routine.

Summary

A dead project is still a project. Enjoyed working on it, despite the frequent pains and having to deal with people aiming for just ‘good enough’, the paragraphs above are proof of the value I got from it.