Five Behaviors of Awesome Engineers

I recently joined TaskRabbit as VP Engineering and the experience has reinforced lessons learned from prior startups about the importance of having a systematic way to recruit, retain and promote great engineers.  One way to help recruit, retain, and promote more effectively is to identify and define a consistent set of behaviors that the company expects of engineers.  I’m constantly amazed at how many companies fail to do this for their technical staff.  

When it comes to defining explicit objectives or desired actions, it seems like many companies take the time to specify those kinds of expectations for marketing, sales, or product management team members.  But when I ask engineers at other companies how they know if they’re doing a good job, I often hear answers like: “we’re generally supposed to kick ass and ship code” or “we just do what PM’s tell us to do.”  IMHO, having a group of prodigious coders or friendly team-players on the engineering staff is a good start, but not sufficient.  And leaving engineers with ambiguous or undefined company expectations borders on neglect! 

I’d like to push more companies into defining clear, specific expectations for how they’ll assess engineers at all points in their career life-cycle — from being pitched as candidates, to being on-boarded as new hire, to getting rewarded as a high performer (or being counseled out).  Ideally, the way a hiring manager describes an engineering role to a recruit should match the role expectations that the company places on that engineer once they’re on board.  And the way promotions happen should further reinforce those behavioral expectations.

To give an example, here are the five behaviors that we expect all TaskRabbit Engineers to exhibit: 

  1. Culture Fit — TaskRabbit is driven by a very lofty vision and company values.  We take those really seriously and folks don’t tend to stick around at the company if they don’t visibly buy into the vision and values; even if they’re brilliant at their particular functional role.
  2. Craftsman — We want engineers to really understand the underlying mechanisms that their code relies on.  The majority of TaskRabbit’s codebase is in the form of Ruby-on-Rails apps. Mature Rails developers will appreciate just how easy it is to get yourself into hot water by including gems without really understanding what dependencies those gems are creating under the hood…  Whenever we can’t find a gem that does exactly what we want, in the way we want, we take pride in developing our own gems and contributing them back to the Rails community, e.g., makara, storehouse, sudo.js, and more
  3. Goalie — We don’t have a QA team, so our Engineers are the first and only line of defense against bugs.  We expect engineers to write their own feature tests and bugfixes, to deploy code only after integration and acceptance testing, and generally to think a lot about how code might break.  That kind of goalie defense against bugs and regressions is an important part of an engineer’s job at TaskRabbit.
  4. MacGyver — Like the TV show character, given a tight set of timelines and resource constraints, we expect engineers to be able to successfully identify the minimum-viable development investment that will make a meaningful impact for users.  This is really another way to say: we solve the classic Time-Quality-Features tradeoff by de-scoping/decomposing features.  We explicitly are unwilling to compromise on time or quality (i.e., we deliver on a pre-set weekly sprint cycle and we hold the quality bar really high for ourselves).  So, within a given sprint, we always focus on paring down to the smallest possible unit of customer value.
  5. Coach — TaskRabbit’s initial web and iOS apps came out of consulting engagements with Pivotal Labs, where we did a lot of pairing.  Coaching and mentoring each other has become an important part of our engineering practices and we really value people who are willing to actively teach their colleagues and learn from each other through 1-on-1 pairing as well as All-Hands meetings.  We specifically like to see engineers standing up in front of a crowd, talking about and demo’ing their work.

Every new recruit hears about these five expectations so they can make an educated decision about what they’re getting into at TaskRabbit.  Our engineering team members talk to each other about their jobs using these terms.  And these five behaviors are baked into our self-assessment and peer-reviews. 

By enumerating these five desired behaviors and weaving them all through out TaskRabbit’s engineering practices, I’m trying to create a more consistent and reliable way to recruit, retain and promote an A+ team.  

Feel free to borrow these if they’re helpful…   YMMV, of course, with transplanting these expectations into your company’s particular culture.  But I think the most fundamental point is to put some thought into defining a consistent set of behavioral expectations for your technical staff.

Thoughts?  Comments?  Let me know!


Structured Approach to Event Blogging

I like the “linkable assets model” in this SEJ post: http://www.searchenginejournal.com/effective-link-building-using-event-blogging/34104/

SeachEngineJournal's "linkable asset model"

It’s a nice, structured way to think about how to get inbound links for your event blog (if you’re an event blogger and care about that kind of thing)  :-)


The Language of Social Networks

Facebook and Mark Zuckerberg have established near monopoly control over the language that we use to talk about social networks.  Think about the words that we use to describe social network sites — terms such as “like”, “share”, “connect”, “transparent”, “more open”, etc.  I’m not saying that Facebook were the first company to use these terms and they certainly didn’t invent them.  But I do believe that Facebook have been the most effective and the most consistent at using those words to frame the conversation about social network policies and features in moral terms.

The moral framework they’re creating around their social network site is really key to focus on.  Facebook are saying: It’s *good* to be liked, right?  It’s *good* to connect to more people, right?  You’re a *better* person if you’re willing to share more transparently, right?  And, conversely, if you’re *not* willing to share a thought or action openly, then what are you hiding; are you doing something that you *shouldn’t* be doing?

Facebook have purposefully created this moral linguistic context for their service.  As Zuckerberg masterfully puts it:

At Facebook we have a deeply held purpose [...] Our mission: “Give people the power to share and make the world more open and connected.” In the world we’re building where the world is more transparent, it becomes good for people to be good to each other. That’s really important as we try to solve some of the world’s problems.

Source: http://www.insidefacebook.com/2008/07/23/live-notes-from-mark-zuckerbergs-keynote-at-f8-developer-conference/

George Lakoff, professor of linguistics and cognitive science tells us that, in any debate, if one side manages to dominate the moral framing of the issues, then they’ve already won the debate before it even begins.  Facebook appear to have learned Lakoff’s lessons well.  By framing the debate about social network policies and features in their own moral terms, Facebook could be walking away with victorious dominance over the social internet without firing a shot!  Every other social internet company has been forced to define themselves using Facebook’s terminology of “sharing”, “openness” and “social graphs” (i.e., attempting to beat Facebook at their own game) or to contrast themselves against Facebook’s feature functionality (i.e., attempting to convince the world that sharing, openness, and transparency are bad things).  It’s a lose-lose proposition either way and that makes it really hard for other companies to market and differentiate themselves using the language and concepts that Facebook own.

What are your thoughts on how the social network industry should move forward from here?  I’ll jot down some thoughts in my next blog post.  Would love to hear your thoughts in the meantime.


Crafting Code Across the Stack: 3 Benefits of Cross-Technology Engineering

One of the engineers in my startup asked some thought-provoking questions in a hallway conversation today:

Should engineers own end-to-end implementation of features that cut across multiple layers of our technology stack?  Or should engineers focus/specialize on specific layers of the stack and collaborate with each other to develop each feature?

I’ve repeatedly observed that small teams of developers run circles around large teams in every company I’ve ever been in.  [Caveat: I focus on consumer internet software, so my observations will be biased towards that industry.]  I have a strong preference for company cultures that encourage individual engineers to develop as much of a featureset as they can on their own.  If there’s a front-end user interface to be created as well as a backend API required to support a particular feature, I’d like to see the same engineer coding it all.   Obviously this isn’t always possible for either technical or personnel constraints, but it is the ideal in my mind.

Today’s announcement of Facebook’s integration with Skype is a timely example of this principle in action.  I wasn’t a direct part of that integration project, but knew the folks on both sides.  With two big companies like Facebook and Skype coming together, there were dozens of people involved in the overall project.  Even so, it was really remarkable to me to see how much of the integration really hinged on two key individuals — one on the Facebook side and one of the Skype side.  Both of those individual engineers are really amazing “multi-lingual” developers who were able to make key coding contributions across multiple layers of technology ranging from Java applets and compiled desktop executables to in-browser Javascript and CSS/HTML to server-based API’s and cloud services.

Seeing that reminded me of the three main benefits of getting engineers to own entire featuresets, even (or maybe especially) if they cut across multiple technologies…

1. Motivation

When an individual engineer works on a feature that they know is going to impact large numbers of users, they viscerally feel the huge contribution they’re making to their company (and society!).  Nothing is more motivating than feeling like your work really makes a difference.

2. Quality

An engineer who really understands an entire featureset front-t0-back is able to grok critical dependencies between technology layers more effectively than a team of multiple individuals.   I assert that the ability to see the whole picture enables the single engineer to more effectively write test cases, identify likely breaking points, and ultimately deliver higher overall quality code that is defensively coded against future commits.

3. Iteration

This is a key advantage for individual engineers: the engineer who designed and coded an entire featureset on their own will be more likely and much faster to modify/iterate their designs when presented with market feedback from real consumers.  In contrast, a team of engineers who each only coded parts of a feature (and therefore relied on some other person, usually a project manager or a product manager, to provide a holistic view and integration guidance) will be much slower to collectively process market feedback and iterate.

Now, this may all be much easier said than done…   For one thing, it’s hard enough to recruit great engineers in any given technical area, much less a team of cross-technology superstars.  And in practice, there are significant friction points to the lone-ranger style of development.  E.g., if every engineer is a gun slinger they may unwittingly produce a lot of conflicting or redundant code that at some point needs to refactored.  Furthermore, even if an engineer is capable of contributing code across many different levels of a tech stack, that doesn’t mean they will always *want* to do so…

Even netting out those potential disadvantages, I think that cross-technology development has a strong positive impact; especially in the consumer internet space where companies actually have a decent shot at recruiting engineers who can effectively contribute code across the entire LAMP (or insert your favorite mobile or web development framework here) stack.  And even if your company doesn’t, in practice, hand over entire featuresets to a single engineer to execute, I think it’s still really important to foster cross-technology understanding.  It can only benefit your team if the javascript front-end wizard really understands the database impact that all her AJAX calls are going to produce; or if your backend API engineer really understands how often and how quickly his REST services will be requested by a mobile client; etc.

Have you ever worked in a company that had a policy of “single engineer owns an entire feature”?


How employees get screwed in private equity deals

learned a hard lesson from working with a bunch of rat bastards leading private equity firm, Silver Lake.  I joined Skype after the company was spun out of eBay  by SilverLake in deal valued at $2.7B and was recruited to help accelerate the pace of product development and make the Skype app more web-oriented.  I was at the company for just over a year in a product management role and felt like my team accomplished some important things along the way, including reduction of software development cycles from months down to 2-weeks and delivery of a whole new advertising revenue stream to the company.   It was a fun and challenging job, involving tons of international travel and I met some amazing people along the way.

Now despite the fact that Skype has a Palo Alto office and kind of seems like it would fit right in with Silicon Valley tech companies, it turns out that the employment terms for a Silver Lake company are *very* different from what most Valley high-tech employees are used to.  Here are three important things to watch out for if you’re thinking about joining a company that is being managed by a private equity firm or if your company gets taken over by a PE bank.

1. Lawyer Up

(image credit: http://weheartit.com/entry/5625871)

The most important lesson I learned from Skype was that compensation and stock policies in PE-owned firms can be very heavily tilted in the owners’ favor and against the employees.  Skype employees have 5-year vesting of stock options, for example, not the usual 4 year schedule that most Valley firms have.  Even worse, Skype’s stock option agreement had special clauses that the Board had slipped in that gives them the right to “repurchase” any vested shares for anyone who leaves the company voluntarily or is terminated with cause — effectively taking “vested” shares and making them worthless.  Here’s a nice letter I got from the Associate General Counsel of Skype that points out exactly how my stock options have “no financial value.”  (see lee.pdf).  Gee, thanks.

Now, I’ve seen my share of legal documents for tech companies.  I’ve worked in Valley tech companies for over 15 years, have founded startups, done VC financings, and invested in companies.  None of that prepared me for the kinds of legal shenanigans that the PE guys at Silver Lake pulled because I had never come across those kinds of terms before, let alone the fact that these clauses were hidden as one-liners in otherwise pretty standard-looking documents.  (see Stock Option Grant Agreement for Kuo-Yee Lee – signed)

So my first point of advice to anyone considering working for a PE-lead firm is to LAWYER UP — it’ll be worth your while to get an attorney to carefully review all employment documents so that you know what you’re really getting into.

2. The Bobs

Working with Silver Lake was my first opportunity to witness up-close-and-personal how a PE firm does its business of restructuring a company that they’ve just taken over.  And it was breath-taking.  The firm inserted itself into every level of the company.  At one point in my tenure at Skype, Silver Lake had representatives or consultants on the Board, in C-level executive roles, in technical leadership and operating roles, and all the way on thru the organization to the person actually running our software deployment schedule…   So Silver Lake put its fingers really deeply into Skype’s pie and they started rearranging things.

You can agree or disagree with the practice of re-organization, but I personally had never been part of a restructuring that ran so deep in a company.  During the year I was at Skype, the company:

  • hired and fired a CFO
  • gained a CEO, CMO, CIO, and CDO
  • created an entirely new product development org structure
  • eliminated every Project Manager role
  • fired, re-interviewed,  and re-hired Product Managers
  • created a two new business units
  • combined two business units into one
  • dissolved one business unit
  • opened a new office and hired several hundred people
  • the list goes on…
I mean, these are crazy changes for any company to go through over the course of years.  To have that all happen within a short number of months was staggering.  The conventional wisdom in Silicon Valley is that good engineers and product designers will always have job security.  Still, there were times at Skype when even really solid engineers and designers were asking me if their jobs were going to be safe from all the changes going on.
So, second major lesson learned: prepare your resume and get ready to re-interview for your job (or a different one) because organizational change is a major part of the private-equity-lead restructuring of a company!

3. It Ain’t Over ’til It’s Over

Even if an employee of a PE-owned company has avoided the legal beartraps and weathered the re-org’ing, they’re still not safe.  Even as Skypers were celebrating the huge potential of the Microsoft deal, the PE bankers were sharpening their knives and plotting which employees to fire in order to maximize profits and minimize payouts to non-owners.   Seriously, how greedy do you need to be to make $5B and still try to screw the people who made that value possible?  I mean, Silver Lake is trying to hyper-optimize their returns to the point that they’re trying to deny employee payouts that amount to less than 0.3% of the returns that they’ll get from the deal.  Srsly.  Really?

So, just be warned: Silicon Valley startup folks may think we’ve had hard dealings with venture capitalists…  But in my opinion, VC greed pales in comparison to the level of greed exhibited by the Silver Lake private equity firm.

And there you have it, my top three lessons learned from being raked over the coals by a PE firm.

Have your own story?  Leave a link or comment below!


Three user-generated content UX principles

Three UGC UX principles that I’m working towards in my products:
1) Fast — sub-200 millisecond response time to any user input
2) Assistive — give users something to react to; rather than forcing them to generate their own novel content
3) Learning — system improves with every user click or action

What are your top three?


How Facebook Ships Code

I’m fascinated by the way Facebook operates.  It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software.

Seems like others are also interested in Facebook…   The company’s developer-driven culture is coming under greater public scrutiny and other companies are grappling with if/how to implement developer-driven culture.   The company is pretty secretive about its internal processes, though.  Facebook’s Engineering team releases public Notes on new features and some internal systems, but these are mostly “what” kinds of articles, not “how”…  So it’s not easy for outsiders to see how Facebook is able to innovate and optimize their service so much more effectively than other companies.  In my own attempt as an outsider to understand more about how Facebook operates, I assembled these observations over a period of months.  Out of respect for the privacy of my sources, I’ve removed all names and mention of specific features/products.  And I’ve also waited for over six months to publish these notes, so they’re surely a bit out-of-date.   I hope that releasing these notes will help shed some light on how Facebook has managed to push decision-making “down” in its organization without descending into chaos…  It’s hard to argue with Facebook’s results or the coherence of Facebook’s product offerings.  I think and hope that many consumer internet companies can learn from Facebook’s example.

HUGE thanks to the many folks who helped put together this view inside of Facebook.   Thanks are also due to folks like epriest and fryfrog who have written up corrections and edits.

Notes:

  • as of June 2010, the company has nearly 2000 employees, up from roughly 1100 employees 10 months ago.  Nearly doubling staff in under a year!
  • the two largest teams are Engineering and Ops, with roughly 400-500 team members each.  Between the two they make up about 50% of the company.
  • product manager to engineer ratio is roughly 1-to-7 or 1-to-10
  • all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs and listening to lectures given by more senior/tenured engineers.  estimate 10% of each boot camp’s trainee class don’t make it and are counseled out of the organization.
  • after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data)
  • [EDIT thx fryfrog] “There are also very good safe guards in place to prevent anyone at the company from doing the horrible sorts of things you can imagine people have the power to do being on the inside. If you have to “become” someone who is asking for support, this is logged along with a reason and closely reviewed. Straying here is not tolerated, period.”
  • any engineer can modify any part of FB’s code base and check-in at-will
  • very engineering driven culture.  “product managers are essentially useless here.” is a quote from an engineer.  engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime.  [EDITORIAL] The author of this blog post is a product manager, so this sentiment really caught my attention.  As you’ll see in the rest of these notes, though, it’s apparent that Facebook’s culture has really embraced product management practices so it’s not as though the role of product management is somehow ignored or omitted.  Rather, the culture of the company seems to be set so that *everyone* feels responsibility for the product.
  • during monthly cross-team meetings, the engineers are the ones who present progress reports.  product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that “product spoke too much at the last meeting.”  they really want engineers to publicly own products and be the main point of contact for the things they built.
  • resourcing for projects is purely voluntary.Engineers decide which ones sound interesting to work on.  a PM lobbies group of engineers, tries to get them excited about their ideas.  Engineer talks to their manager, says “I’d like to work on these 5 things this week.”  Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
  • Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between.  If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on.  Same for Architect help.  But in general, expectation is that engineers will handle everything they need themselves.
  • arguments about whether or not a feature idea is worth doing or not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.
  • engineers generally want to work on infrastructure, scalability and “hard problems” — that’s where all the prestige is.  can be hard to get engineers excited about working on front-end projects and user interfaces.  this is the opposite of what you find in some consumer businesses where everyone wants to work on stuff that customers touch so you can point to a particular user experience and say “I built that.”  At facebook, the back-end stuff like news feed algorithms, ad-targeting algorithms, memcache optimizations, etc. are the juicy projects that engineers want.
  • commits that affect certain high-priority features (e.g., news feed) get code reviewed before merge. News Feed is important enough that Zuckerberg reviews any changes to it, but that’s an exceptional case.
  • [CORRECTION -- thx epriest] “There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn’t look at every change personally.”
  • [CORRECTION thx fryfrog] “All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don’t invite them to. It would take intentionally malicious behavior to get un-reviewed code in.”
  • no QA at all, zero.  engineers responsible for testing, bug fixes, and post-launch maintenance of their own work.  there are some unit-testing and integration-testing frameworks available, but only sporadically used.
  • [CORRECTION thx fryfrog] “I would also add that we do have QA, just not an official QA group. Every employee at an office or connected via VPN is using a version of the site that includes all the changes that are next in line to go out. This version is updated frequently and is usually 1-12 hours ahead of what the world sees. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon.”
  • re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code.  it’s just that they don’t have an incentive to do so at most companies.  when there’s a QA department, it’s easy to just throw it over to them to find the errors.”  [EDIT: please note that this was subjective opinion, I chose to include it in this post because of the stark contrast that this draws with standard development practice at other companies]
  • [CORRECTION thx epriest] “We have automated testing, including “push-blocking” tests which must pass before the release goes out. We absolutely do not believe “most engineers are capable of writing bug-free code”, much less that this is a reasonable notion to base a business upon.”
  • re: surprise at lack of PM influence/control — product managers have a lot of independence and freedom.  The key to being influential is to have really good relationships with engineering managers.  Need to be technical enough not to suggest stupid ideas.  Aside from that, there’s no need to ask for any permission or pass any reviews when establishing roadmaps/backlogs.  There are relatively few PMs, but they all feel like they have responsibility for a really important and personally-interesting area of the company.
  • by default all code commits get packaged into weekly releases (tuesdays)
  • with extra effort, changes can go out same day
  • tuesday code releases require all engineers who committed code in that week’s release candidate to be on-site
  • engineers must be present in a specific IRC channel for “roll call” before the release begins or else suffer a public “shaming”
  • ops team runs code releases by gradually rolling code outthere are 9 concentric levels for rolling out new code
  • facebook has around 60,000 servers
  • [CORRECTION thx epriest] “The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.”
  • the smallest level is only 6 servers
  • e.g., new tuesday release is rolled out to 6 servers (level 1), ops team then observes those 6 servers and make sure that they are behaving correctly before rolling forward to the next level.
  • if a release is causing any issues (e.g., throwing errors, etc.) then push is halted.  the engineer who committed the offending changeset is paged to fix the problem.  and then the release starts over again at level 1.
  • so a release may go thru levels repeatedly:  1-2-3-fix. back to 1. 1-2-3-4-5-fix.  back to 1.  1-2-3-4-5-6-7-8-9.
  • ops team is really well-trained, well-respected, and very business-aware.  their server metrics go beyond the usual error logs, load & memory utilization stats — also include user behavior.  E.g., if a new release changes the percentage of users who engage with Facebook features, the ops team will see that in their metrics and may stop a release for that reason so they can investigate.
  • during the release process, ops team uses an IRC-based paging system that can ping individual engineers via Facebook, email, IRC, IM, and SMS if needed to get their attention.  not responding to ops team results in public shaming.
  • once code has rolled out to level 9 and is stable, then done with weekly push.
  • if a feature doesn’t get coded in time for a particular weekly push, it’s not that big a deal (unless there are hard external dependencies) — features will just generally get shipped whenever they’re completed.
  • getting svn-blamed, publicly shamed, or slipping projects too often will result in an engineer getting fired.  “it’s a very high performance culture”.  people that aren’t productive or aren’t super talented really stick out.  Managers will literally take poor performers aside within 6 months of hiring and say “this just isn’t working out, you’re not a good culture fit”.  this actually applies at every level of the company, even C-level and VP-level hires have been quickly dismissed if they aren’t super productive.
  • [CORRECTION, thx epriest]  “People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren’t around to support them in case something goes wrong (and haven’t found someone to cover for you).”
  • [CORRECTION, thx epriest] “Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.”
  • [CORRECTION, thx fryfrog] “I also don’t know of anyone who has been fired for making mistakes like are mentioned in the article. I know of people who have inadvertently taken down the site. They work hard to fix what ever caused the problem and everyone learns from it. The public shaming is far more effective than fear of being fired, in my opinion.”

It’ll be super interesting to see how Facebook’s development culture evolves over time — and especially to see if the culture can continue scaling as the company grows into the thousands-of-employees.

What do you think?  Would “developer-driven culture” work at your company?


Follow

Get every new post delivered to your Inbox.

Join 2,414 other followers