How Facebook Ships Code

I’m fascinated by the way Facebook operates.  It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software.

Seems like others are also interested in Facebook…   The company’s developer-driven culture is coming under greater public scrutiny and other companies are grappling with if/how to implement developer-driven culture.   The company is pretty secretive about its internal processes, though.  Facebook’s Engineering team releases public Notes on new features and some internal systems, but these are mostly “what” kinds of articles, not “how”…  So it’s not easy for outsiders to see how Facebook is able to innovate and optimize their service so much more effectively than other companies.  In my own attempt as an outsider to understand more about how Facebook operates, I assembled these observations over a period of months.  Out of respect for the privacy of my sources, I’ve removed all names and mention of specific features/products.  And I’ve also waited for over six months to publish these notes, so they’re surely a bit out-of-date.   I hope that releasing these notes will help shed some light on how Facebook has managed to push decision-making “down” in its organization without descending into chaos…  It’s hard to argue with Facebook’s results or the coherence of Facebook’s product offerings.  I think and hope that many consumer internet companies can learn from Facebook’s example.

HUGE thanks to the many folks who helped put together this view inside of Facebook.   Thanks are also due to folks like epriest and fryfrog who have written up corrections and edits.

Notes:

  • as of June 2010, the company has nearly 2000 employees, up from roughly 1100 employees 10 months ago.  Nearly doubling staff in under a year!
  • the two largest teams are Engineering and Ops, with roughly 400-500 team members each.  Between the two they make up about 50% of the company.
  • product manager to engineer ratio is roughly 1-to-7 or 1-to-10
  • all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs and listening to lectures given by more senior/tenured engineers.  estimate 10% of each boot camp’s trainee class don’t make it and are counseled out of the organization.
  • after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data)
  • [EDIT thx fryfrog] “There are also very good safe guards in place to prevent anyone at the company from doing the horrible sorts of things you can imagine people have the power to do being on the inside. If you have to “become” someone who is asking for support, this is logged along with a reason and closely reviewed. Straying here is not tolerated, period.”
  • any engineer can modify any part of FB’s code base and check-in at-will
  • very engineering driven culture.  “product managers are essentially useless here.” is a quote from an engineer.  engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime.  [EDITORIAL] The author of this blog post is a product manager, so this sentiment really caught my attention.  As you’ll see in the rest of these notes, though, it’s apparent that Facebook’s culture has really embraced product management practices so it’s not as though the role of product management is somehow ignored or omitted.  Rather, the culture of the company seems to be set so that *everyone* feels responsibility for the product.
  • during monthly cross-team meetings, the engineers are the ones who present progress reports.  product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that “product spoke too much at the last meeting.”  they really want engineers to publicly own products and be the main point of contact for the things they built.
  • resourcing for projects is purely voluntary.Engineers decide which ones sound interesting to work on.  a PM lobbies group of engineers, tries to get them excited about their ideas.  Engineer talks to their manager, says “I’d like to work on these 5 things this week.”  Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
  • Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between.  If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on.  Same for Architect help.  But in general, expectation is that engineers will handle everything they need themselves.
  • arguments about whether or not a feature idea is worth doing or not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.
  • engineers generally want to work on infrastructure, scalability and “hard problems” — that’s where all the prestige is.  can be hard to get engineers excited about working on front-end projects and user interfaces.  this is the opposite of what you find in some consumer businesses where everyone wants to work on stuff that customers touch so you can point to a particular user experience and say “I built that.”  At facebook, the back-end stuff like news feed algorithms, ad-targeting algorithms, memcache optimizations, etc. are the juicy projects that engineers want.
  • commits that affect certain high-priority features (e.g., news feed) get code reviewed before merge. News Feed is important enough that Zuckerberg reviews any changes to it, but that’s an exceptional case.
  • [CORRECTION -- thx epriest] “There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn’t look at every change personally.”
  • [CORRECTION thx fryfrog] “All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don’t invite them to. It would take intentionally malicious behavior to get un-reviewed code in.”
  • no QA at all, zero.  engineers responsible for testing, bug fixes, and post-launch maintenance of their own work.  there are some unit-testing and integration-testing frameworks available, but only sporadically used.
  • [CORRECTION thx fryfrog] “I would also add that we do have QA, just not an official QA group. Every employee at an office or connected via VPN is using a version of the site that includes all the changes that are next in line to go out. This version is updated frequently and is usually 1-12 hours ahead of what the world sees. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon.”
  • re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code.  it’s just that they don’t have an incentive to do so at most companies.  when there’s a QA department, it’s easy to just throw it over to them to find the errors.”  [EDIT: please note that this was subjective opinion, I chose to include it in this post because of the stark contrast that this draws with standard development practice at other companies]
  • [CORRECTION thx epriest] “We have automated testing, including “push-blocking” tests which must pass before the release goes out. We absolutely do not believe “most engineers are capable of writing bug-free code”, much less that this is a reasonable notion to base a business upon.”
  • re: surprise at lack of PM influence/control — product managers have a lot of independence and freedom.  The key to being influential is to have really good relationships with engineering managers.  Need to be technical enough not to suggest stupid ideas.  Aside from that, there’s no need to ask for any permission or pass any reviews when establishing roadmaps/backlogs.  There are relatively few PMs, but they all feel like they have responsibility for a really important and personally-interesting area of the company.
  • by default all code commits get packaged into weekly releases (tuesdays)
  • with extra effort, changes can go out same day
  • tuesday code releases require all engineers who committed code in that week’s release candidate to be on-site
  • engineers must be present in a specific IRC channel for “roll call” before the release begins or else suffer a public “shaming”
  • ops team runs code releases by gradually rolling code outthere are 9 concentric levels for rolling out new code
  • facebook has around 60,000 servers
  • [CORRECTION thx epriest] “The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.”
  • the smallest level is only 6 servers
  • e.g., new tuesday release is rolled out to 6 servers (level 1), ops team then observes those 6 servers and make sure that they are behaving correctly before rolling forward to the next level.
  • if a release is causing any issues (e.g., throwing errors, etc.) then push is halted.  the engineer who committed the offending changeset is paged to fix the problem.  and then the release starts over again at level 1.
  • so a release may go thru levels repeatedly:  1-2-3-fix. back to 1. 1-2-3-4-5-fix.  back to 1.  1-2-3-4-5-6-7-8-9.
  • ops team is really well-trained, well-respected, and very business-aware.  their server metrics go beyond the usual error logs, load & memory utilization stats — also include user behavior.  E.g., if a new release changes the percentage of users who engage with Facebook features, the ops team will see that in their metrics and may stop a release for that reason so they can investigate.
  • during the release process, ops team uses an IRC-based paging system that can ping individual engineers via Facebook, email, IRC, IM, and SMS if needed to get their attention.  not responding to ops team results in public shaming.
  • once code has rolled out to level 9 and is stable, then done with weekly push.
  • if a feature doesn’t get coded in time for a particular weekly push, it’s not that big a deal (unless there are hard external dependencies) — features will just generally get shipped whenever they’re completed.
  • getting svn-blamed, publicly shamed, or slipping projects too often will result in an engineer getting fired.  “it’s a very high performance culture”.  people that aren’t productive or aren’t super talented really stick out.  Managers will literally take poor performers aside within 6 months of hiring and say “this just isn’t working out, you’re not a good culture fit”.  this actually applies at every level of the company, even C-level and VP-level hires have been quickly dismissed if they aren’t super productive.
  • [CORRECTION, thx epriest]  “People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren’t around to support them in case something goes wrong (and haven’t found someone to cover for you).”
  • [CORRECTION, thx epriest] “Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.”
  • [CORRECTION, thx fryfrog] “I also don’t know of anyone who has been fired for making mistakes like are mentioned in the article. I know of people who have inadvertently taken down the site. They work hard to fix what ever caused the problem and everyone learns from it. The public shaming is far more effective than fear of being fired, in my opinion.”

It’ll be super interesting to see how Facebook’s development culture evolves over time — and especially to see if the culture can continue scaling as the company grows into the thousands-of-employees.

What do you think?  Would “developer-driven culture” work at your company?


313 Comments on “How Facebook Ships Code”

  1. jawali says:

    Wonder what exactly Product Managers do….

    • foobar says:

      Perhaps they act as cheerleaders/soundingboards for ideas.

      • JiggaBoo says:

        “foobar Says:” Haha! That’s absolutely right!

      • thepiedpipes says:

        that’s pretty ignorant. a good product manager is often the font of good ideas, not just the first defense against bad ones.

      • Nithin says:

        If its an Engineering driven environment, what role do Product Managers play? Seriously.

      • thepiedpipes says:

        @nithin sounds like you haven’t experienced any good product managers. they are often the only people in the organisation that can simultaneously understand the complexities of the technical effort, the needs of the market, and the nuances of the user experience, whilst providing balance to some of the egos that can quickly torpedo a viable business.

      • Jason says:

        Managers in general seem pretty worthless to me.

      • rb says:

        I agree with thepiedpipes. Project managers are very useful to a developer if they are good at what they do. If you need answers to something they go get it, if you don’t want to talk to some annoying account exec just refer them to the PM. Personally they make great “meat shields” and “go getters”. They have a tough job. They do everything in a project that no one else wants to do. Having a good PM, I can concentrate on the quality of my code and I can trust that everything else is taken care of. But, I’ve also had bad PMs and a bad PM just makes the project worse that if there weren’t one at all.

      • Max says:

        Project managers and QA are completely worthless! They bring no value to the team.

    • Opdelt says:

      Well-well look. I already told you: I deal with the god damn customers so the engineers don’t have to. I have people skills; I am good at dealing with people. Can’t you understand that? What the hell is wrong with you people?

    • PM Hut says:

      Here’s an explanation on what product managers are: http://www.pmhut.com/scrum-product-manager-product-owner-roles-and-responsibilities

      It explains the roles and the responsibilities of the Scrum Product Manager.

    • thepiedpipes says:

      @rb Project Managers != Product Managers

      Completely different skill sets. PMs are task masters, organisational gurus and often scrum masters. Product managers should own the vision for the product, steer its features backlog, have the right blend of UX/tech and business knowledge, and communicate the road map for the product outside the business and within.

      @jason That’s probably because you aren’t one and don’t know what a good manager is able to do.

    • Mr. G. says:

      I’m a product manager and in that capacity I put together specs, coordinate designers, developers and marketing teams to make sure they execute the product vision and attain the financial goals set by management, put together the roadmap, get dev excited about new ideas, download every possible app out there and sign-up to any possible website to make sure I’m in tune with the market, etc… A developer can not spend its time analyzing the financials of a product to see if its profitable, nor deal with UI issues, needs to stay focus on what they do best, i.e. code. We stay focus on what we do best, i.e. sending emails, putting together useless Powerpoint and building business models. I agree that product managers are pretty useless if they can’t talk the language of developers (or designers as a matter of fact) and that business school is not necessarily the best training ground for that. Rather be passionate about geekish stuff. Regarding code, I always thought that a PM should be able to read, not necessarily write.

  2. William Heckman says:

    The cubicle is the result of culture driven work environment. Intel used it so that engineers could have privacy yet communicate if they needed to rapidly. Its interesting that other companies simply employ cubicles because of the successful nature of intel, not necessarily because of what it actually does to improve the work environment or improve the operations of the company.

  3. epriestley says:

    There are some significant inaccuracies in this article. See my post on reddit for some clarifications:

    http://www.reddit.com/r/programming/comments/f3u0n/how_facebook_ships_code/c1d3b37

    (I do not speak for my employer, Facebook.)

    • yeeguy says:

      Thanks for the clarifications re: mandatory code review, automated tests, and emphasis on bug fixing. Sounds like Facebook’s practices are actually more defensive around code checkins than my notes may indicate.

      The thing I still find remarkable is how all of those practices (and it seems like the rest of the culture) centers on engineers…

  4. [...] Apparently, Facebook has around 60,000 servers. How are there ever times when Facebook is down? [...]

    • Nope says:

      A few ways:

      1) Organizing server is down, so those 60,000 servers all can’t get to something.
      2) You only connect to one of those, and that might be down before their system can detect it.

  5. lucentshoe says:

    So what’s the other half of the company, and what all do they do? How do these other units get on with Engineering and with Ops?

  6. Drew says:

    You know, designers do actually have a purpose. Especially on a site like Facebook.

    I hate to say this, because Facebook’s approach seems to generally work pretty well. Most engineers (read: developers) have little to no grasp of the user experience. And in terms of Facebook causing widespread user experience panic every time they shuffle the layout, I’m not surprised to learn that FB allows engineers to design the user experience most of the time. No wonder the layout this time around especially sucks…engineers are designing in a linear fashion.

    • rojo says:

      Agreed. The UX on facebook really sucks.

      • starmonkey says:

        How about you qualify that statement with some specific examples? I personally have no (significant) problem with FB’s UI. It’s rather straight-forward to me.

        I used to have trouble with privacy options but I figured that was intentional :)

    • Stard0g101 says:

      You know that, most of the layout changes are to do with a backend process, or to do with improving the performance metrics of teh system as a whole; holistic application development cannot be undertaken by designers as they know what they are asking for ;)

    • R says:

      The layout is just fine. It’s noobs like you that made MySpace popular in the first place.

      • Drew says:

        Ok, so… I’m a noob because I believe that engineers make terrible front-end designers? Engineers think logically and end-users generally don’t. So to have a group of people (engineers) changing the layout to fit back-end needs, rather than the needs of the people who’ll actually be using it is absurd. Especially when you’re talking about 400 million users.

      • NorbertH says:

        It’s pretty rare to find an engineer that has a decent grasp of the complexities and subtleties of Ux and UI design but only the best engineers will not kid themselves that they do and actually respect people that do (equally so inversely).

        FB’s Ux/UI is flawed on many levels, breaks a bunch of industry standards and rules of thumb but that does not render it useless by any means, it just will prevent a percentage of users from accessing certain features/functionality easily. If these are non-critical areas then it is less of an issue than say “privacy settings” for example.

    • Frank Barth says:

      If the UX is so bad, I wonder why so many sites try to copy Facebook’s, and why I know so many UXler who love FB.

      And: There are designers (and all should be able to do UX also, I do not see why these fields should be separated), if the developers think they need any. In most cases a general template is used however, and the developers can copy the structure and do the rest rather easily.

      • Olivier St-Cyr says:

        It’s easy to say that people love this and people love that, it’s another thing to demonstrate it with a solid used-centered process. We all have opinions about what we love and what we hate, but “Your opinion, while interesting is often irrelevant”. What’s relevant is what UX practitioners can demonstrate through the data collected from real users of the product. I’ve been in this battle many times and it’s what I consider to be the biggest problem with UI/UX design:

        Since everyone uses a computer, everyone thinks they are able to design a UI and have an opinion on the UI. That’s the core of the problem here. If a civil engineer comes to you and makes a recommendation about a structure, most likely people will not debate, on the basis of his/her expertise (civil engineers may debate between them, but Joe and Jane from let’s say social studies won’t). When it comes to Human Factors Engineering/UX design, while there are specialists out there with the appropriate training who can make recommendations based on an established process, we all think we can have our say on the basis that we all used a UI at some point in our life. That’s a very dangerous approach to follow.

        So why so many sites try to copy Facebook? Because they don’t know better. They think if millions of people use it, it most be good, so let’s do it. WRONG! It does not mean millions of people use it that it’s a good design. Millions of people may have been forced to adapt to a bad design (humans have great abilities to adapt over time, even the worst design may not be too bad for some users because they have adapted to it).

        If you are carefully reading people’s comments on facebook about their UI when major UX overhauls are pushed through, you’ll read a lot of things like “OH NO, FACEBOOK UI CHANGES AGAIN #$%*&&#$” and after a few days, people painfully adapt to the new UI. Not the best approach to push a new UI through if you ask me. That being said, not every aspect of their UI is like that. There are definitely good aspects.

    • Olivier St-Cyr says:

      I agree with you on that one. (see comment I posted today below). I mean, I have met a few developers who can do UX design, but the reality is, most of them can’t. They just don’t think in a user-centered manner, they think from the structure of the code. So basically, if you want a UI that shows you the structure of the code (or the organization of your data structures), developers are good to produce that. The problem is that users don’t think that way. Thinking from a user-centered point of view is something different.

      Now, as for designers, yes I agree with the one of the post that says designers should be able to do the UX (after all they are designers). The problem here again is that many of them never heard of UX in their design training. While they should not be UX specialist, they should have at least a minimum training in UX.

      At the end of the day, it all comes down to how capable is someone (in any position) to think from a user-centered point of view. History shows us that certain people are capable of doing this naturally (these are the very good designers, engineers, developers, trade workers, etc… but are rare). Most often though, people don’t naturally think that way. I always use the example of an electrician when I teach my UI/UX class (nothing against BTW). If you hire an electrician to wire your house, you really want to hire the one who will think about how people in the house are going to move around and where switches, receptacles, and three-way switches should be located. That will lead to well-designed electrical system configurations that are usable by the people live in the house (the primary users). Do these people exist? Sure they do, in a very limited quantity. The experience most of us have when we move into our houses is that some switches are behind doors, some three-way switches are missing to allow you to turn off the lights at the end of the hallway, and a few receptacles are in usable spots. Yet, from a wiring point of view, everything is working… That does not mean electrician are not good at what they do. It just shows that their natural tendencies may not be to wire a house from a user-centered point of view.

      • Scott Shattuck says:

        Wait, we’re arguing about a site who’s user primary interface is a single text field right?

      • Dave G says:

        Olivier,
        Rather than being so long winded, why not just give example sites with a good interface. I think Facebook has a wonderful interface, one of the best out there, and that is why I use it everyday.

      • Ryan Kane says:

        RE Dave G: I thought the point here was to discuss how does UX fits in the facebook development process described in the article rather than comment on the facebook UI

    • The layout that FB have now is very very good for the users.

  7. [...] I wonder if this method really does scale well. I don’t think it would work everywhere. [...]

  8. Pete says:

    The multi-level release cycle is interesting, but how does Facebook deal with database schema changes etc? Are they simply using a really great key->value nosql type of solution?

  9. whatsthebeef says:

    I most areas responsability is shifting towards engineers generally with the introduction of agile methodologies, and the more successful companies who are able to hire more talented, well rounded engineers are generally able to carry this off more effectively.

    I think such practices become harder to maintain as the work force grows.

  10. critic says:

    If all the engineers write bug free code or at least caught in the release process where they are paged to fix their issues or face public shaming or at worst getting fired then why are boot campers only dedicated to fixing bugs and listening to lectures? Wouldn’t that mean the boot campers have nothing to do but listen to lectures?

  11. DW says:

    Interesting notes given what a hair-pulling nightmare their public APIs are to integrate with. Stuff changes regularly and gets dropped on apparent whims with no prior warning so it makes perfect sense that the development is run from the bottom up with no overarching vision to ensure that not just the sexy features du jour are getting attention.

  12. [...] this post today on Frame Think: http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ about how Facebook ships code. My favorite part: arguments about whether or not a feature idea is [...]

  13. Chris says:

    First it’s either unique or not. It can’t be “very unique”

    Secondly this process would seem to explain why Facebook seems to break its own APIs so often.

    • starmonkey says:

      re: Uniqueness – That depends on the scope of reference. An object can be considered “quite unique” if many of it’s attributes are unique, but some are not.

      The syntax of human language evolves over time. Stop being pedantic else be left behind :)

      • killercacti says:

        The semantics of human language changes far less over a short period of time.

        And for the word unique, it hasn’t changed at all in many years.

        Stop being an idiot and perhaps you’ll have a fighting chance to catch up to those who are actually _in_ the race.

    • Bean Taxi says:

      Disagree . . . imo something can be “very unique”, if it’s not just different from everything else, but *very* different from everything else.

      Agree though that “very unique” is heavily abused, and in most cases is redundant.

  14. [...] I was just reading this article found on Hacker News about the development process at facebook. [...]

  15. Kenny says:

    I would rather work at Google :)

  16. [...] Story: How Facebook Ships Code) This entry was posted in web2.0 and tagged development, facebook. Bookmark the permalink. [...]

  17. serg says:

    bullshit

    from line 1 to line {end}

    • Mr Tube Steak says:

      whoa. look at the amazing work environment. all that focus on product and how powerful engineers are. it really shows in how profitable facebook is. their balance sheet is a glowing example of the success of this company.

  18. Rizzo says:

    I +1 a lot of that…

    no QA, check-in at will, everyone has access to production

    That is the peace through anarchy model.

    Rizzo

  19. Jack says:

    Even with the corrections (re: the FB employee) I find the cultural implications at Facebook very interesting. I work in a company the requires intense engineering development, but still utilizes the cubicle model. I find that this is a major drain on productivity and slows our release schedule for a variety reasons (less camaraderie, less face time, lowered team cohesion, etc) . I’m sure that the FB culture isn’t right for everyone, but I’m sure there is a nice midpoint between cubiclism and full-open culture.

  20. [...] Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility”Five Emotions Invented By The Internet [...]

  21. Amber Yust says:

    WPStats (wordpress stats) tracking bug image.

  22. Quora says:

    Are product managers useless at Facebook?…

    http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ It does look so…

  23. [...] Questions, Topics and PeopleAdd QuestionAdd QuestionAre product managers useless at Facebook?http://framethink.wordpress.com/…It does look so  BIU     @   Edit Link Text Show [...]

  24. [...] Even with the corrections (re: the FB employee) I find the cultural implications at Facebook very interesting. I work in a company the requires intense engineering development, but still utilizes the cubicle model. I find that this is a major drain on productivity and slows our release schedule for a variety reasons (less camaraderie, less face time, lowered team cohesion, etc) . I’m sure that the FB culture isn’t right for everyone, but I’m sure there is a nice midpoint between cubiclism and full-open culture. ~ Jack Amplify’d from framethink.wordpress.com [...]

  25. [...] thorough look at how Facebook ships code. From internal engineering processes to the lack of project manager influence/control, how and [...]

  26. [...] today, FrameThink posted an entry about how Facebook ships code. There were quite a lot of “juicy” bits to the story [...]

  27. Quora says:

    How accurate is the “How Facebook Ships Code” article written by ‘yeeguy’?…

    Thanks a ton for the corrections and additional material. I think it’d be fantastic for more companies to learn about how Facebook’s development process works — “developer-driven culture” is really the ultimate expression of pushing decisions “do…

  28. [...] January 17, 2011 by Brian McKay Leave a Comment I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. It's been over six months since I assembled these observations and I'm sure Facebook has continuously evolved its software development practices in the meanti … Read More [...]

  29. John Jensen says:

    I think techniques like these may be very successful only in organizations where developers are working on consumer end-products that they themselves use (like Facebook). In that environment, the fundamental role that a product manager plays of identifying, prioritizing and translating the needs of end-users is more easily obviated. I can’t see this working on software projects where the problem domain is much more complex or esoteric.

    • A Product Guru says:

      Good point John. The other item that’s overlooked is the fact that product managers are less important when you have lots of money to burn on “experiments” and the BEST engineers who are capable of understanding what users want and what features to build/prioritize. Facebook has that luxury today, as did Google a couple years ago. When the quality of engineering talent goes down, the importance of a talented product team goes up. Historically, you can see that shift at companies like Yahoo, and you can see it starting to happen at Google now (guys like Max Levchin and Brad Horowitz are undoubtably ‘product’ leaders who are hired when engineers can’t experiment their way through things). Lastly, it’s undeniable that Zuck is a ‘product’ guy just as he is an ‘engineer’ – this is common with most of the major web 2.0 companies.

  30. Rocky Madden says:

    Great insight, thank you. I am surprised most of all about the lack of unit testing and formal QA department. I’d have to work in such an environment to give a fair opinion, but it sounds like the wild west for quality. Their stability would seem to prove me wrong though.

  31. aforty says:

    This is fascinating. I would love to work for a company that has that kind of culture.

  32. [...] How Facebook Ships Code I’m fascinated by the way Facebook operates.  It’s a very unique environment, not easily replicated (nor would their [...] [...]

  33. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People interesting [...]

  34. I think that this is a great set of notes — extremely interesting. However, I think that the title is off, as is the notion that Facebook “ships and releases” software. Facebook does neither of those things: they publish a service and an API for other developers to use, they don’t ship any software.
    Shipping software is a completely different matter than publishing a service, and demands a different development process. They’re still a brilliant organization filled with smart people, but it’s clear to see why their process wouldn’t work for a company that ships code. For example, if a nasty little bug makes it way into production during a weekly release at Facebook, they can page the engineer and get it fixed (downtime will be small and isolated). If this happens with shipped software, running in customer environments, a lot of people might get screwed and you’ll have to test and release an update that will cost your customers time and money to apply (that doesn’t make customers very happy). You definitely need more thorough testing and verification processes when the responsible engineer can get on the server and dig in to the problem immediately to fix it.
    As a model for service delivery however, I think that organizations can learn a lot from Facebook’s developer-driven culture. If you make engineers directly responsible for their own stuff (including problems they cause), they’ll be pretty motivated to avoid public shame.

  35. [...] ran across this article today on FrameThink about Facebook’s deployment process. For a company that has grown and [...]

  36. [...] by liotier [link] [322 comments] [...]

  37. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. It's been over six months since I assembled these observations and I'm sure Facebook has continuously evolved its software development practices in the meanti … Read More [...]

  38. [...] stuff here.   If you enjoyed this article, please consider sharing [...]

  39. [...] How Facebook Ships Code – “I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software….” [...]

  40. [...] back-end of the social network Facebook.com is complex. Facebook’s servers run on a LAMP stack with Memcache. They also use MySQL databases and PHP, [...]

  41. Imsorry ButIdont says:

    facebook sucks. myspace sucked. people only use it because their friends use it. everyone left myspace and everyone can leave FB just as easily.

    no designers and no real QA?

    wtf?

  42. Bunnios says:

    There is no modifier for unique. It’s just unique. Not “very unique”, not “really unique”.

  43. [...] they should know a bit about the science of pricing by nowHow Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility”Five Emotions Invented By The Internet [...]

  44. [...] Read the rest of this post on the original site Tagged: Facebook, digital, frontpage, social networking, software, FrameThink, organization, Yee Lee | permalink var SurphaceSettings = { url: "http://voices.allthingsd.com/20110118/how-facebook-ships-code/", siteid: "atd" }; var _surphld = document.createElement("script"); _surphld.type = "text/javascript"; _surphld.src = "http://cdn11.surphace.com/rcwidget/loader.js"; (document.getElementsByTagName("head")[0] || document.getElementsByTagName("body")[0]).appendChild(_surphld); « Previous Post Next Post » ord=Math.random()*10000000000000000; document.write(''); [...]

  45. almiralabs says:

    I guess FB engineers should be not too happy about the article as it might look they are a bunch of crazy wild-west coders. Do not believe they do not have a discipline or that PMs or MKT do not play a role, might it be different from other companies standards.
    Engineers involvement can be good or bad. I see some positive aspects like being responsible for their code and features, fact that empowers them, but things like that they decide the products features just give me the frights.
    As a final comment, I think you are looking for a silver bullet that reading SW-Eng savvy people like Tom deMarco in “Peopleware” you can conclude does not exist. SWEng practices are there for a reason and a SW process is always needed, be it a formal RUP process or an Agile one. BTW, Agile does not mean “do what I want”. It is a formal process like any other, with its disciplines and musts.
    Problem is many people out there just cover their anarchy and lack of process stating ” I am agile”.
    Honestly do not think the process described is magic or a novelty, although has attractive bits. Hope it works fine for them.

  46. Karl says:

    This article is obviously written by a developer frustrated with pm’s, with sources of developers who get a huge hard on by the power and responsiblity they THINK they have. This is just not a realistic scenario.

  47. [...] purpose or not, there is no denying that Facebook is a high traffic, technically challenging site. These rumors on how Facebook handles development and deployment are of course to take with a grain of salt, but [...]

  48. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People. [...]

  49. [...] Chan at 18:35 (7 seconds ago) in Blog No Response9 viewsInteresting blog post by yeeguy entitled How Facebook Ships Code. mig33 do share some common things with Facebook =)Here are 8 interesting snippets:All engineers go [...]

  50. eurgh says:

    I’d hate to work at Facebook based on the whole shaming thing, they’ll kill all their engineers with stress.

  51. HB says:

    From my experience, this model always works, BUT:
    -you need a team of excellent engineers (all of them, no exceptions)
    -you need to convince management that it is the way to go.

    So, in the real world, it is seldom used because:
    -great engineers cost more money (and they just write code right? they’re all the same)
    -management rarely has an engineering background to understand why this model is worth it

  52. Andrew says:

    I’ve worked in an organisation that had a similar setup.

    It was a disaster so much so that the organisation is now refocusing very strongly on a product management culture.

    To me reading this suggests Facebook have too many developers and too little focus on quality and doing right by their users. No surprise that they frequently have users up in arms about UI and privacy.

    You need balance in an organisation to get the best out of your team. Putting all the emphasis on developers does not give a balance of skills.

  53. You forgot to mention how they ship their code technically:
    The use BitTorrent! Every updated server will be another seeder. it´s pure genius if you have thousand of servers to update.

  54. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People (tags: to_read facebook development programming management code) [...]

  55. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  56. [...] recently read a great article about how Facebook ships code and what their values and practices are. It revolves around a list of different examples of how [...]

  57. Calvin Crane says:

    Sounds hideous to my mind. Similarly working for ferrari must be boring. Striving for perfection in the mundane. I prefer concept level and cant quite get excited about this but judging by all the comments I may be alone :) and happy about it.

  58. [...] read two interesting articles about development in Facebook (How Facebook Ships Code and LAUNCH002: What I Learned from Zuckerberg’s Mistakes). Apparently the programmers choose [...]

  59. [...] encontrado un interesante post en el blog FrameThink, en el cuál su autor que usa un pseudónimo "yeeguy", dice que ha tenido [...]

  60. Olivier St-Cyr says:

    I wonder how User Experience/Usability people work within an organization like that. Typically, the fact that the engineers have the final words would probably turn off many UX practitioners who may feel that the work/studies they are doing and the findings they are proposing does not get listen to.

  61. Simon White says:

    I think UX need to get people excited too. So what if engineering own the front end. UX would need to get the engineers excited about making features more usable. So the engineer that is coding a new interaction could use a UX mindset.

    UX is a state of mind anyway, the UX team at Facebook could probably be a good catalyst role.

  62. [...] especially not when you’re working in a team. So does Facebook handle this problem? Check out this interesting blogpost! [...]

  63. [...] Want to know a little about Facebook behind the scenes? [...]

  64. This will not work on all organizations, however, some of the ideas presented here are already well practiced in a number of places – I for example have been doing most of these for the last 12 years at different organizations. I got to a point where I did 63 releases in one year (medium and mayor, small where par-for-the-course)

    I like that the systems guys are business oriented and well respected. In most organizations that is not the case. I have always struggled with that and solved it in couple of places but not in all places I have worked for.

    I am not sure about the lack of QA. Maybe I am old school about that. I do agree that developers are capable of writing solid code and they should totally and completely unit test the heck out of the code they write – take ownership of what you do for goodness sakes – but I still feel that a formal QA step is of great importance.

  65. [...] read for the geeks and developers out there: How Facebook ships and develops its code. I like the “Engineers Rule” culture of the [...]

  66. ed says:

    haha… development driven culture doesn’t exist in any tech company in the financial industry… i can only dream of the day when i can check in at will… perhaps when i’m outta this place.

  67. [...] Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

  68. [...] How Facebook Ships Code (tags: facebook development programming culture management) [...]

  69. [...] Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

  70. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

  71. [...] Shared How Facebook Ships Code « FrameThink – Frameworks for Thinking People. [...]

  72. Guido Barosio says:

    Aha, this is what I call a company 100% SOX compliance!

    I wouldn’t put my money here, at all.

  73. [...] Camp” training where they learn the Facebook system by fixing bugs.After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

  74. [...] the same time, a blog post on “How Facebook ships code” has been making the rounds, claiming to detail the company’s “hacker” culture (hacker [...]

  75. Edward says:

    sounds like organised mayhem

  76. [...] Manager Yee Lee talked to a bunch of friends who work at Facebook, and his resulting post on how Facebook ships code makes it sound like the biggest startup in the world. Some of the points in the original post were [...]

  77. [...] Manager Yee Lee talked to a bunch of friends who work at Facebook, and his resulting post on how Facebook ships code makes it sound like the biggest startup in the world. Some of the points in the original post were [...]

  78. arch says:

    I’m curious to understand the function of architecture at FB. To level set, I run enterprise Architecture at my company and I’m constantly seeking clarity on how architecture and architects enable the business and engineering. The article briefly alludes to architecture in FB as an advisory function akin to the designer function being a support function when engineering needs them. So…what do architects get called upon by the engineers to help them out with.

    Traditionally, the word “architect” is viewed very negatively by engineers – something I’m keenly aware of. How is this at FB?

  79. This is one of the best “programming in the real world” posts I have read. Great job!

  80. [...] boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

  81. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  82. [...] boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

  83. Sid says:

    This will work when the following has been taken care of:
    – business strategy & missions clear
    – architects have defined the strategic technology direction to be taken, and identified key technologies that need to be used
    – the company is able to hire & retain highly competent developers over a period of time
    – focus & accountability areas are defined for business, architects, developers

  84. Adam says:

    sounds like something that would work great until the big buyout/ipo, at which point people would stop caring, the threat of being fired wouldn’t carry much weight and i suspect things would go to utter and complete hell.

  85. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

  86. [...] Facebook Ships Code coding, startups, facebook, culture, developer-driven culture… [full post] yeeguy FrameThink – Frameworks for Thinking People business managementproduct [...]

  87. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

  88. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

  89. mikestitch says:

    What does Facebook use to package and deploy PHP code? We currently use git for deployment at my company but we’ve come to the conclusion that using a packaged deployment system would be much better, especially for our cloud-based systems.

  90. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

  91. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  92. [...] How Facebook Ships Code Interessante Einblicke [...]

  93. afyanet says:

    The FB engineers remind me of another group of R&D engineers in Kenya working on GWT UI for ADempiere Opensource ERP client http://www.at.co.ke/blog/trend-analysis/a1erp-the-web-2-0-erp%e2%80%93-the-technology-the-challenge
    Engineers can truly drive UI developments but PM’s still remain key to guild the overall vision of the development.

  94. Piet says:

    So it all comes to this:
    Security is not important.
    Privacy of the user is not important.
    Is is fun to program.

    A developer driven development cycle is not the way to create a useful application. They teach this a long time ago already.
    Thank you for sharing this information. Facebook development is not the way to create applications.

  95. Stuart Rubin says:

    If the Engineers don’t think that their managers do much, and that the Engineers have a lot of freedom, and that the managers aren’t very influential, then my guess is that the managers are really excellent managers! Their job is to keep their team as productive as possible. Believe me, I am sure that the managers are doing TONS of work behind the scenes to let the developers work in the best possible environment and to feel the best about what they do. Those managers are keeping things together!

  96. Kenneth says:

    I do not think so, In my opinion it would hardly work anywhere else. Why ? it all about business strategy, from what i understood, facebook need creativity, and that culture also instill a sense of leadership in each employee, which kind of ensures that only good ideas transform into code. Whereas in most businesses, programming must be aligned with business goals( which comes down from the top, to make sure everything is well coordinated) which somehow kills creativity. what do u guys think?

  97. [...] dream to be a Facebook  developer and be recruited by Mark Zuckerberg? I advice you to take a look here (just for developers). LikeBe the first to like this [...]

  98. [...] Lee在博客上发表了一篇名为How Facebook Ships Code的文章,迅速登上Reddit和Hacker [...]

  99. danno says:

    I remember when I couldn’t create an event in facebook because the state dropdown was empty and yet it was “required”…that remained for days and really sucked! As a developer I am amazed that facebook prospers as it does, it really doesn’t seem so special – beyond having a massive user base. I don’t even find it particularly intutive but I am glad they exist and give the web more press!

  100. [...] That’s one of the things that got me while reading How Facebook Ships Code « FrameThink – Frameworks for Thinking People [...]

  101. nixo says:

    facebook doesn’t strike me as a terribly complex system except for it’s scalability. I don’t think this process would work if they were building an office suite or something like that.

    It’s also the worst development environment I’ve ever seen or even heard of.

    I have an EXE file I built in turbo pascal in 1983 and that same binary will still run on the lastest version of windows on the latest hardware.

    I wrote a facebook app and within 3 months it wasn’t working anymore because facebook doesn’t understand the concept of backward compatibility. This doesn’t strike me as a good feature of their culture being so entitled that they can break their api willy nilly without any regard for the work they’re creating for their VOLUNTEER application developers.

    When the fad wears out they’re going to find that people aren’t so willing to cater to their every api change just because “It’s facebook.”

  102. [...] came an interesting article about Facebook’s internal development operations. It’s a great read. [...]

  103. rtpHarry says:

    I hope this article isn’t accurate because it makes it sound like a real cackhanded way to run things. I mean I get that it sounds like Mark has made his website and is idealistic about the way he wants to keep the power in the devs hands but come on!

    People have specialities, there is toooooo much to learn for a single dev. Making everyone do everything from db through to UI? I’m a developer, but as a regular user of Facebook I’m kind of disappointed at this haphazard way of doing new features. I mean apart from the fact that there aren’t that many new big new features, but I kind of expected there to be some real deep knowledge going into this. Like psychology professors thinking about person to person interactions. And dedicated people that take care of the scaling problems that arise when you are developing for 500 million users.

    I cant believe this is really the way that Facebook is run, there must be some broader picture that isn’t being shown here. Are you seriously saying that Facebook dev division is 500 people just doing what they want? I really want to believe that the best minds in their fields are coming together to produce the website that shapes the lives of so many people.

    Even if they have really high quality developers on the team they just simply cannot gain the depth of knowledge to offer the best if they do the whole gamut. We’ve all heard Seth’s theories of 10,000 hours before you are an expert. Either all the developers are over 60 or they just simply are not experts in what they do.

    I can only guess that the real picture of this is that anyone can seed a feature idea but then teams take over various sub aspects of that feature and apply their various expertise to say, fleshing out the UI design, or optimising the caching, until really its just a nice notion that some developer “invented” a feature, just the same as I’m sure 99% of the code that Mark originally wrote has been replaced by now.

    • nixo says:

      I don’t know if you heard, but most of facebook is written in php. Not exactly the language of geniuses. From what I’ve read, there’s a group of heavy hitters that wrote the php compiler and the javascript reinterpreter thing, but they pride themselves on being wide and short rather than thin and tall if you get what I mean. These guys aren’t writing compilers, or 3d rendering engines, they’re moving text from the database to the screen.

  104. [...] they should know a bit about the science of pricing by nowHow Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility” Powered by Fresh From This entry was posted in [...]

  105. rudy42 says:

    Sounds like the screenplay for “The Social Network 2.0″

  106. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  107. [...] that exposed user data, and the user IDs that rogue apps associated with personal information, every damn engineer at Facebook has access to the database? That’s right, all one thousand engineers can look at [...]

  108. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  109. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

  110. [...] facebook是如何管理代码的 Posted by Niko on January 20, 2011 in Reprinted. 0 Comments Tags: facebook. var jiathis_config = {"data_track_clickback":true}; 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  111. [...] 原文: http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ [...]

  112. [...] 英文原文:How Facebook Ships Code This entry was posted in 流言蜚语. Bookmark the permalink. ← 思考 [...]

  113. motslots says:

    The inmates are running the asylum!
    This goes a long ways to explaining the missteps and lack of real user experience that Facebook has.
    I have over 31 years of experience building commercial software products and I can tell you what’s described here is a recipe for disaster in the long run.
    This sort of cowboy culture can work early in the development conceptual stage of product development but it’s no way to run a business.
    Developers are not your typical users and are generally terrible at building usable products.

  114. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

  115. [...] A really nice article, on how Facebook ships code. I am not sure if it is 100% correct but even it has some truth in it, it should give other companies a clue about how to treat engineers. [...]

  116. Christoph says:

    With so many corrections one must ask itself: Based on what information did the author come up with this?

    “Oh, they have QA and UnitTests bot nobody uses them”. Bam, some other guy writes the exact opposite: “No, we use UnitTests all over the place.”

    “If you make many mistakes and get blamed now and then, by time you will eventually get fired” — “Oh no, nobody here gets fired for mistakes!”

    Seriously, WTF?! Do you actually know *anything* about the internals of FB or did you just want to write something about FB, because, well, to be cool and stuff? There’s nothing wrong with comments that add information and clearify some points, but those totally contradicted your statements, rendering this whole article useless.

    Just a big game of bullshit bingo going on here, IMHO.

    PS: No, I don’t know anything myself about FB’s internals. But at least I’m not going around and writing articles with made up details.

  117. blackeye says:

    Emm… In this case, “developer driven” sounds like a happy title for a sloppy development process covered by a *very good* operations process. It’d be really interesting to get an idea of the ratio of bounced pushes to successful full-production releases.

  118. takeshi says:

    Trying again to post a comment… somehow it doesn’t seem to work…
    I posted a translation of your article in Japanese.

    http://74k3.blogspot.com/2011/01/read-interesting-article-on-facebook.html

    • takeshi says:

      Okay, it worked with the Website field empty…

      Thanks for a very interesting article. It’s great to know how exactly developers supporting such a huge number of users work. I really enjoyed reading the article. As for some topics, I thought it would be more interesting to see them in detail, like how much super-productive they are supposed to be, how is a public shaming specifically, etc.
      Now I wonder how/if the environment have changed since then and if it’s working as expected.

  119. [...] according to this post Facebook simply lets the code live on a microcosm of their users and sees how they react. This is [...]

  120. [...] a tantalizing look into the different processes and practices Facebook uses to build and release code. Granted, it’s a set of notes culled from second-hand information, it’s still an [...]

  121. FAcebook has to be a very complicated system because it looks really simple… That said think about the amount of functionality it provides, AND that it processes probably bilions of information a day. Thing is a lot of this stuff is going behind the scenes. Have You noticed that news feed only feeds information with only the info about friends You have recently interacted… There is a lot of initiative going up there – search, events, privacy model, platform, internal data warehouse, analytics… soon probably geo features and even more mobile support.
    Do not forget that they have introduced a payment system (it NEEDS coding, security…) and many other stuff. And they claim it’s only 50 devs. That’s small amount. Very agile though. And very smart, but still it sounds like to organization is very RIGID in its agility.

    HAve You noticed that the only part of information You can actually edit is Your profile (which only You can edit obviously) and small objects like events. This very smart of them (think about propagation of changes…) and makes the problem of advanced rights management and access managment go away.

    I truly admire FB for its model (true also for google and android). And even though I love MS (.net rocks) it can’t be that agile in the consumer world (but is very good in business which will probably be it’s target market or at least where the money is.)

  122. Al Lee says:

    Describing Facebook as not having QA is inaccurate; the operations team is “Quality Assurance”. No similated (test) environment will tell you how live users will react to most changes; better to have your operations team measure what happens.

    Some features (like, e.g., search ranking improvements at Google) may be testable via replaying logs, which the dev should do anyway in developing the feature.

    For the rest, the operations team having great user behavior metrics, live feedback on A/B tests, the ability to incremental rollout, and the ability to patch errors “instantly” on the live site, is much better than any traditional test team for Facebook.

    Facebook isn’t shipped Office 95 on a gold CD. The old methodology of large internal test orgs simulating the user experience, rather than just measuring the actual user experience, no longer makes sense.

    Very useful post: we have been moving in this direction for testing at PayScale, but it is great to see we aren’t the only ones.

  123. Jason says:

    this is amazing… really great stuff.

  124. darwinl says:

    First of all, very impressed with the process. Particularly with the OPS metrics/monitoring. Looks like they truly achieve development / release agility, particularly with a huge and complex service like Facebook.

  125. [...] simply did not care about privacy. I am not sure, but if you read Skype Manager Yee Lee’s post on “How Facebook Ships Code” one could argue that the way in which the organization is set up certainly lends itself to some of [...]

  126. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  127. [...] Weit gefehlt. Bei Facebook ist jede Art der Entwicklung unheilschwanger. Jeder neue Funktion kann im absoluten Supergau enden. Das belegen zumindest die Veröffentlichungen eines Internetprofis, der Aussagen und Fakten diverser Facebook Mitarbeiter in seinem Blog sammelte. [...]

  128. [...] interesting post about the development process at Facebook. Read it here. Share/Bookmark This entry was written by aft, posted on January 21, 2011 at 8:22 am, filed [...]

  129. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People RT @Jason: killer article on How @Facebook/@finkd Ships Code « FrameThink – Frameworks for Thinking People [...]

  130. [...] just read a very interesting blog entry about the dev operations are in facebook here [...]

  131. [...] How Facebook Ships Code [...]

  132. Jimbo says:

    Hmm … engineering driven culture good … public shaming … um … what’s next? Flogging with a whip?

    And yes, there is no QA based on the edits to this article. Engineers testing their own code is not QA it’s just retarded. No wonder every release has bugs and lately has cause the site to go down.

  133. [...] How Facebook Ships Code – 50% of the company is ops/engineering, 90k servers, 9 levels of rollout, no QA. [...]

  134. [...] How Facebook Ships CodeVery interesting insights into Facebook’s developer driven culture. [...]

  135. This article and one first referenced(concerning Mahalo) were INCREDIBLE! Thanks so much for writing it. A developer-driven culture is exactly what my employer needs to become and here is why:

    1. I’m on a project where the business analyst (who owns the project) announced the product on May 3, 2010, but never got completed requirements done until Jan 21, 2011. Our 1st designer quit for personal reasons and our 2nd designer is grudingly moving along…not liking to make frequent UI designs.

    I’m the single developer/architect for a product that may have 10,000 plus users, all the while doing maintenance on other projects. Priority is based on the squeaky wheel theory.

    We have two QA teams at our firm. The fitst QA team has only found 1 bug in all my projects they’ve tested. The 2nd QA team has tested two other projects : they had one project for 6 weeks and found 0 bugs and another project they had for 4 weeks and found 1.

    In a developer-driven culture we’d have saved on the QA and designers. These two articles really opened my eyes. Now to find such places in Chicago.

  136. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People – [...]

  137. [...] How Facebook Ships Code 中文:Facebook是如何管理代碼的 [...]

  138. [...] »自动草稿By Aaron, on 一月 24th, 2011英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的全文转载如下: [...]

  139. [...] denkt. Letzte Woche veröffentlichten Frame Think in ihrem Blog den Artikel “How Facebook Ships Code“. Einfach zusammengefasst: Jeder Entwickler kann frei drauflos programmieren, solange er nur [...]

  140. [...] How Facebook ships code. Facebook Management. [...]

  141. iron_3gs의 생각…

    How Facebook Ships Code « FrameThink – Frameworks for Thinking People…

  142. [...] think this post is a very interesting look into how Facebook does or does not do things like CM, testing, code [...]

  143. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

  144. [...] It came to me as a shock when I read “Managers are kind of useless here, we don’t have real QAs here (Facebook)”. [...]

  145. Very interesting article. Seems like a very unique culture that they have developed that would definately succeed in motivating staff. Especially letting them choose which projects they want to work on!

  146. prashant says:

    Developer driven culture brings innovation. Visible from the growth facebook had over years, adding new new features by the day.

  147. I have an experience works at software project developing Social Network. It’s really hard to develop and maintain it.

  148. 만박의 생각…

    페이스북의 개발문화에 대한 글, How Facebook Ships Code (번역문) “극단적으로 능동적인 엔지니어, 모든 개발자가 테스트/버그수정/유지보수까지 자기 코드에 대해 책임. 별도의 QA가 없다. 개발자가 UI 구현부터 테스트까지 모두 책임을 지는 구조.”…

  149. [...] or developer-centric organizations is being actively discuss lately thanks to the likes of Facebook. Some people, mostly developers, love it for its freedom and lack of heavy processes. I love it too [...]

  150. [...] : littlebigdetails, un Tumblr qui recense les détails pratiques des interfaces ; – Les coulisses de Facebook, du coté des [...]

  151. [...] Recently Slashdotted article about how Facebook ships code [...]

  152. [...] как организован процесс разработки программного [...]

  153. [...] Follow this link: How Facebook Ships Code « FrameThink – Frameworks for Thinking People [...]

  154. [...] found this post very interesting: How Facebook Ships Code. In short, it explains how Facebook is a developer-centric company. Whether you’re a web-dev, [...]

  155. [...] Around the same time that I wrote my last post “How to make Agile work for product development… It’s all about the people” a colleague sent me a link to an interesting and insightful post by @yeeguy on “How Facebook Ships Code” [...]

  156. Stuart Lynn says:

    Very insightful post, riffed a few comments, added a few of my own observations, and linked back to your post over here http://stuartlynn.co.uk/2011/02/02/agile-ultra-agile-or-anarchy-an-inside-look-at-facebook-development/

  157. [...] How Facebook Ships Code [...]

  158. Doesn't matter says:

    I really don’t find anything unique here, this is process people follow in every Start-up until they fail and then they get back to normal and standard process… Face-book fluke in success is just prolonged and it wont take them much time to change as well.

  159. [...] текст статьи, на английском языке, можно прочитать по этой ссылке. Думаю, что она будет интересна не только [...]

  160. Rei says:

    Makes me cry. /sigh

  161. Glen says:

    I suggest a new post, with all the corrections applied inline. The current display is difficult to follow.

  162. [...] 如何发布代码 (How Facebook Ships Code 译文) 2011-02-11添加留言 按:这篇 How Facebook Ships Code [...]

  163. [...] en suposiciones o cosas que leyó/escuchó en alguna parte, me ha parecido interesante el documento How Facebook Ships Code. Cita citable: Most engineers are capable of writing bug-free code. It’s just that they don’t [...]

  164. [...] 上個月國外流傳著一篇「How Facebook Ships Code」,描述Facebook內部如何管理程式碼,把Facebook描述成軟體工程師的天堂。文章傳來傳去,不但有了簡體中文翻譯版,最近連我的一些朋友的mailing list都在傳。基本上我是很願意相信Facebook有著良好的程式管理制度,但是這篇文章的內容實在也太神奇了。這兩天本來想要戳一下這篇,沒想到翻翻原文,底下的討論區早已經有許多人〈包括Facebook的工程師〉也提出質疑,到最後原文作者也修改了不少內容。也罷,那麼今天這個就來看這篇文章到底有哪些問題。 [...]

  165. [...] came an interesting article about Facebook’s internal development operations. It’s a great [...]

  166. [...] 英文原文:How Facebook Ships Code [...]

  167. [...] do the exact opposite, and I admire them for it. They allow the programmers free reign to do their work whatever way they [...]

  168. [...] on to the video above there is this article. Which could use some editing, but is very informative. http://framethink.wordpress.com/… Some pretty damn disruptive and clever concepts in there:1) No dedicated QA, for realz2) Any [...]

  169. [...] How Facebook ships code is an interesting peep into both the organization culture and the development process inside Facebook. A very interesting read! Share this post [...]

  170. [...] שמכירים את החברה (רוצים ללמוד יותר? הציצו לתוך הפוסטים: איך פייסבוק מדלוורת קוד, מה עובדי פייסבוק חושבים על הפוסט הזה ומה מנהל [...]

  171. [...] How Facebook Ships Code « FrameThink – These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. [...]

  172. [...] na deweloperów. Podobna kultura panuje (podobno) w Facebooku. Została ona opisana w tym poście. W skrócie jest to miejsce, w którym deweloperzy sami decydują nad czym chcą pracować i są w [...]

  173. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  174. Chris Chan says:

    Here’s a post which contains a link to a great detailed case study of Facebook with comments from Don Reinertsen, Alistair Cockburn, Jeff Sutherland, Yishan Wong (former Director of Engineering at Facebook).

    http://c2reflexions.com/2011/03/23/how-people-and-process-enabled-facebook-to-become-a-phenomenal-company/

  175. James says:

    This approach certainly does not apply to other environment.

    For one, FB is not a mission critical application. If you update your friend’s wall but it doesn’t show up in half an hour (occasionally), nobody would notice. Think about that if you are submitting an order online…

    The only reason why FB can let dev push out code freely is because they can afford to. Many businesses/applications do not have that luxury.

  176. [...] How Facebook Ships Code – Facebook has over 650 million users and over 60 000 servers. Integration, testing and deploying a new version of the application must a daunting task. This article explains how they do it. [...]

  177. [...] 文章转自:http://www.dbanotes.net/arch/facebook_how_facebook_ships_code.html 按:这篇 How Facebook Ships Code [...]

  178. Montagist says:

    Facebook sounds like Voltron.

  179. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  180. [...] et souvent au péril de ses utilisateurs. On doit s’attendre à cela lorsqu’on en sait plus sur la culture de développement de l’entreprise: développer le produit le plus rapidement possible, quitte à négliger la documentation destiné [...]

  181. [...] 페이스북에서 어떤 식으로 코드를 만들어 쉽하는 지에 대해 인터넷에 회자된 글이다. 그 중에서 인상 깊은 아이템들 몇 개를 추려와서 밑줄을 그어 봤다. [...]

  182. [...] שמכירים את החברה (רוצים ללמוד יותר? הציצו לתוך הפוסטים: איך פייסבוק מדלוורת קוד, מה עובדי פייסבוק חושבים על הפוסט הזה ומה מנהל הפיתוח [...]

  183. [...] boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

  184. 쯔이의 생각…

    Interesting Article : “How Facebook Ships Code” by yeeguy (2011. Jan)…

  185. [...] Posted: January 17, 2011 | Author: yeeguy | Filed under: business management, product management, social networks,startups | 257 Comments » [...]

  186. [...] product). It doesn’t matter if you are  doing short release cycles for web products (such as facebook – see here) or much longer ship cycles for larger products such as Microsoft [...]

  187. [...] entre culture américaine et française s’exprime. Cet article est une synthèse d’un article publié aux US . Il a été publié grâce aux relations de l’auteur avec certains membres du staff [...]

  188. Facebook is as past the curve as Skype. Forget about how “unique” and “special” it is. It’s going the way of the dodo in the next 5 years. MySpace will be gone sooner.

    Man, you geeks really get excited about NOTHING don’t you…

  189. Amanda says:

    Appreciate your inforamtion. It is just too great!

  190. [...] 【原文首发于《FrameThink》(英文版,需翻墙),感谢老崔的推荐(译文在此),推荐者说:“看完之后终于明白为什么优秀的工程师都去了或想去facebook,因为那里是工程师们的天堂。】 [...]

  191. [...] one person is the key to fixing many issues, it is a sign of a software process gone awry. This is how Facebook ships code, and it’s worth a critical [...]

  192. Wonkyung Lee says:

    Good for understanding how facebook works!

  193. [...] no shortage of articles about Facebook these days, but I suggest you read this interesting take on How Facebook Ships Code. According to the author, over 50 percent of Facebook employees reside in the Engineering or [...]

  194. [...] “se ne parla assieme“, “dipende dal mercato” ecc. Consiglio la lettura di come funzionano le cose in Facebook, dove il terzo approccio sembra essere quello applicato.  Personalmente la quarta e’ la [...]

  195. [...] 英文原文链接:http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ [...]

  196. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  197. [...] 本文是从 How Facebook Ships Code 这篇文章翻译而来。 [...]

  198. [...] 31, 2011 at 04:10PM Aqee 本文是从 How Facebook Ships Code [...]

  199. Waldi Syafei says:

    What a great IT development they have..

  200. lvp says:

    wow, beautiful mind!

  201. [...] then, don’t trust me on this one. Read, for example, how Facebook delivers code. LinkedIn releases software multiple times a day. (Disclosure: Greylock Partners is an investor in [...]

  202. [...] 3、开什么调试,直接看代码 有些人总是很牛,他们不用调试,不用测试,写好的代码只要编译通过,经过他们的大脑检测“通过”,他们就敢把这些更新提交到线上 如果是这类人员,没什么好说的,建议看看《Facebook是如何管理代码的》,原文《How Facebook Ships Code》 可能我们不需要很严格的审查,但是基本的测试还是必要的 你这样做只会浪费更多人力物力,当然有可能是紧急Bug的修改,偶尔可以直接在线上改,然后测 做这行就要专业点,前人的经验还是有用的,测试是必须的 [...]

  203. Nice topic…you may be interested to know about Social India Conference 2011 in Bangalore,India which is organized by Akshay Patra Foundation to raise funds for a non-profit cause. The event brings together world’s well known social media speakers at the event…visit
    http://www.socialindiaconference.in for more details

  204. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  205. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

  206. [...] Share framethink.wordpress.com       8 months [...]

  207. [...] features, live, at any time. From Google they borrow a free-ranging engineering-led culture, but with almost zero central coordination. “[E]ngineers can modify specs mid-process, re-order work projects, and inject new feature ideas [...]

  208. [...] What happens inside Facebook Engineering: How Facebook Ships Code [...]

  209. [...] his article How Facebook Ships Code, Yee Lee notes that at [...]

  210. [...] no shortage of articles about Facebook these days, but I suggest you read this interesting take on How Facebook Ships Code. According to the author, over 50 percent of Facebook employees reside in the Engineering or [...]

  211. [...] вам статью об особенностях работы внутри FB по мотивам вот этого бурного [...]

  212. […] Interesting article about developer-driven culture: How Facebook Ships Code […]

  213. […] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

  214. […] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

  215. […] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

  216. […] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

  217. […] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

  218. […] On th&#1077 desktop, Facebook &#1110&#1109 a machine designed t&#959 m&#1072k&#1077 itself better. Th&#1077&#1091 hoover up data &#1072b&#959&#965t h&#959w &#1091&#959&#965′re using &#1077&#957&#1077r&#1091 piece &#959f th&#1077 service. Th&#1077&#1091 A/B (&#959r A/X) test &#1077&#957&#1077r&#1091 single &#1088&#1072rt &#959f th&#1077 user experience. Based &#959n &#1072&#406&#406 th&#1072t feedback, th&#1077&#1091 tweak &#1072n&#1281 tweak &#1072n&#1281 tweak. Th&#1077n, th&#1077&#1091 &#1281&#959 &#1110t &#1072&#406&#406 over again. Disputes &#1089&#1072n b&#1077 settled b&#1091 simply testing alternatives &#959n a small base &#959f users: m&#1072&#1091 th&#1077 best data win. An&#1281 th&#1077&#1091 &#1089&#1072n &#1281&#959 &#1072&#406&#406 th&#1110&#1109 quickly b&#1077&#1089&#1072&#965&#1109&#1077 &#959n th&#1077 web, Facebook &#1089&#1072n change th&#1077 code &#1072n&#1091 time th&#1077&#1091 damn well please (although mostly once a week &#959n Tuesdays). […]

  219. […] On the desktop, Facebook is a machine designed to make itself better. They hoover up data about how you’re using every piece of the service. They A/B (or A/X) test every single part of the user experience. Based on all that feedback, they tweak and tweak and tweak. Then, they do it all over again. Disputes can be settled by simply testing alternatives on a small base of users: may the best data win. And they can do all this quickly because on the web, Facebook can change the code any time they damn well please (although mostly once a week on Tuesdays). […]

  220. […] 源:http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ […]

  221. […] 之前网上有篇文章How Facebook Ships Code谈到了Facebook内部运作的很多细节,感兴趣的同学可以阅读,与传统的企业相比确实有很多值得借鉴的地方。 […]

  222. […] 英文原文: How Facebook Ships Code […]

  223. […] voltados para software online como Facebook, Twitter e Netflix continuam a crescer e a inovar a partir daquilo que já fizeram, ao investir em […]

  224. […] process, especially early on in their growth. Wrote up this blog post on that topic in Jan 2011:http://framethink.wordpress.com/…I'm sure their PD process has evolved since then, especially now that they've gone public […]

  225. […] Facebook ships code link […]

  226. […] framethink.wordpress.com […]

  227. […] become part of the culture. For example, at Facebook engineers can change anybody else’s code without asking. And with Microsoft integrating Yammer into Office 365, collaborating on writing will become the […]

  228. […] How Facebook Ships Code […]

  229. […] são feitas em grupo, mas de forma contínua. Leiam este paper “How Facebook ships code” em http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/. Será que este conceito só aplica ao Facebook? Também no mundo das apps os upgrades são […]

Follow

Get every new post delivered to your Inbox.

Join 2,420 other followers