Facebook and Mark Zuckerberg have established near monopoly control over the language that we use to talk about social networks. Think about the words that we use to describe social network sites — terms such as “like”, “share”, “connect”, “transparent”, “more open”, etc. I’m not saying that Facebook were the first company to use these terms and they certainly didn’t invent them. But I do believe that Facebook have been the most effective and the most consistent at using those words to frame the conversation about social network policies and features in moral terms.
The moral framework they’re creating around their social network site is really key to focus on. Facebook are saying: It’s *good* to be liked, right? It’s *good* to connect to more people, right? You’re a *better* person if you’re willing to share more transparently, right? And, conversely, if you’re *not* willing to share a thought or action openly, then what are you hiding; are you doing something that you *shouldn’t* be doing?
Facebook have purposefully created this moral linguistic context for their service. As Zuckerberg masterfully puts it:
At Facebook we have a deeply held purpose […] Our mission: “Give people the power to share and make the world more open and connected.” In the world we’re building where the world is more transparent, it becomes good for people to be good to each other. That’s really important as we try to solve some of the world’s problems.
George Lakoff, professor of linguistics and cognitive science tells us that, in any debate, if one side manages to dominate the moral framing of the issues, then they’ve already won the debate before it even begins. Facebook appear to have learned Lakoff’s lessons well. By framing the debate about social network policies and features in their own moral terms, Facebook could be walking away with victorious dominance over the social internet without firing a shot! Every other social internet company has been forced to define themselves using Facebook’s terminology of “sharing”, “openness” and “social graphs” (i.e., attempting to beat Facebook at their own game) or to contrast themselves against Facebook’s feature functionality (i.e., attempting to convince the world that sharing, openness, and transparency are bad things). It’s a lose-lose proposition either way and that makes it really hard for other companies to market and differentiate themselves using the language and concepts that Facebook own.
What are your thoughts on how the social network industry should move forward from here? I’ll jot down some thoughts in my next blog post. Would love to hear your thoughts in the meantime.
I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software.
Seems like others are also interested in Facebook… The company’s developer-driven culture is coming under greater public scrutiny and other companies are grappling with if/how to implement developer-driven culture. The company is pretty secretive about its internal processes, though. Facebook’s Engineering team releases public Notes on new features and some internal systems, but these are mostly “what” kinds of articles, not “how”… So it’s not easy for outsiders to see how Facebook is able to innovate and optimize their service so much more effectively than other companies. In my own attempt as an outsider to understand more about how Facebook operates, I assembled these observations over a period of months. Out of respect for the privacy of my sources, I’ve removed all names and mention of specific features/products. And I’ve also waited for over six months to publish these notes, so they’re surely a bit out-of-date. I hope that releasing these notes will help shed some light on how Facebook has managed to push decision-making “down” in its organization without descending into chaos… It’s hard to argue with Facebook’s results or the coherence of Facebook’s product offerings. I think and hope that many consumer internet companies can learn from Facebook’s example.
- as of June 2010, the company has nearly 2000 employees, up from roughly 1100 employees 10 months ago. Nearly doubling staff in under a year!
- the two largest teams are Engineering and Ops, with roughly 400-500 team members each. Between the two they make up about 50% of the company.
- product manager to engineer ratio is roughly 1-to-7 or 1-to-10
- all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs and listening to lectures given by more senior/tenured engineers. estimate 10% of each boot camp’s trainee class don’t make it and are counseled out of the organization.
- after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data)
- [EDIT thx fryfrog] “There are also very good safe guards in place to prevent anyone at the company from doing the horrible sorts of things you can imagine people have the power to do being on the inside. If you have to “become” someone who is asking for support, this is logged along with a reason and closely reviewed. Straying here is not tolerated, period.”
- any engineer can modify any part of FB’s code base and check-in at-will
- very engineering driven culture. “product managers are essentially useless here.” is a quote from an engineer. engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. [EDITORIAL] The author of this blog post is a product manager, so this sentiment really caught my attention. As you’ll see in the rest of these notes, though, it’s apparent that Facebook’s culture has really embraced product management practices so it’s not as though the role of product management is somehow ignored or omitted. Rather, the culture of the company seems to be set so that *everyone* feels responsibility for the product.
- during monthly cross-team meetings, the engineers are the ones who present progress reports. product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that “product spoke too much at the last meeting.” they really want engineers to publicly own products and be the main point of contact for the things they built.
- resourcing for projects is purely voluntary.Engineers decide which ones sound interesting to work on. a PM lobbies group of engineers, tries to get them excited about their ideas. Engineer talks to their manager, says “I’d like to work on these 5 things this week.” Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
- arguments about whether or not a feature idea is worth doing or not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.
- engineers generally want to work on infrastructure, scalability and “hard problems” — that’s where all the prestige is. can be hard to get engineers excited about working on front-end projects and user interfaces. this is the opposite of what you find in some consumer businesses where everyone wants to work on stuff that customers touch so you can point to a particular user experience and say “I built that.” At facebook, the back-end stuff like news feed algorithms, ad-targeting algorithms, memcache optimizations, etc. are the juicy projects that engineers want.
commits that affect certain high-priority features (e.g., news feed) get code reviewed before merge.News Feed is important enough that Zuckerberg reviews any changes to it, but that’s an exceptional case.
- [CORRECTION — thx epriest] “There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn’t look at every change personally.”
- [CORRECTION thx fryfrog] “All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don’t invite them to. It would take intentionally malicious behavior to get un-reviewed code in.”
no QA at all, zero. engineers responsible for testing, bug fixes, and post-launch maintenance of their own work. there are some unit-testing and integration-testing frameworks available, but only sporadically used.
- [CORRECTION thx fryfrog] “I would also add that we do have QA, just not an official QA group. Every employee at an office or connected via VPN is using a version of the site that includes all the changes that are next in line to go out. This version is updated frequently and is usually 1-12 hours ahead of what the world sees. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon.”
- re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.” [EDIT: please note that this was subjective opinion, I chose to include it in this post because of the stark contrast that this draws with standard development practice at other companies]
- [CORRECTION thx epriest] “We have automated testing, including “push-blocking” tests which must pass before the release goes out. We absolutely do not believe “most engineers are capable of writing bug-free code”, much less that this is a reasonable notion to base a business upon.”
- re: surprise at lack of PM influence/control — product managers have a lot of independence and freedom. The key to being influential is to have really good relationships with engineering managers. Need to be technical enough not to suggest stupid ideas. Aside from that, there’s no need to ask for any permission or pass any reviews when establishing roadmaps/backlogs. There are relatively few PMs, but they all feel like they have responsibility for a really important and personally-interesting area of the company.
- by default all code commits get packaged into weekly releases (tuesdays)
- with extra effort, changes can go out same day
- tuesday code releases require all engineers who committed code in that week’s release candidate to be on-site
- engineers must be present in a specific IRC channel for “roll call” before the release begins or else suffer a public “shaming”
- ops team runs code releases by gradually rolling code outthere are 9
concentriclevels for rolling out new code
- facebook has around 60,000 servers
- [CORRECTION thx epriest] “The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.”
- the smallest level is only 6 servers
- e.g., new tuesday release is rolled out to 6 servers (level 1), ops team then observes those 6 servers and make sure that they are behaving correctly before rolling forward to the next level.
- if a release is causing any issues (e.g., throwing errors, etc.) then push is halted. the engineer who committed the offending changeset is paged to fix the problem. and then the release starts over again at level 1.
- so a release may go thru levels repeatedly: 1-2-3-fix. back to 1. 1-2-3-4-5-fix. back to 1. 1-2-3-4-5-6-7-8-9.
- ops team is really well-trained, well-respected, and very business-aware. their server metrics go beyond the usual error logs, load & memory utilization stats — also include user behavior. E.g., if a new release changes the percentage of users who engage with Facebook features, the ops team will see that in their metrics and may stop a release for that reason so they can investigate.
- during the release process, ops team uses an IRC-based paging system that can ping individual engineers via Facebook, email, IRC, IM, and SMS if needed to get their attention. not responding to ops team results in public shaming.
- once code has rolled out to level 9 and is stable, then done with weekly push.
- if a feature doesn’t get coded in time for a particular weekly push, it’s not that big a deal (unless there are hard external dependencies) — features will just generally get shipped whenever they’re completed.
- getting svn-blamed, publicly shamed, or slipping projects too often will result in an engineer getting fired. “it’s a very high performance culture”. people that aren’t productive or aren’t super talented really stick out. Managers will literally take poor performers aside within 6 months of hiring and say “this just isn’t working out, you’re not a good culture fit”. this actually applies at every level of the company, even C-level and VP-level hires have been quickly dismissed if they aren’t super productive.
- [CORRECTION, thx epriest] “People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren’t around to support them in case something goes wrong (and haven’t found someone to cover for you).”
- [CORRECTION, thx epriest] “Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.”
- [CORRECTION, thx fryfrog] “I also don’t know of anyone who has been fired for making mistakes like are mentioned in the article. I know of people who have inadvertently taken down the site. They work hard to fix what ever caused the problem and everyone learns from it. The public shaming is far more effective than fear of being fired, in my opinion.”
It’ll be super interesting to see how Facebook’s development culture evolves over time — and especially to see if the culture can continue scaling as the company grows into the thousands-of-employees.
What do you think? Would “developer-driven culture” work at your company?
- Jay Walker http://www.ted.com/talks/jay_walker_on_the_world_s_english_mania.html
- Aimee Mullins http://www.ted.com/talks/aimee_mullins_prosthetic_aesthetics.html
- Cryptozoo http://cryptozoo.ning.com/
- “Today is the day”
- “We are the search engine to our solution”
- “public intellectuals help us reflect on who we are and who we should be as individuals and as a society”
- concept: “cause-based movements/companies”
- 3 things to look for in startups: “scalability, margins, defensibility”
- “web 1.0 was about going online to this different place [AOL chatroom] where weird behavior happens, web 2.0 is about real identities”
- “internet: cheap, scalable infrastructure to reach millions of people”
- “we use the same language to talk about cause-based movements as the internet: building ‘networks’, ‘marketplaces’, etc.”
- “the fastest growing internet corporations are ‘pure plays’ — entirely online. It’s harder to get people to actually change what they’re going to do on a Saturday afternoon. But as people get more familiar with being in a network, being in a marketplace, that [behavior change] will become easier. And it will be sustainable because the low-cost, scalable infrastructure of the Internet helps [reinforce] that behavior.”
- concept: “large scale group coordination & distributed effort”
- quote: “the future is sooner and stranger than you think”
- concept: “collective problem identification, crowd-sourced ideation”
Leila Janah, Samasource
- “here in the US, it’s easy to think that we live in a meritocracy”
- “well meaning outsiders have created a culture of handouts [in Ghana]”
- “we are witnessing a tremendous surge in human capacity — global literacy on the rise & 90% decrease in cost to get online.”
- “digital microwork: crowd-sourced labor, small tasks worth a few cents each. this is the future of work.”
- “future is about humans and computers working together to get stuff done”
- “people don’t need charity, they need a decent way to make a living”
- “building social bridges with art”
- “bridges: art & science, culture & technology”
- concept: “negative space: the space between brush strokes, it’s what’s NOT there. it’s inherently participatory and collaborative. it’s where imagination can play.”
- “social entrepreneurship is about building bridges. art is uniquely empowered to build those bridges.”
- concept: tension of top-down vs. bottom-up organization in social change… MLK = top-down, prophetic visionary, difficult to emulate. Bob Moses = bottom-up organizer, empowering individuals
- “organizer: building self-reliant, grassroots leaders.”
- “organizer: our job is to work ourselves out of a job”
- “skilled organizing can create an enormous amount of energy from poor people with small resources”
- “our movement may have still happened without King, but it would have been a very different kind of movement”
- “good organizing, good teaching sets free the best qualities of those who have never been allowed to shine”
Peter Hirschberg & Josette Melchor
- “the meeting place of art & technology”
- “photography — the quintessential meeting of art and technology — initially dissed by French literacy establishment”
- “impressionism — the new ‘platform’ or ‘ecosystem’ of its day — also panned on first reviews”
- new art doesn’t fit labels, categories, critics can’t write about, people don’t understand, it gets shunned
- “what’s been bubbling up in the silicon valley technology community is now spilling over into the art community”
- [Emperor’s March from Star Wars] “Do you think of Darth Vader? Luke? I think of the Stormtroopers! They really made things work.”
- “We are obsessed with leadership. Google ‘CEO’ and you get 4MM hits. Google ‘COO’ and you get far fewer…”
- “We are obsessed with ‘new’, perhaps at the expense of ‘impact’ — we don’t need another new cancer oragnization, we need the 300 existing ones to work better together.”
- “You have permission to JOIN something [instead of ‘founding’]. Permission to make IMPACT. Permission to coin a new title: ‘DO-er'”
- “On the CNN map, borders look solid, permanent — between us and them. Up close, borders are fluid; they’re social.”
- “Less than 10% of us cross borders for tourism. [Correlation to tourist dollars.] 10% of countries take 70% of tourist receipts.”
- “273 cross-border groundwater aquifers — governments need to manage these with each other. higher potential for conflict.”
- thought-provoking: “Who in this room spends time with their friends in East Palo Alto? The borders that we draw in Silicon Valley have just as much impact as the ones in Gaza.”
Yvonne Lee Schultz
- what happens when you turn guns into chocolate?
Alberto Vollmer – CEO of leading Venezuelan rum distillery & bottler
- Project Alcatraz – criminal re-insertion into employment
- [opposing violent criminal gangs in Venezuela] “We have to retaliate, but it has to be creative; it can’t be violent.”
- [presenting a recruiting pitch in middle of gang-dominated slum] “I came here to make you dream. Let’s talk about what kind of TV you’re going to have, what do you want for your kids, what kind of restaurants will be here.”
- quote: “hope is so much more powerful than fear, it will always win”
Victor Tsaran – advocate for the blind, musician, Yahoo accessibility lead
- “Transformation through technology. Using technology to overcome perceptions of what is possible.”
- quote: “Life is upbeat, rhythmic, and full of surprises”
- “We create technology to make life meaningful. With technology, we can do certain things that we could not without technology.”
- [teens in Bosnia were tired of foreign journalists coming in to steal their stories] “They wanted to tell their own stories, in their own voices”
- “The #1 fear in this country is public speaking, more than death”
- Poll: which disaster is most likely?
- * robots kill/enslave humans
- * biotech-driven pandemic
- * runaway nanotech gray goo
- * nuclear war
- * governments use computers to control everyone
- * runaway global warming
- * singularity takes too long to happen <== Peter’s worried about this
- “our entire civilization is predicated on accelerating technological change”
- “example: retirement planning is based on assumption of 8.5% annual return on investments. that assumption is based on last 100 years of economic data. problem: last 100 years have been a time of incredible accelerating technological improvement. if you ran that same analysis on 1200-1300 A.D. or the last 100 years of the Roman Empire, you’d be lucky to get 0% return, just to keep your money. Trouble is that the pace of technological progress seems to be slowing…”
- “people seem to be working harder, running really hard just to stay in place.”
- “when people talk about progress in the developing world — China, India — it’s always a 20 year story. Life there will be so much better in the next 20 years because they’ll get all the things that the developed world has now. But when we talk about progress in the developed countries, it’s always on 6 month time scale. Is the recession over? What’s the market going to be like in 6 months? Why do we not talk about how life will be dramatically better in the developed world in 20 years?”
- “of 538 congress people, 11 have degrees in engineering. they all think that ‘science and technology are solved by other people.’ but if everyone thinks that, then that’s a problem.”
Thomas Goetz, ex-Wired editor, Public Health MA
- “Life is an experiment. How can we turn this into more of a controlled experiment?”
- Health information is dumped on us, scattered, disconnected. Not effective. Health problems are increasing in US.
- “Hawthorne effect — when people know they are being observed, they changed behavior. Usually experimenters want to get rid of the Hawthorne effect. But Kansas City orthodontist experiment shows that we can use Hawthorne effect to drive behavior change [brushing teeth more with ‘experimental’ toothpaste]. Also applies to weight loss [e.g., taking daily photo of your weight on scale].”
- “Whitehall experiment — controlling for diet and habits, people of high social standing were 2X less likely to die of heart disease than those of low social standing. Why? Control. People at top of social ladder have more control over their lives than those at the bottom.”
- “Feedback and Groups are effective, powerful driver of behavior change.”
- “Need a control tool: an algorithm, a decision tree that anyone can follow.”
- “We want tools and techniques that have minimal friction and scale.”
- “Data means more when it’s our own.”
- “The more we mind our health, the better we are.”
- “Don’t just make people responsible, but give them the tools and data to make behavioral change.”
David de Rothschild – explorer, both poles
- “When people ask about expeditions, the question that has stuck with me: ‘How was it out there?’ What does that mean, ‘out there’?”
- “Kids are learning about nature, but they’re not out there, in it.”
- “Nature Defiancy Disorder”
- “We view nature as chaotic, intimidating. But we like to think things in a linear fashion, things have a start and an end.”
- “We are starting to manufacture nature at the expense of nature itself.”
- “GDP is a deception — we treat the cost on nature as an externality. It’s like kid who puts up a lemonade stand, says they made $20 when the lemons, the jug, the sugar, all the materials were paid for by his mom.”
- “when you undertake a dream it becomes an adventure. adventures create stories. humanity is based on stories. stories become the inspiration for more dreams.” [sidenote: that’s a viral loop]
- “humanity always wants to take the path of least resistance. curiousity and innovation may lead you down a different path, NOT the path of least resistance.”
- “you never change something by fighting existing reality. you change things by building a new model that makes the existing model obsolete.” [Buckminster Fuller quote]
Andrew Hessel – “the end of cancer” & PinkArmy.org
- “computer systems are analogy for biological systems. insight: you can look at humans as a network of trillions of computers.”
- “computers and networks break in ways that are similar to biological systems.”
- “DNA is genetic instructions, biological software.”
- “We generated so much data, information about genetics & cancer through research. we generate lots of research, we pay a lot of researchers. but fundamentally, we need to realize that research does not make therapies. Who makes therapies? Developers.”
- “we’re spending more and more on research, while getting fewer and fewer and therapies actually produced.”
- “research is exponential [more and more every year], development is linear [can’t keep up, obsolete before it’s even delivered]”
- “how about focus on the very end of the long tail? one therapy for one person: YOU”
- “build viruses based on your own DNA, that attack one particular type of cancer in your own body while leaving the rest of your cells alone.”
- “co-operative business model – the first drug company for people that don’t need to make a profit”
I recently had the pleasure of connecting with TS Ramakrishnan. Super smart guy and clearly one who know a thing or two about how to launch app platforms/APIs for social networks.
TS has an interesting short-list of four requirements for a successful social network app platform:
- Users — the social network must be large enough to bring lots of distribution to app publishers
- Language — platform must be backwards compatible with existing web development languages/frameworks
- Marketing — the social network must expend significant effort to recruit, retain, and support app developers
- Transactions — the social network must facilitate secure transactions (i.e., payments)
I found his last requirement to be really thought-provoking… It sounds like a pre-requisite for ecommerce transactions, but we haven’t seen a whole lot of apps pursue ecommerce models yet on social networks. The vast majority of social apps are 100% ad-supported, but we’re starting to see some “freemium” models develop. For instance, Slide’s SuperPoke app has Premium poke actions that users can get access to via a monthly subscription.
Even in an advertising-dominated economy, I wonder if the social apps on Facebook might monetize more efficiently if Facebook were to facilitate the payments/collections process between ad networks and app publishers? E.g., would social ad networks operate more efficiently if ad units were rendered in FBML and Facebook provided standardized tools for measuring impressions, clicks, follow-on actions, pathing; and financial transaction support for buying/selling social ads…