HEROKUCAMINHO ATÉ A ALTA ESCALA E

DISPONIBILIDADEfabio@heroku.com

QCONSP 2014

FABIO KUNGTech Lead, Runtime Systems at Heroku

heroku scale web=3 worker=2

alta-escala-disponibilidade.herokuapp.com

milhões de aplicações (web)

um dos maiores deployments de Linux containers (LXC) domundo

>60k requisições por segundo

>5G requisições por dia

12FACTOR.NET

portáveisplataformas modernas (cloud)elasticidade

duas regiões em produção: us-east and eu-west

diversas Availability Zones

2008

2009

2010

CRESCIMENTO

facebook/heroku

2011

2013

VIBE

starving-samurai-42.herokuapp.com

https://www.flickr.com/photos/timriley/9361949580

cultura hackerflickr/dominicotine

TIMES E COMPONENTES

TOTAL OWNERSHIPDependências?AutonomiaPoliglotaFull stack

CORE -> MICROSERVICES

no free lunch

INTERFACES IMPLÍCITASdocumentação pobre, informal

evolução, updates, releases coordenadosmanifest-driven APIs

SISTEMAS DISTRIBUÍDOS

retrycircuit breakerrate limitingrollback (transações distribuidas)state replicationcache...

HEROKU SCALE WEB=3 WORKER=5

HEROKU SCALE WEB=3 WORKER=5

TROUBLESHOOTINGassincronicidadedistributed tracingvisibilidade!

TESTES

DEPLOYS

DUPLICAÇÃO!

EPHEMERALIZATIONDo more with less.

DOGFOODING

TOOLS TEAM

DEVCLOUDS

boot your own Heroku

@merman boot my cloud

KERNEL PLATFORM

DIREWOLF

POSTGRESQL

contra exemplo: RabbitMQ

ORG ACCOUNTS

MÚLTIPLAS TECNOLOGIAS

diretrizesservice toolkitsproduto poliglota

#OPSLIFE

plantões semanais

ESCALATION PATHtime todo na rotaçãogerente do timeIncident Commander

TRANSPARÊNCIAstatus.heroku.com

csquared's Heroku Outage Lights System

TIME DE OPSTotal ownership?

SRESITE RELIABILITY ENGINEERS

confiabilidade globalcapacity planningreviewsretrospectivas de incidentestools, dashboardsfardo do plantão

MUDANÇASatualizar instâncias existentes vs. substituir por novas

instâncias

AVERSÃO A RISCOmudanças simples de uma linha -> catástrofe

RIGOR

"Hackers write Too Much Software.

Need to change Process.

Heroes mask Too Many Problems.

Need to change Teamwork."

-- , Engineering ManagerNoah

REVISÃO DE CÓDIGO

async, membros remotos

DOCUMENTAÇÃO

DIAGRAMAS

DESIGN

BLOG DRIVEN DEVELOPMENT

CFPgrandes decisões difîceis

CHECKLISTS

Example: production checklist

✓ Has ops docs with executable instructions

✓ Has a high-fidelity staging setup with production parity

✓ Requested audit from the security team

✓ Alerts a human if it is down

✓ Simulated failures

✓ Uses structured logging✓ Enforces SSL access

✓ Creds and rotation procedures are documented

✓ Send a launch email to engineering@

✓ Move to Production on the Engineering Lifecycle board

SUPORTEembutido

BUS FACTORTotal ownership?

BENEVOLENT DICTATORSHIPBDFL

COOPETIÇÃOCOMPETIÇÃO COOPERATIVA

LXCex.: DotCloud, container-rfc

We lost the standards game for virtualmachine images, but it feels like this

community is tight nit enough we might beable to do something for Linux Containers.

-- Alex Polvi (coreos.com)

GIT$ git push heroku masterCounting objects: 1, done.Writing objects: 100% (1/1), 181 bytes | 0 bytes/s, done.Total 1 (delta 0), reused 0 (delta 0)

-----> Ruby app detected-----> Compiling Ruby...To git@heroku.com:myapp.git 91dfe0b..f251ba7 master -> master

ex.: GitHub

2012

CLOUD

ex.: AWS, AppEngine

PESSOASpolítica de "não jerks"

CORE -> TIMES INDEPENDENTES

TOTAL OWNERSHIP

FOCO?

SREproduto-heróis+coordenação

HEROKU NA EUROPAFuracão Sandy (2012) -> us-east -> us-west

GERÊNCIA

's mdz Scaling Human Systems

SLACKalways too busy

O QUE MUDOU?valores (Adam Wiggins)

EPHEMERALIZATIONDo more with less.

MAKE IT REALIdeas are cheap.

SHIP IT

Nothing is real until it's being used by a real user.

DO IT WITH STYLEAesthetic matters.

INTUITION-DRIVEN | DATA-DRIVEN

Users don't really know what they want.

... PROVE COM DADOS

bikeshed@heroku.com

DIVIDE AND CONQUER

If it's hard, cut scope.

TIMING MATTERSMaybe now isn't the right time.

THROW THINGS AWAYNever be afraid to throw something away and do it again.

https://www.flickr.com/photos/teich/9427507382/

SMALL SHARP TOOLSComposability. .The Art of Unix Programming

Also teams. Several small, autonomous, focused teams.

PUT IT IN THE CLOUDServices, not software.

RESULTS, NOT POLITICS

"get ahead" in your career by delivering real value.Not by impressing your boss or with big talk.

DECISION-MAKING VIA OWNERSHIPNOT CONSENSUS OR AUTHORITY

Ownership can't be given, only taken.

Ownership can't be declared, only demonstrated.

DO-OCRACY / INTRAPRENEURSHIPAsk forgiveness, not permission.

EVERYTHING IS AN EXPERIMENT

Everything is always subject to change.Ending an experiment isn't a failure.

OWN UP TO FAILURE

Admit your mistake, say you're sorry, and feel the failure tomake sure you learned from it. Then, get back to work.

GRADUAL ROLLOUTS

Incremental. Adjust.

DESIGN EVERYTHING

Be intentional.

QUESTION EVERYTHINGThe status quo is never good enough.

MANIACAL FOCUS ON SIMPLICITY

There is no step 1.

... CÓDIGO ESPERTO DEMAIS

load averagesELB -> unicorns

CLI 4 LIFECommand-line interfaces are the heart of developer

workflows.

IGNORE THE COMPETITIONExcept to borrow good ideas.

WRITE WELLClear writing is clear thinking.

STRONG OPINIONS, WEAKLY HELDBe willing to change your mind.

OBRIGADO!fabio@heroku.com