HEROKU - QConSP · 2014. 8. 27. · [email protected]. QCONSP 2014 FABIO KUNG Tech Lead, Runtime...
115
Transcript of HEROKU - QConSP · 2014. 8. 27. · [email protected]. QCONSP 2014 FABIO KUNG Tech Lead, Runtime...
QCONSP 2014
FABIO KUNGTech Lead, Runtime Systems at Heroku
heroku scale web=3 worker=2
alta-escala-disponibilidade.herokuapp.com
milhões de aplicações (web)
um dos maiores deployments de Linux containers (LXC) domundo
>60k requisições por segundo
>5G requisições por dia
12FACTOR.NET
portáveisplataformas modernas (cloud)elasticidade
duas regiões em produção: us-east and eu-west
diversas Availability Zones
2008
2009
2010
CRESCIMENTO
facebook/heroku
2011
2013
VIBE
starving-samurai-42.herokuapp.com
https://www.flickr.com/photos/timriley/9361949580
cultura hackerflickr/dominicotine
TIMES E COMPONENTES
TOTAL OWNERSHIPDependências?AutonomiaPoliglotaFull stack
CORE -> MICROSERVICES
no free lunch
INTERFACES IMPLÍCITASdocumentação pobre, informal
evolução, updates, releases coordenadosmanifest-driven APIs
SISTEMAS DISTRIBUÍDOS
retrycircuit breakerrate limitingrollback (transações distribuidas)state replicationcache...
HEROKU SCALE WEB=3 WORKER=5
HEROKU SCALE WEB=3 WORKER=5
TROUBLESHOOTINGassincronicidadedistributed tracingvisibilidade!
TESTES
DEPLOYS
DUPLICAÇÃO!
EPHEMERALIZATIONDo more with less.
DOGFOODING
TOOLS TEAM
DEVCLOUDS
boot your own Heroku
@merman boot my cloud
KERNEL PLATFORM
DIREWOLF
POSTGRESQL
contra exemplo: RabbitMQ
ORG ACCOUNTS
MÚLTIPLAS TECNOLOGIAS
diretrizesservice toolkitsproduto poliglota
#OPSLIFE
plantões semanais
ESCALATION PATHtime todo na rotaçãogerente do timeIncident Commander
TRANSPARÊNCIAstatus.heroku.com
csquared's Heroku Outage Lights System
TIME DE OPSTotal ownership?
SRESITE RELIABILITY ENGINEERS
confiabilidade globalcapacity planningreviewsretrospectivas de incidentestools, dashboardsfardo do plantão
MUDANÇASatualizar instâncias existentes vs. substituir por novas
instâncias
AVERSÃO A RISCOmudanças simples de uma linha -> catástrofe
RIGOR
"Hackers write Too Much Software.
Need to change Process.
Heroes mask Too Many Problems.
Need to change Teamwork."
-- , Engineering ManagerNoah
REVISÃO DE CÓDIGO
async, membros remotos
DOCUMENTAÇÃO
DIAGRAMAS
DESIGN
BLOG DRIVEN DEVELOPMENT
CFPgrandes decisões difîceis
CHECKLISTS
Example: production checklist
✓ Has ops docs with executable instructions
✓ Has a high-fidelity staging setup with production parity
✓ Requested audit from the security team
✓ Alerts a human if it is down
✓ Simulated failures
✓ Uses structured logging✓ Enforces SSL access
✓ Creds and rotation procedures are documented
✓ Send a launch email to engineering@
✓ Move to Production on the Engineering Lifecycle board
SUPORTEembutido
BUS FACTORTotal ownership?
BENEVOLENT DICTATORSHIPBDFL
COOPETIÇÃOCOMPETIÇÃO COOPERATIVA
LXCex.: DotCloud, container-rfc
We lost the standards game for virtualmachine images, but it feels like this
community is tight nit enough we might beable to do something for Linux Containers.
-- Alex Polvi (coreos.com)
GIT$ git push heroku masterCounting objects: 1, done.Writing objects: 100% (1/1), 181 bytes | 0 bytes/s, done.Total 1 (delta 0), reused 0 (delta 0)
-----> Ruby app detected-----> Compiling Ruby...To [email protected]:myapp.git 91dfe0b..f251ba7 master -> master
ex.: GitHub
2012
CLOUD
ex.: AWS, AppEngine
PESSOASpolítica de "não jerks"
CORE -> TIMES INDEPENDENTES
TOTAL OWNERSHIP
FOCO?
SREproduto-heróis+coordenação
HEROKU NA EUROPAFuracão Sandy (2012) -> us-east -> us-west
GERÊNCIA
's mdz Scaling Human Systems
SLACKalways too busy
O QUE MUDOU?valores (Adam Wiggins)
EPHEMERALIZATIONDo more with less.
MAKE IT REALIdeas are cheap.
SHIP IT
Nothing is real until it's being used by a real user.
DO IT WITH STYLEAesthetic matters.
INTUITION-DRIVEN | DATA-DRIVEN
Users don't really know what they want.
DIVIDE AND CONQUER
If it's hard, cut scope.
TIMING MATTERSMaybe now isn't the right time.
THROW THINGS AWAYNever be afraid to throw something away and do it again.
https://www.flickr.com/photos/teich/9427507382/
SMALL SHARP TOOLSComposability. .The Art of Unix Programming
Also teams. Several small, autonomous, focused teams.
PUT IT IN THE CLOUDServices, not software.
RESULTS, NOT POLITICS
"get ahead" in your career by delivering real value.Not by impressing your boss or with big talk.
DECISION-MAKING VIA OWNERSHIPNOT CONSENSUS OR AUTHORITY
Ownership can't be given, only taken.
Ownership can't be declared, only demonstrated.
DO-OCRACY / INTRAPRENEURSHIPAsk forgiveness, not permission.
EVERYTHING IS AN EXPERIMENT
Everything is always subject to change.Ending an experiment isn't a failure.
OWN UP TO FAILURE
Admit your mistake, say you're sorry, and feel the failure tomake sure you learned from it. Then, get back to work.
GRADUAL ROLLOUTS
Incremental. Adjust.
DESIGN EVERYTHING
Be intentional.
QUESTION EVERYTHINGThe status quo is never good enough.
MANIACAL FOCUS ON SIMPLICITY
There is no step 1.
... CÓDIGO ESPERTO DEMAIS
load averagesELB -> unicorns
CLI 4 LIFECommand-line interfaces are the heart of developer
workflows.
IGNORE THE COMPETITIONExcept to borrow good ideas.
WRITE WELLClear writing is clear thinking.
STRONG OPINIONS, WEAKLY HELDBe willing to change your mind.
