CaseTailTarget:infraestruturaemodelodedesenvolvimento

FernandoBabadopulos@babadopulos

NuvemAplicaçãoEscalável

Automa4zarTarefasDeployCon=nuo

IntegraçãoCon=nuaTestesAutoma4zados

ArquiteturasEscaláveis

Arquiteteparapelomenos3

Assumaquealgumacoisairáfalhar

Isoleosseusserviços

Architec4ngforScalability

SHAREDNOTHING

SharedNothing

AppServer1 AppServer2 AppServer3

Cache1 Cache2 Cache3

DB1 DB2 DB3

SharedNothing

AppServer1 AppServer2 AppServer3

Cache1 Cache2 Cache3

DB1 DB2 DB3

Cache4

-  Semserver-sidesession

-  SemsMckysessions

SharedNothing

PlayFramework PlayFramework PlayFramework

RedisShard1/Replica1

RedisShard2/Replica1

RedisShard1/Replica2

MongoDBMaster

MongoDBSlave1

MongoDBSlave2

RedisShard2/Replica2

Escalando

UseopoderdaNuvem

masuseosseusDADOS!

PrediçãodeTráfego–Porque?

Podedemorarmuitosminutosparaterumamáquinanoar,atéhoras.Dáparaesperartudoisso?Evitefalsasquedasdetráfego

PrediçãodeTráfego

700

800

900

1000

1100

1200

1300

1400

1500

1600

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

Requests

Requests

PrediçãodeTráfego

700

800

900

1000

1100

1200

1300

1400

1500

1600

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

Requests

Requests

Linear(Requests)

PrediçãodeTráfego

700

800

900

1000

1100

1200

1300

1400

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

Requests

Requests

Linear(Requests)

PrediçãodeTráfego

700

800

900

1000

1100

1200

1300

1400

1500

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

Requests

PredictedTraffic

Linear(Requests)

Muitasvezesvocênãoprecisaprocessartodososseusdados

705.166

1.288.822.081

62ms

1,5h

6.931

12.678.931

x1.190.642.440651.448

= 12.667.692

Erro≈0.088%

Totalderegistrosconhecido

amostragem

Totalderegistrosamostrados

REDIS01 REDIS02 REDIS03 REDIS04 REDIS05

for (Jedis jedis : cluster.getAllShards()) { }

long dbSize = jedis.dbSize();

String key = jedis.randomKey();

String value = jedis.get(key);

.

.

.

.

.

F 0.728 OUT

Tamanhodaamostrafinal

A

0.461

B

0.352

C

0.831

D

0.314

E

0.816F 0.728

IN

ReservoirSamplingDistribuído

1 2 3 4 5

A:0.1B:0.3C:0.2D:0.7E:0.9F:0.11G:0.4H:0.6I:0.76

J:0.8K:0.2L:0.54M:0.4N:0.21O:0.33P:0.56Q:0.32R:0.23

S:0.21T:0.32U:0.22V:0.7X:0.12Y:0.23W:0.3Z:0.76

private SortedMap<Double, MyObject> reservoir; ... if (reservoir.size() < SAMPLE_SIZE) { reservoir.put(score, myObject); } else if (score > reservoir.firstKey()) { reservoir.remove(reservoir.firstKey()); reservoir.put(score, myObject); }

ReservoirSamplingDistribuído

O L P I Z

1 2 3 4 5

A:0.1B:0.3C:0.2D:0.7E:0.9F:0.11G:0.4H:0.6I:0.76

J:0.8K:0.2L:0.54M:0.4N:0.21O:0.33P:0.56Q:0.32R:0.23

S:0.21T:0.32U:0.22V:0.7X:0.12Y:0.23W:0.3Z:0.76

H:0.6D:0.7E:0.9F:0.11I:0.76 R:0.23Q:0.32O:0.33L:0.54P:0.56 S:0.21U:0.22Y:0.23T:0.32Z:0.76

COMBINER

ReservoirSamplingDistribuído

import org.apache.crunch.lib.Sample; Sample.reservoirSample(PCollection<T> input, int sampleSize)

ApacheCrunch:

CaseTailTarget:infraestruturaemodelodedesenvolvimento

FernandoBabadopulos@babadopulos