TDC2016POA | Trilha PHP - Desenvolvendo um buscador com PHP e Elasticsearch
Elasticsearch
-
Upload
ricardo-peres -
Category
Software
-
view
249 -
download
3
Transcript of Elasticsearch
![Page 1: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/1.jpg)
Ricardo Peres@rjperes75
http://netponto.org1ª Reunião Presencial em Aveiro - 7/5/2016
![Page 2: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/2.jpg)
Elasticsearch 2
Elasticsearch
Base de dados NoSQL para conteúdos JSONMuito rápida: documentos indexados em < 1sDistribuídaBaseada no LucenePuramente RESTSuporte a grafosOpen source
![Page 3: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/3.jpg)
Elasticsearch 3
Colecção de servidores (nós)Um único mestre em cada instanteDescoberta automática ou explícita
Cluster
![Page 4: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/4.jpg)
Elasticsearch 4
Os índices são distribuídos por shards – por omissão 5 shards e 1 réplica por cluster
Forma de separar fisicamente conteúdosDefinidos aquando da criação do índiceTransparenteÉ possível fornecer o algoritmo de sharding
Shards
![Page 5: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/5.jpg)
Elasticsearch 5
Índices
Colecção de tiposPossui um schemaSemelhante a uma base de dados no mundo relacional
![Page 6: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/6.jpg)
Elasticsearch 6
Tipo
Colecção de documentosPossui um schema, herdado do índiceSemelhante a uma tabela no mundo relacional
![Page 7: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/7.jpg)
Elasticsearch 7
Documentos
JSONExistem num tipoTêm um identificador únicoVersão (1, …)Têm um schemaPodem ter expiração
![Page 8: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/8.jpg)
Elasticsearch 8
Campos
Os documentos contêm camposSempre presentes:
_id: chave primária (string) _index: índice onde o documento existe (string) _type: tipo onde o documento existe (string) _uid: _id + _type
Opcionais: _timestamp: data e hora de criação (date) _all: todos os campos concatenados (string) _source: JSON original (string) _ttl: duração (date) _meta: metadados (object) _parent _routing
Têm um tipo de dados associado
![Page 9: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/9.jpg)
Elasticsearch 9
string long, integer, short, byte,
double, floatdatebooleanbinary geo_point geo_shapeobjectnested
ip completion token_count
arrays (de qualquer tipo)
Outros (extensões)
Tipos de Dados
![Page 10: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/10.jpg)
Elasticsearch 10
Id gerado automaticamentePOST /website/blog{
"title" : “My Blog", "url" :
"http://my/blog", "tags" :
[ "development" ]}
Id explícitoPOST /website/blog/1{
"title" : "My Blog", "url" :
"http://my/blog", "tags" : [ "
development " ]}
Criar um Documento
![Page 11: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/11.jpg)
Elasticsearch 11
ParcialPOST /website/blog/1/_update{
"doc" :{
"tags" : [ "testing" ],
"views": 1}
}
Total (substituir)POST /website/blog/1/_update{
"title" : "My Blog", "url" :
"http://my/blog", "tags" : [ "testing" ]
}
Actualizar um Documento
![Page 12: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/12.jpg)
Elasticsearch 12
Actualizar um Documento por Script Adicionar campos ou modificar campos existentes por meio de
scripts
"scripted_upsert": true, "script": { "inline": "if (ctx._source.likes == null) { ctx._source.likes = 0 }; ctx._source.likes += count", "params": { "count": 1 } }
![Page 13: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/13.jpg)
Elasticsearch 13
ÚnicoDELETE /website/blog/1
ÍndiceDELETE /website
Com base numa queryDELETE /website/blog/_query?q=tags:java(Plugin delete by query)
Apagar um Documento
![Page 14: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/14.jpg)
Elasticsearch 14
MapeamentoCriado ao nível do índice ou do tipo, implícita ou explicitamenteNão é possível modificar, apenas adicionarObrigatórios ou não
PUT website
{
"mappings": {
"blog": {
"dynamic" : "strict",
"properties": {
"title": {
"type": "string",
"analyzer": "standard"
}
}
}
}
}
![Page 15: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/15.jpg)
Elasticsearch 15
Modelos de Mapeamentos
Aplicar mapeamentos automaticamente a novos tipos
PUT website{ "mappings": { "post": { "dynamic_templates": [ { "timestamp": {
"match": "timestamp", "match_mapping_type": "date", "mapping": { "type": "date", "format" : "yyyy-MM-dd HH:mm" }
} } ] } }}
![Page 16: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/16.jpg)
Elasticsearch 16
Pesquisas: resultados ordenados por relevância
Filtros: restringir o que apareceCache
Contexto de pesquisa ou filtragem
![Page 17: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/17.jpg)
Elasticsearch 17
Search API URL/<index>/<type>/_search?q=something/<index>/<type1>,<type2>/_search?q=something_search?q=something_search?q:field:value_search?q=+firstname(john mary)&-surname:smith
Query DSL Pesquisas e filtros simple_query_string,
query_string, match, term, terms, range, multi_match, match_phrase, missing, exists, regexp, fuzzy, prefix, ids
bool, dis_max more_like_this, script,
template
Pesquisas
![Page 18: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/18.jpg)
Elasticsearch 18
Full-text search query_string,
simple_query_string, match, multi_match, match_phrase, term, terms
Inexacta regexp, fuzzy, prefix, wild-
card
Tipos de Pesquisas
![Page 19: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/19.jpg)
Elasticsearch 19
Ordenação de documentos encontrados por relevância
Campos:Informação do clusterTempo que demorouEncontrou ou não_score: relevância_source: documento
Resultados de Pesquisas
![Page 20: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/20.jpg)
Elasticsearch 20
Paginação, Ordem e Projecções
Paginação: size, fromOrdem: sortProjecções: fields
POST website/post/_search{
“size”: 10,“from”: 0,“sort”: {“timestamp”: {“order”: “desc”}},“fields”: [ “title”, “_id” ]
}
![Page 21: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/21.jpg)
Elasticsearch 21
É possível fazer ordenação e incluir resultados calculados (scripts)
"script": { "inline": "_value.toUpperCase()" }
Scripts
![Page 22: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/22.jpg)
Elasticsearch 22
Percolator
Primeiro definir a pesquisa Verificar, por documento, que pesquisas é que cumpre
PUT products/.percolator/high_ratings{ "query" : { "range": { "ratings": { "gte": 4 } } }}
GET products/product/iPhone6s/_percolate
![Page 23: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/23.jpg)
Elasticsearch 23
Relações
Não são possíveis JOINs, mas há algumas alternativas
Relações pai/filho: has_child, has_parent
Objectos embebidos
Lookup de termos: terms com type e id
![Page 24: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/24.jpg)
Elasticsearch 24
Relevância
Term Frequency (TF), Inverse Document Frequency (IDF), Field Length Norm (FLN)Possível fornecer funções de scoringPossível pedir explicação sobre o processo
![Page 25: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/25.jpg)
Elasticsearch 25
Boost de campos:"query": { "multi_match": { "query": "elasticsearch", "fields": [ “title^5", "body" ] } }
Should:"bool": { "should": [ { "match": { "body": { "query": "elasticsearch", "boost": 1 } } }, { "match": { "body": { "query": "technology", "boost": 0.5 } } } ] } , "minimum_number_should_match": "50%"
Pesquisa por Relevância
![Page 26: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/26.jpg)
Elasticsearch 26
Máximo de disjuntas:"dis_max": { "queries": [ { "term": { "tags": { "value": "technologies" } } }, { "term": { "tags": { "value": "others" } } } ] }
Boosting:"boosting": { "positive": { "term": { "tags": "technologies" } }, "negative": { "term": { "tags": "books" } }, "negative_boost": 0.5 }
Pesquisa por Relevância
![Page 27: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/27.jpg)
Elasticsearch 27
Funções de scoring:
"functions": [ { "field_value_factor": { "field": "ratings", "factor": 1.2 }, "gauss": { "price": { "origin": "0", "scale": "100" } } } ] }
Scripting:
"script_score": { "script": "_score * doc['person'].value.length() * 5" }
Scoring
![Page 28: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/28.jpg)
Elasticsearch 28
Usando os tipos geo_point e geo_shape
"filter": { "geo_distance": { "distance": "200km", "location": "40.2, -8.4166667" } }
"script_fields": { "distance": { "params": { "lat": 40.2, "lon": -8.4166667 }, "script": "doc['location'].distanceInKm(lat, lon)" } }
Pesquisas Geo Referenciadas
![Page 29: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/29.jpg)
Elasticsearch 29
Indexação
Transformação em tokensStemmingNormalização
Um campo pode ser ou não analisadoAnalisadores para cada línguaPossível definir os nossosPode dar resultados inesperados
![Page 30: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/30.jpg)
Elasticsearch 30
Aliases de Índices
Abstrair um ou mais índices, opcionalmente com um filtroÚtil para índices “móveis" (mês, ano, país, zona, etc)Apenas útil para queries
POST /_aliases{
"actions" : [ {"add" : {
"indices" : [ "social-2015", "social-2016" ],"alias" : "social-testing","filter" : {
"term" : {"tag" : "testing"
} }
} } ]
}
![Page 31: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/31.jpg)
Elasticsearch 31
Modelos de Alias Adiciona a um alias um tipo quando este for criado
POST /_template/social{ "order": 0, "template": "social-*", "settings": { "index": { "refresh_interval": "5s" } }, "mappings": {}, "aliases": { "social": {} }}
![Page 32: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/32.jpg)
Elasticsearch 32
Operações Bulk
Executar várias operações (index, update, delete) de uma só vez
POST bulk/data/_bulk{ "index" : { "_id" : "1" } }{ "field1" : "value1" }{ "index" : { "_id" : "2" } }{ "field1" : "value1" }{ "index" : { "_id" : "3" } }{ "field1" : "value1" }{ "update" : { "_id" : "2" } }{ "doc": { "field2": "value2" } }{ "delete" : { "_id" : "3" } }
![Page 33: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/33.jpg)
Elasticsearch 33
Análise Uma ou mais agregações Podem ser combinadas Podem usar scripts
GET /megacorp/employee/_search{ "aggs": { "all_interests": { "terms": { "field": “feature“ }, “aggs”: { “average_price”: { “field”: “price” } } } }}
![Page 34: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/34.jpg)
Elasticsearch 34
APIs
REST (nativo) .NET JavaScript/Node.js Python Java Groovy PHP Perl Ruby
![Page 35: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/35.jpg)
Elasticsearch 35
Marvel Sense Watcher Graph Shield Hadoop
Head Kopf Elasticsearch-SQL Delete by query Bigdesk
Plugins
![Page 36: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/36.jpg)
Elasticsearch 36
ReportingDashboards
Kibana
![Page 37: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/37.jpg)
Elasticsearch 37
Recolher e transformar dados Input – Filters – Outputs Fontes e destinos:
Elasticsearch File Syslog Windows Eventlog Redis RabbitMQ GitHub HTTP Beats Twitter WebSocket …
Logstash
![Page 38: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/38.jpg)
Elasticsearch 38
https://www.elastic.cohttps://
www.gitbook.com/book/allen8807/elasticsearch-definitive-guide-en/details
https://github.com/elastic/cookbook-elasticsearchhttps://github.com/elastic/elasticsearch-nethttps://github.com/elastic/kibanahttps://github.com/elastic/logstashhttps://github.com/elastic/elasticsearchhttp://joelabrahamsson.com/elasticsearch-101
Referências
![Page 39: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/39.jpg)
Patrocinadores “Gold”
https://fusioncowork.com/https://www.facebook.com/FUSIONCoWork/https://twitter.com/fusioncowork
![Page 41: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/41.jpg)
Patrocinadores “Bronze”
![Page 42: Elasticsearch](https://reader033.fdocumentos.com/reader033/viewer/2022042907/5877cb821a28ab39588b6b21/html5/thumbnails/42.jpg)
Elasticsearch 42
Obrigado
Obrigado por participarem!
@[email protected]://weblogs.asp.net/ricardoperes