Uma Solução para Identificação da Causa Raiz de Problemas no Gerenciamento de Mudanças de TI

XXIX Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos XXIX Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos

Uma Solução para Identificação da Causa Raiz de Problemas no Gerenciamento de Mudanças de TI Ricardo L. dos Santos, Juliano A. Wickboldt, Roben C. Lunardi, Bruno L. Dalmazo, Lisandro Z. Granville e Luciano P. Gaspary

Universidade Federal do Rio Grande do Sul, Brasil

XXIX Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos

• Introdução

• Solução Proposta

• Processo de Diagnóstico

• Modelo de Informação

• Arquitetura Conceitual

• Root Cause Analyzer

• Outputs

• Conclusão

• Trabalhos Futuros

Roteiro

2


Introdução

• Contexto

• A complexidade das infraestruturas de TI torna a tarefa de gerenciamento uma missão crítica

• ITIL (Information Technology Infrastructure Library) tornou-se importante para o Gerenciamento de Processos de TI

• Gerenciamento de Mudanças de TI

• Dita a forma como uma infraestrutura de TI deve evoluir de maneira consistente e segura

• Determina como uma mudança deve ser conduzida

3


Introdução

• Gerenciamento de Problemas

• Define como o ciclo de vida de problemas de TI deve ser gerenciado

• Os principais objetivos são

• Prevenir a ocorrência de problemas relacionados

• Eliminar problemas recorrentes

• Minimizar o impacto quando estes não podem ser impedidos

• Permite simplificar procedimentos que objetivam a detecção da causa raiz de problemas de TI, reduzindo os custos associados

4


Introdução

• Cenário Atual

• Muitas organizações utilizam ferramentas que auxiliam na identificação da causa de falhas

• Quando não eliminam a assistência, por parte do pessoal técnico, lhes fornecem informações relevantes

• Possibilita a reutilização do conhecimento, o que resulta em uma redução de custos

• Porém, tais ferramentas possuem sérias limitações, entre elas

• Não possuem um padrão para descrever os casos, que são especificados manualmente

• Dificuldade na reutilização de casos

• Baixa manutenabilidade

5


Introdução

• Problemas

• Diagnósticos frequentemente deixam de ser concluídos corretamente devido a desatualização dos casos

• A complexidade da infraestrutura de TI onde ocorre a falha é muitas vezes ignorada

• Dados de identificações anteriores não são considerados em novos diagnósticos

6


Introdução

• Objetivos

• Propor um método interativo de identificação de causas de falhas em processos de mudança

• Baseado no histórico de identificação de falhas anteriores em atividades de mudança

• Considerando a visão atualizada da infraestrutura de TI, onde a falha ocorreu

• Otimizar o processo de identificação de causas de problemas em mudanças nas organizações reutilizando o conhecimento adquirido

7


Iterative Diagnosis

Solução Proposta Processo de Diagnóstico

Incident Report Answered

Question

Root Cause Question

Selection

8

IR RC

Help Desk Root Cause

Analyzer

Operator


Config. Mgmt.

Database

Diagnosis System

Change Management System

Change

Planner

Change

Designer

Solução Proposta Arquitetura Conceitual

Operator

9

Deployment

System CI CI CI

Diagnosis Log

Recorder

RC

Root Cause

Analyzer

Root Cause Analyzer

Input

Processor

Weight

Calculator

Question

Selector

Question

Verifier IR RC

CI CI CI CI CI L

RC RC RC


Solução Proposta Modelo de Informação

10

▲ determinesProblem

possiblesAnswers

determinesOthersQuestions

CategoryParentChild

1 1..*

0..1 1

0..1

1..*

*

ServiceProblem

SolutionCategory

* 1..*

ManagedSystem

Element

ManagedElement

Logical Element

EnabledLogical

Element

MessageLog

ExchangeElement

SolutionElement

*

QuestionCategory

RecordLog

RecordedAnswers

RecordedQuestions

Category

1

0..1

Question

RootCause

1..* *

1 1 1 1

1

0..*

PRS_ServiceInciden

t

PRS_Problem

Answer

0..1

1..*

RecordedProblem


Log Log Log CI CI CI

Solução Proposta Root Cause Analyzer

11

Reports the

incident

CI

Priority

…

Operator




12

IDENTIFIES:

•Logs of previous diagnoses

•RCs, Questions and Answers (categories)

•Affected CIs

Operator


Root Cause Analyzer

Weight Calculator Question Verifier


13

Input Processor Question Selector

CI CI L

RC RC RC Identif ication

based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer



• Um exemplo de infraestrutura de TI



15


CI Category

CI Category

Internet Third Party Services

WEB Server (215) System Computer System Server Web Server

DNS Server (423) System Computer System Server DNS Server

Apache 2.2 Software Web Server

PHP 5.0.2 Software Programming Language

Hard Disk Device Storage Devices

Network Card Device Network Cards


Root Cause Analyzer



16


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer



17


Select questions and

answers based on the

categories of CIs

identified

RC RC RC RC RC

Configuration

Mgmt. System

Set of RCs, Answers, and Questions



18

Sy

ste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n

Co

mp

ute

r Sy

ste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

20

RC

RC

RC RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC RC

RC

System Computer System Server Web Server

Application Web Server

Device Network Cards

Third Party Services



19

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard


Root Cause Analyzer



20


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer


Root Cause Analyzer



21


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer



22

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20


Root Cause Analyzer



23


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer



24

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



25

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20


Root Cause Analyzer



26


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight Obvious?

Threshold

80% with the

same answer



27

Syste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Questions RC 1,3 (50)


Root Cause Analyzer



28


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the


Threshold

80% with the

same answer



29

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Questions RC 1,3 (50)


Root Cause Analyzer



30


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the


Threshold

80% with the

same answer


Root Cause Analyzer



31


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer


Solução Proposta Outputs

32

Question Path of Apache

cfg is Ok?

Question Network adapter

is configured?

Question Switch ports are

configured?

Question Exim is installed?

Question POP3 is running?

Question Apache is

installed?

Executions: 25 Answers: 88% YES

Operator’s Answer: YES

Executions: 22 Answers: 63,6% NO


Executions: 16 Answers: 93,7% YES







Operator’s Answer: NO

Question Switch is

reachable by

ping? Executions: 36 Answers: 77,8% NO


Question Server is

reachable by



Question Switch is up?

Executions: 48 Answers: 75% NO



cfg is Ok?

Question Exim is running?

Question Path of Exim cfg is

Ok?



Question Apache is

installed?














Solução Proposta Outputs

33


cfg is Ok?

Question Network adapter

is configured?

Question Switch ports are

configured?



Question Apache is

installed?













Question Switch is

reachable by



Question Server is

reachable by



Question Switch is up?

Executions: 48 Answers: 75% NO



cfg is Ok?

Question Exim is running?

Question Path of Exim cfg is

Ok?



Question Apache is

installed?













The ports of the Switch were not

configured properly

Path to e-mail files is wrong


Conclusão

34

• A solução proposta mostrou-se exequível com as seguintes características

• Interatividade

• Considerando apenas a infraestrutura de TI afetada

• Baseada em informações de diagnósticos anteriores

• Reutilizando o conhecimento adquirido

• Melhorou o processo de identificação de causa raiz de falhas em mudanças de TI


Trabalhos Futuros

35

• Melhorar o bootstrapping do sistema

• Melhorar o algoritmo de seleção

• Considerando outras métricas (idade)

• Considerando outros métodos de seleção (popularidade)

• Como avaliar apropriadamente a solução?

Obrigado pela sua atenção!

Perguntas?


References

37

• J. P. Sauvé, R. A. Santos, R. R. Almeida et al., “On the Risk Exposure and Priority Determination of Changes in IT Service Management,” in XVIII IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM 2007), 2007, pp. 147–158

• ITIL, “ITIL - Information Technology Infrastructure Library. Office of Government Commerce (OGC),” 2009, Available: http://www.itilofficialsite.com/. Accessed: aug. 2010

• G. Machado, F. Daitx, W. Cordeiro et al., “Enabling rollback support in IT change management systems,” in Network Operations and Management Symposium, 2008. NOMS 2008. IEEE, April 2008, pp. 347–354

• W. Cordeiro, G. Machado, F. Andreis et al., “ChangeLedge: Change design and planning in networked systems based on reuse of knowledge and automation,” Computer Networks, vol. 53, no. 16, pp. 2782 – 2799, 2009

• ITIL, “ITIL - Information Technology Infrastructure Library: Service Operation Version 3.0. Office of Government Commerce (OGC),” 2007

• DMTF, “Distributed Management Task Force: Common Information Model. Distributed Management Task Force (DMTF),” 2009, Available: http://www.dmtf.org/standards/cim. Accessed: aug. 2010


References

38

• J. Sauvé, R. Santos, R. Reboucas, A. Moura, and C. Bartolini, “Change priority determination in it service management based on risk exposure,” Network and Service Management, IEEE Transactions on, vol. 5, no. 3, pp. 178 –187, september 2008

• A. Brown and A. Keller, “A best practice approach for automating it management processes,” in Network Operations and Management Symposium, 2006. NOMS 2006. 10th IEEE/IFIP, 3-7 2006, pp. 33 –44

• A. Moura, J. Sauve, and C. Bartolini, “Business-driven it management - upping the ante of it : exploring the linkage between it and business to improve both it and business results,” Communications Magazine, IEEE, vol. 46, no. 10, pp. 148 –153, october 2008

• A. Keller, J. Hellerstein, J. Wolf, K.-L. Wu, and V. Krishnan, “The champs system: change management with planning and scheduling,” in Network Operations and Management Symposium, 2004. NOMS 2004. IEEE/IFIP, vol. 1, 23-23 2004, pp. 395 –408 Vol.1

• M. Jantti and A. Eerola, “A Conceptual Model of IT Service Problem Management,” in Service Systems and Service Management, 2006 International Conference on, vol. 1, Oct. 2006, pp. 798–803

• R. Gupta, K. Prasad, and M. Mohania, “Automating itsm incident management process,” in Autonomic Computing, 2008. ICAC ’08. International Conference on, 2-6 2008, pp. 141 –150


References

39

• K. Appleby, G. Goldszmidt, and M. Steinder, “Yemanja-a layered event correlation engine for multi-domain server farms,” in Integrated Network Management Proceedings, 2001 IEEE/IFIP International Symposium on, 2001

• M. Steinder and A. S. Sethi, “Probabilistic fault diagnosis in communication systems through incremental hypothesis updating,” Computer Networks, vol. 45, no. 4, pp. 537 – 562, 2004

• W. L. C. Cordeiro, G. Machado, D. F.F. et al., “A template-based solution to support knowledge reuse in IT change design,” in Network Operations and Management Symposium, 2008. NOMS 2008. IEEE, April 2008, pp. 355–362

• J. A. Wickboldt, L. A. Bianchin, R. C. Lunardi et al., “Improving it change management processes with automated risk assessment,” in XII IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM 2009), 2009

• R. C. Lunardi, F. G. Andreis, W. L. d. C. Cordeiro, J. A. Wickboldt, B. L. Dalmazo, R. L. d. Santos, L. A. Bianchin, L. P. Gaspary, L. Z. Granville, and C. Bartolini, “On strategies for planning the assignment of human resources to it change activities,” in Network Operations and Management Symposium, 2010. NOMS 2010. IEEE, apr. 2010, pp. 248–255


Root Cause Analyzer



40


CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer


Proposed Solution Root Cause Analyzer

41

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



42

Syste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



43

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



44

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Qestions RC 1, 3 (50)



45

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions



46

How many

bits is the

application?

16 bits 32 bits

64 bits


We

b S

erv

er

70


47

Syste

m

Th

ird P

arty

Se

rvic

es

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

95 35

30

26

20

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions

So

ftwa

re

We

b S

erv

er


We

b S

erv

er

70


48

Syste

m

Th

ird P

arty

Se

rvic

es

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

95 35

30

26

20

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions

So

ftwa

re

We

b S

erv

er


We

b S

erv

er

70


49

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

95 35

30

26

20

95 – 70 = 25

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions

We

b S

erv

er


We

b S

erv

er

70


50

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

We

b S

erv

er


We

b S

erv

er

70


51

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

We

b S

erv

er


We

b S

erv

er

70


52

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

We

b S

erv

er


We

b S

erv

er

We

b S

erv

er

70


53

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)

Questions RC 11 (25)


We

b S

erv

er

We

b S

erv

er

70


54

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)




55

How many

bits is the

application?

16 bits 32 bits

64 bits

The devices

are well

plugged?

No Yes


We

b S

erv

er

We

b S

erv

er

70


56

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)



We

b S

erv

er

We

b S

erv

er

70


57

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)




58

How many

bits is the

application?

16 bits 32 bits

64 bits

The devices

are well

plugged?

Network

card

detected?

No Yes

No Yes


We

b S

erv

er

We

b S

erv

er

70


59

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)



We

b S

erv

er

We

b S

erv

er

70


60

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)




61

How many

bits is the

application?

16 bits 32 bits

64 bits

The devices

are well

plugged?

Network

card

detected?

Network

card is

configured

correctly?

Network card

not

configured

No Yes

No Yes

Yes

No

Uma Solução para Identificação da Causa Raiz de Problemas no Gerenciamento de Mudanças de TI

Documents

Transcript of Uma Solução para Identificação da Causa Raiz de Problemas no Gerenciamento de Mudanças de TI