AVALIES -UFRGS

AVALIAÇÃO DA EDUCAÇÃO SUPERIOR

PAINEL 1:

RANQUEAMENTO DA EDUCAÇÃO SUPERIOR

Jorge A. Guimarães

jguimaraes14@gmail.com

Porto Alegre, 17de setembro de 2015

Se os rankings são uma doença o

benchmarking pode ser a cura?

Jamil Salmi

who prepares the rankings?

0

1

2

3

4

5

6

7

8

9

Asia & the Pacific

Eastern Europe & Central Asia

Latin America & Caribbean

Middle Eastern & North Africa

North America Sub-Saharan Africa

Western Europe

Government Agency

Independent Agency

Press / Media

ranking systems in 2014

Region National and International Ranking SystemNational and International Ranking System

Eastern Europe and Central Asia Albania (A), Bulgaria (A), Kazakhstan (A, B), Latvia ©, Poland (C),

Slovakia (B), Romania (B/C), Russia (B, IB), Ukraine (B/C)

East Asia and Pacific

Australia (B), China (B, C, IB), Hong Kong (C), Japan (B, C), Malaysia (A),

New Zealand (A), South Korea ( B, C), Taiwan (B, IB), Thailand (A),

Vietnam (A)

Latin America and the Caribbean Argentina (A), Brazil (A), Chile (C), Colombia (B), Mexico (B, C), Peru (B)

Middle East and North Africa Israel (C), Tunisia (A)

North America Canada (B, C, B/C), United States (C)

South Asia India (A, B/C), Pakistan (A)

Sub-Saharan Africa Kenya (A), Nigeria (A)

Western Europe

France (IB), Germany (B/C, C), Ireland (C), Italy (C), Netherlands (A, IB),

Portugal (C), Spain (B, C, IC), Sweden (C), Switzerland (B/C), United

Kingdom (A, B, IC)

top 50 universities (2013)

ARWU ARWU 20132013

THETHE

20122012--1313

JAPAN; 2 CANADA; 2

UK; 5

WESTERN EUROPE; 6

USA ; 35

JAPAN, 1 AUSTRALIA, 2

CANADA, 3

OTHER ASIA, 4

WESTERN EUROPE, 4

UK, 7

USA, 29

Anglo-Saxon bias

s

top 100 universities / 1 million

people

0,01

0,02

0,04

0,05

0,06

0,09

0,11

0,14

0,16

0,18

0,18

0,20

0,29

0,31

0,36

0,38

0,52

0,00 0,10 0,20 0,30 0,40 0,50 0,60

Russia

Japan

Australia

Germany

France

Belgium

Canada

UK

USA

Netherlands

Finland

Norway

California

Sweden

Denmark

Israel

Switzerland

so should we just get rid of rankings?

1212

Autonomy

Academic Freedom

Students

Teaching Staff

Researchers

Leading-Edge

Research

Dynamic

Knowledge &

Technology

Transfer

Concentration

of Talent

Abundant

Resources Favorable

Governance

Leadership Team

Strategic Vision

Culture of Excellence

Public Budget Resources

Endowment Revenues

Tuition Fees

Research Grants

WPU Supportive

Regulatory

Framework

Top

Graduates

Characteristics of a Well-Performing University

Alignment of Key Factors

Source: Elaborated by Jamil Salmi

results

drivers of performance

2000

2014

clear goals

Building Minnesota’s Building Minnesota’s

worldworld--leading statusleading status

in the knowledge in the knowledge economy requires economy requires

setting goals for HE and setting goals for HE and measuring results.measuring results.

Governor Tim PawlentyGovernor Tim Pawlenty

benchmarking

assessing research excellence

Research Excellence

Quantity

Impact Quality

Internationalization of Brazilian Universities:

Challenges and Perspectives for

Institutions to Reach the Level of

World-Class University

Jorge A. Guimarães

jguimaraes@capes.gov.br

Brasília, September 09, 2014

First Grade Challenges

for Brazilian Institutions to

Achieve WCU Stage

AUTONOMY

ACCOUNTABILITY

GOVERNANCE

Second Grade Challenges for Brazilian

Institutions to Achieve WCU Stage:

Top Ten Obstacles • Internationalization of Teaching and

Research;

• Reduce Informational Teach and Increase Formative Activities;

• Adopt International Curriculum;

• Offer Regular Courses in English and Other Languages;

• Apply Effective International Collaboration;

• Increase International Mobility of Students, Professors and Scientific Staff;

• Attract Foreign Students and Scientific Researchers;

• Offer Campus Residence;

• Stimulate International and Collaborative Publications;

• Offer Internships in Industry.

Challenges for Brazilian

Institutions to Achieve a World-

Class University Category:

How Many Brazilian Universities

Could Reach This Goal Today?

A Brief Study Based on Existing

Postgraduate Programs

CAPES RANKING - BRAZILIAN UNIVERSITIES

RANKING

NUMBER OF GRADUATE COURSES PER GRADE

University Grade 6 Grade 7 TOTAL 6 + 7 % CUMULATIVE %

1 USP 47 45 92 22.2 22.2

2 UFRJ 18 16 34 8.2 30.4

3 UNICAMP 17 16 33 8.2 38.6

4 UFRGS 19 13 32 7.7 46.3

5 UFMG 18 13 31 7.5 53.8

6 UNESP 14 3 17 4.1 57.9

7 UFSC 14 3 17 4.1 62.0

8 UNIFESP 6 6 12 2.9 64.9

9 UNB 10 2 12 2.9 67.8

10 UFV 8 3 11 2.7 70.5

11 PUC-RS 10 1 11 2.5 73.0

12 PUC-RJ 7 2 9 2.2 75.2

13 UFPE 8 ---- 8 1.9 77.1

14 UFPR 7 ---- 7 1.7 78.8

15 UERJ 4 3 7 1.7 80.5

16 UFSCAR 3 3 6 1.5 82.0

17 UFC 5 --- 5 1.2 83.0

18 UFF 3 1 4 1.0 84.0

19 UFLA 3 1 4 1.0 85.0

20 UFBA 3 1 4 1.0 86.0

24 IES 34 5 39 9.4 95.4

10 RESEARCH

INSTITUTES 11 8 19 4.6 100.0

TOTAL BRAZIL 270 145 415 11.8 100.0

CAPES RANKING - BRAZILIAN UNIVERSITIES

RANKING NUMBER OF GRADUATE COURSES PER GRADE

University Grade 6 Grade 7 TOTAL 6 + 7 PG % 6+7

1 USP 47 45 92 173 53.2

2 UFRJ 18 16 34 107 31.8

3 UNICAMP 17 16 33 71 46.5

4 UFRGS 19 13 32 84 38.0

5 UFMG 18 13 31 77 40.3

6 UNESP 14 3 17 105 16.2

7 UFSC 14 3 17 68 25.0

8 UNIFESP 6 6 12 53 22.6

9 UNB 10 2 12 89 13.5

10 UFV 8 3 11 41 26.8

11 PUC-RS 10 1 11 25 44.0

12 PUC-RJ 7 2 9 30 30.0

13 UFPE 8 ---- 8 75 10.7

14 UFPR 7 ---- 7 72 10.0

15 UERJ 4 3 7 53 13.2

16 UFSCAR 3 3 6 47 12.7

17 UFC 5 - 5 76 8.2

18 UFF 3 1 4 70 5,7

19 UFLA 3 1 4 29 13.8

20 UFBA 3 1 4 62 5.3

Outras 24 IES 34 5 39 869 4.5

10 RESEARCH

INSTITUTES 11 8 19 61 31.1

TOTAL BRAZIL 270 145 415 2276 18.2

BRAZILIAN UNIVERSITIES COMPARATIVE

RANKINGS

CAPES RANKING RUF RANKING (2012)

Ranking University Ranking University

1 USP 1 USP

2 UFRJ 2 UFRJ

3 UNICAMP 3 UFMG

4 UFRGS 4 UFRGS

5 UFMG 5 UNICAMP

6 UNESP 6 UNESP

7 UFSC 7 UFSC

8 UNIFESP 8 UNB

9 UNB 9 UFPR

10 UFV 10 UFPE

11 PUC-RS 11 UNIFESP

12 PUC-RJ 12 UFSCAR

13 UFPE 13 UERJ

14 UFPR 14 UFSM

15 UERJ 15 PUC-RJ

16 UFSCAR 16 UFC

17 UFC 17 UFBA

18 UFF 18 UFF

19 UFLA 19 PUC-RS

20 UFBA 20 UFV

22 UFSM 14 UFSM

19 UFLA 31 UFLA

BRAZILIAN UNIVERSITIES COMPARATIVE RANKINGS

CAPES RANKING RUF RANKING (2014)

Ranking University Ranking University

1 USP 1 USP

2 UFRJ 2 UFMG+1

3 UNICAMP 3 UFRJ-1

4 UFRGS 4 UFRGS

5 UFMG 5 UNICAMP

6 UNESP 6 UNESP

7 UFSC 7 UFSC

8 UNIFESP 8 UNB

9 UNB 9 UFPR

10 UFV 10 UFSCar+2

11 PUC-RS 11 UFPE-1

12 PUC-RJ 12 UNIFESP-1

13 UFPE 13 UFC+3

14 UFPR 14 UFBA+3

15 UERJ 15 UFSM-1

16 UFSCAR 16 UFF+2

17 UFC 17 UERJ-4

18 UFF 18 PUC-RS+1

19 UFLA 19 UFV+1

20 UFBA 20 PUC-RJ-5

21 UFLA 15 UFSM-1

22 UFSM 27 UFLA+4

Position Institution Courses Articles Position

1 USP 92 40,184 1

2 UFRJ 34 11,696 4

3 UNICAMP 33 12,845 3

4 UFRGS 32 10,325 5

5 UFMG 31 9,260 6

6 UNESP 17 14,563 2

7 UFSC 17 5,439 8

8 UNIFESP 12 8,001 7

9 UNB 12 4,323 12

10 UFV 11 4,680 10

11 PUC-RS 11 1,176 26

12 PUC-RJ 9 994 31

13 UFPE 8 4,498 11

14 UFPR 7 5,102 9

15 UERJ 7 3,498 17

16 UFSCar 6 4,048 13

17 UFC 5 4,013 14

18 UFF 4 3,631 16

19 UFLA 4 2,919 18

20 UFBA 4 2,832 20

Note: FIOCRUZ and EMBRAPA not included in the table,

produced 5,616 and 5,279 articles, respectively.

Sources: CAPES, GEOCAPES, September 2013

and INCITES, ISI, Thomson Reuters, September 2014.

Ranking CAPES: Courses 6 + 7

Ranking of Brazilian

Scientific Production

2009 - 2013

Fig. 2.

From William H. Press 2013, Science 342:817-822

Published by AAAS

outline

uses and abuses of rankings

from rankings to benchmarking

what benchmarking involves

a thin line between love and hate

disagreement with principle

criticism of methodology

boycotts

court actions (New Zealand, Holland)

create your own ranking

methodological problems • validity of criteria

• research bias & hard sciences

• peer evaluations

• English language publications

methodological problems •validity of criteria

• research bias& hard sciences

• peer evaluations

• English language publications

•validity of weights given to various criteria

• statistical robustness

methodological problems

(cont.)

• do all the indicators taken together actually measure quality?

• do the score differences reflect actual quality differences?

• are the rankings comparing the same types of institutions and programs?

danger of rankings

• changes guided by rankings criteria

• priority given to top students (equity concern) and/or

foreign students

• resource allocation (research)

• fraud in data presentation or survey participation,

payment of students

outline

uses and abuses of rankings

from rankings to benchmarking

positive features

system-wide focus

control for size

large range of indicators (multi-dimensional)

methodological problems

overall ranking with arbitrary weights

lack of statistical robustness

mix of results and determinants of these results

selection of indicators partly based on data availability,

not meaningfulness (equity)

mix of data from various years

outline

uses and abuses of rankings

from rankings to benchmarking

what benchmarking involves

what is benchmarking?

• comparing the performance of one’s tertiary

education system/institution to that of other

systems/institutions

• competitors

• good practices

proposing a theoretical

framework

• distinction between performance and health of system

• how good are the system’s actual outcomes?

• does it operate under conditions known to lead to high

performance?

• definition of outcomes / outputs / results

• identification of determinants and causality relationships

criteria for choosing

comparators?

• internal characteristics

• mission / philosophy / values

• types of programs (niche)

• research / teaching / learning philosophy

• characteristics of the environment

finding the data

• existing databases

• ad-hoc surveys

• field visits

• partnerships

• countries

• organizations

steps for benchmarking &

strategic planning

• what and how are we doing now?

• the right thing? is it relevant?

• is it effective and efficient (value for money)?

• what has changed in the environment?

• competition (domestic & international)

• resources

• regulatory framework

steps for benchmarking &

strategic planning

• where do we want to go?

• what sort of institution do we want to be (aims, values

and mission)?

• how do we get from where we are to where we want to

be?

• action plan (tasks, resources, structures)

• incentives

• strategic partnerships

accelerating factors

strategic planning and benchmarking

internationalization

being a niche institution and / or offering niche

programs

curriculum, pedagogical and managerial innovations