Voice recognition in the LabTablet electronic laboratory notebook

Susana Ventura Ricardo Amorim João Rocha da Silva Cristina Ribeiro

ei12009@fe.up.pt rcamorim@fe.up.pt joaorosilva@gmail.com mcr@fe.up.pt

Faculdade de Engenharia Universidade do Porto

Faculdade de Engenharia Universidade do Porto

Faculdade de Engenharia Universidade do Porto / INESC-TEC

Faculdade de Engenharia Universidade do Porto / INESC-TEC

Contents

1. Context and motivation

2. Managing research data with LabTablet and Dendro

a. LabTablet, an electronic laboratory notebook

b. Dendro, a collaborative data management platform

3. Speech recognition in mobile environments

4. Speech in LabTablet

5. Conclusions

2

Context and Motivation

3

Researchers know a lot about their data, so we should make it easier for researchers to describe

it adequately.

This should motivate them to do it sooner, encouraging later sharing and reuse.

4

Managing research data with LabTablet and Dendro

5

Gather

Process

Describe

Publish

Researchers Curators

InstitutionsDevelopers

Research Managers

Funders

Data Providers

http://dendro.fe.up.pt 6

LabTabletan electronic laboratory notebook

7

8

LabTablet

● An Electronic Laboratory Notebook (ELN)● Runs on Android devices● Allows researchers to record metadata during field runs

and others● Uses device’s onboard sensors to record metadata (GPS

location, Luminosity, Temperature, Camera…)

9

LabTablet

● Metadata are represented as descriptor values● Descriptors can be generic of domain-specific

○ e.g. “Author”, “[temporal / geographical] Coverage”, “Temperature”, “Depiction”

○ They can depend on the research domain○ Researchers can be assisted in choosing which

descriptors to fill in

LabTablet interacts with Dendro

● Dendro recommends descriptor sets○ Researchers fill in the descriptors during data

production○ Metadata records are pushed back to Dendro ○ Researchers then upload the data to a Dendro folder

● Experiment Metadata + Data combined

10

Why speech recognition?

1. Researchers are free to use their hands while they dictate to the tablet

2. Reduces the amount of interaction with the tablet to produce metadata

11

Dendroan ontology-based RDM platform

12

Screenshot taken from http://dendro-prd.fe.up.pt:3007/project/dendrorecommendation/data/Base%20Data 13

14

File explorer

Metadata Editor

15

Descriptor selection area

16

Speech recognition in mobile environments

17

Speech recognition solutions

● Speech-based apps are becoming a part of daily life○ Google Now (Android)○ Siri (iOS)○ Cortana (Windows)

● Challenges○ Noisy environments○ Large amount of vocabulary

18

Evaluating speech recognition solutions

● Field work means that network access may be limited○ We needed offline speech recognition○ Selected library had to be open-source

● Online solutions, however, are very effective○ Faster translation speed ○ Better recognition overall

● We considered both scenarios

19

Online vs. Offline

● Online → Google Speech Recognition API○ Recognizes full sentences for note-taking○ Always-on speech recognition is taxing on the mobile device

■ Only active during note-taking○ LabTablet allows Portuguese and English keywords when in online

mode

20

Online vs. Offline (cont’d)

● Offline → CMUSphinx○ Training a speech recognition model hinders rapid prototyping○ Limited to basic word recognition

■ Keywords: “Battery” for battery temperature sensor, “luminosity” for light sensor values

○ Dictionary-based recognition■ Some very specific words are not recognized (e.g.

“descriptor”)

21

Speech in LabTablet

22

Application’s field mode

23

Descriptors gathering

“Descriptor”

Which descriptor?

“Description”

Save...

24

Online speech recognition configuration

Customization of:

● Language (ENG or PT)● Voice speed rate● Keywords

25

Conclusions

26

Conclusions

● LabTablet + Dendro

○ Tools to help researchers manage and describe data from creation to deposit

● LabTablet

○ Android-based Electronic Laboratory Notebook for researchers to use on field work or the lab

○ Uses readings from onboard sensors to fill in metadata descriptors

● Dendro

○ A web-based collaborative data management platform

○ Captures data and metadata within the research group

27

Conclusions (cont’d)

● Voice recognition in LabTablet

○ Voice commands for various operations

■ Record audio, take temperature/luminosity reading, record a note…

○ Uses online and offline voice recognition (Google + CMUSphinx)

● Tablet as an unobtrusive companion

○ Hands-free interaction

○ Less touch-based interactions mean less time spent handling the tablet

28

Visit us athttp://dendro.fe.up.pt

Questions

This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the FCT – Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) within project POCI-01-0145-FEDER-006961.

Acknowledgements

30