Table 19

Manual analysis results

EPI

ID

Manual

Gold standard

Manual

Gold standard

Source

Corr.

Acc.

Inc.

Match

Prec.

Source

Corr.

Acc.

Inc.

Match

Prec.


GOLD

96

1

3

100

100.0

GOLD

94

5

1

100

100.0


MSR-NLP

87

1

12

83

77.60

FAUST

83

7

10

68

66.35

FAUST

85

0

15

79

80.25

UMass

77

4

19

66

62.39

ConcordU

83

0

17

79

76.71

Stanford

67

7

26

55

56.37

UMass

80

0

20

74

73.30

PNNL

68

4

28

49

52.62

UTurku

75

0

25

72

69.20

UTurku

61

8

31

38

49.91

Stanford

74

0

26

68

70.22

ConcordU

55

5

40

34

43.37

CCP-BTMG

71

0

29

65

63.37

PredX

42

5

53

36

35.18


Number of manually evaluated events judged correct (Corr.), acceptable (Acc.) and incorrect (Inc.) out of samples of 100, with number matching gold annotation (Match) and, for reference, core task precision (Prec.). Analysis performed independently for EPI and ID tasks.

Pyysalo et al. BMC Bioinformatics 2012 13(Suppl 11):S2   doi:10.1186/1471-2105-13-S11-S2

Open Data