On the efficacy of per-relation basis performance evaluation for PPI extraction and a high-precision rule-based approach

Table 5 Performance of the "ours+SVM" model with incremental feature sets

Min #Inst. per Relation	#Uniq. Relation		#Instances		Methods	Per-Relation					Per-Instance
	Pos	Neg	Pos	Neg		TP	FP	Precision	Recall	F-Score	TP	FP	Precision	Recall	F-Score
1	618	2,312	1,000	4,834	ours+SVM	152	44	0.776	0.246	0.374	172	67	0.720	0.172	0.278
					+dep. len.	149	35	0.810	0.241	0.371	170	52	0.766	0.170	0.278
					+dist.	150	39	0.794	0.243	0.372	171	63	0.731	0.171	0.277
					+both	151	38	0.799	0.244	0.374	173	58	0.749	0.173	0.281
2	197	695	579	2,827	ours+SVM	94	36	0.723	0.477	0.575	116	63	0.648	0.200	0.306
					+dep. len.	94	32	0.746	0.477	0.582	117	53	0.688	0.202	0.312
					+dist.	92	35	0.724	0.467	0.568	117	54	0.684	0.202	0.312
					+both	94	32	0.746	0.477	0.582	116	51	0.695	0.200	0.311
3	89	314	363	1,835	ours+SVM	58	22	0.725	0.652	0.687	78	32	0.709	0.215	0.330
					+dep. len.	57	22	0.722	0.640	0.679	76	31	0.710	0.209	0.323
					+dist.	57	22	0.722	0.640	0.679	75	31	0.708	0.207	0.320
					+both	57	22	0.722	0.640	0.679	76	31	0.710	0.209	0.323
4	51	181	249	1,349	ours+SVM	38	15	0.717	0.745	0.731	55	25	0.688	0.221	0.335
					+dep. len.	36	13	0.735	0.706	0.720	52	19	0.732	0.209	0.325
					+dist.	38	15	0.717	0.745	0.731	55	25	0.688	0.221	0.335
					+both	36	13	0.735	0.706	0.720	52	19	0.732	0.209	0.325
5	28	107	157	984	ours+SVM	25	12	0.676	0.893	0.769	40	18	0.690	0.255	0.372
					+dep. len.	25	11	0.694	0.893	0.781	40	15	0.727	0.255	0.378
					+dist.	25	12	0.676	0.893	0.769	40	18	0.690	0.255	0.372
					+both	25	11	0.694	0.893	0.781	40	15	0.727	0.255	0.378

¹Note that the result shown here is different from the ones reported in [6]. It may be due to the differences in SVM optimization parameters used for the experiments. We obtained the codes from the authors' web page at http://staff.science.uva.nl/ $\tilde{b}$ ui/PPIs.zip and ran as is with the parameters: RBF kernel gamma - 0.0145; C = 9; Weka Cost-SensitiveClassifier optimization.
²In [20], the authors reported the macro-averaged precision, recall, and F-score, which are incomparable to other performance results. Following the general convention in PPI research, we compared the performance using the precision, recall and F-score computed with only positive class prediction results. The original implementation was not available. We implemented it on SVM-LIGHT-TK ver 1.2 obtained from http://disi.unitn.it/moschitti/Tree-Kernel.htm. The optimization parameters used are C = 8 and λ = 0.6 (as reported in [20])

ISSN: 1472-6947