DTREG Benchmarks of Predictive Model Methods

Benchmarks of Predictive Model Methods The following table shows the results for various types of predictive models applied to a large number of benchmarks. Note that different types of models work best for different types of data.

All of these benchmarks are classification problems (i.e., the target variable is categorical). Some types of models work better with regression problems where the target variable is continuous.

The following predictive models were tested using DTREG:

The numbers in the table are the percent misclassification (error) for each method on each benchmark. Smaller error values imply greater accuracy. The misclassification percentages were computed using 10-fold cross-validation for all methods except for Decision Tree Forests which used Out-Of-Bag (OOB) validation and Probabilistic Neural Networks which used Leave-One-Out (LOO) validation.

Logistic regression could not be performed for benchmark problems that have more than two target categories. Some of the other methods were not suitable for a few of the benchmarks.

Many of the benchmark data sets came from the UCI Machine Learning Repository.

The best result for each benchmark is shown in bold face type. The worst result is in italics.

Percent Misclassification Using Validation Data
BenchmarkSingle treeTreeBoostTree forestSVMANNPNNGMDHCCNN RBFLDAK-MeansLinear Reg.Logistic reg.
Abalone 46 45 45 45 43 44 45 44 44 45 52 45  
Ad 3 3 3 3 16 2   3 9 3 20 45  
AdultCensus 20 14 15 20 15 13 16 15 14 16 44 19  
Anneal steel 11 5 23 11   6 14 9 24 18 11 17  
Argentina currency 27 14 24 32 40 16 29 20 22 24 41 23 28
Astroparticle 4 2 3 3 4 3 4 3 3 10 6 11 5
Audiology 61 19 36 15   9 24 20 19 25 49 20  
AustralianCrabSex 14 8 7 3 3 4 5 2 5 4 13 4 4
AustralianCredit 15 13 13 24 14 12 14 14 15 15 38 15 15
Balance 22 15 18 0 3 10 9 4 9 14 30 14  
Banana shape 10 10 11 10 10 10 32 45 11 44 11 44 44
Bands 29 19 20 25 36 7 33 24 24 26 34 25  
Bioinformatics 38 18 21 16 14 16 20 18 26 17 27 17 21
Bisbey 19 10 12 18 22 14 19 16 25 18 29 1  
Bridges 30 24 36 57 81 25 35 24 31 33 61 27  
Car 4 1 4 0 1 13 13 2 8 10 26 16  
ChurchParticipation 63 56 58 55 54 47 55 49 54 49 56 51  
ClevelandHeart14 58 46 44 48 46 39 42 41 43 40 71 58  
ColonTumor 19 8 16 20 19 10 19 24 35 23 14    
Contraception 44 44 44 46 44 45 46 45 46 49 54 49  
CreditApplication 14 9 13 14 45 12 15 14 14 14 41 15  
Cushing's Syndrome 30 33 18 41 28 7 37 52 30 41 26 44  
DeathPenalty 31 25 28 31 28 7 25 24 24 25 36 24 26
Dermatology 4 3 3 0 65 0 3 4 3 3 9 2  
DNA 7 4 30 4 6 4 7 5 5 6 10 5  
Ecoli 22 14 14 18 15 12 12 13 14 13 18 15  
ElectroCardiogram 26 23 24 25 29 22 27 23 32 26 49 24 27
EvansCounty 12 11 11 25 11 10 11 10 12 10 30 10 10
Federalist 5 5 5 4 5 0 7 6 11 15 10 15 12
Flags 45 32 46 36 31 25 46 44 37 36 62 34  
Fraud 26 29 31 45 31 14 29 33 36 29 31 31 29
GermanCredit 33 24 25 23 25 20 26 24 26 24 45 24 24
GfaNormaux 6 5 6 4 8 2 8 6 5 8 12 7 8
Glass 30 24 22 34 35 12 36 34 28 36 28 40  
GymTutor 7 4 10 4 4 2 7 4 4 11 25 11  
Haberman 30 33 39 33 27 24 27 25 28 25 49 26 25
Hayes-Roth 15 20 33 23 30 16 22 26 21 33 29 37  
Heart13 46 21 16 16 31 14 21 13 17 21 56 15 25
Hepatitis 21 15 21 21 16 6 21 15 16 15 28 14 20
HorseColic 20 17 18 16 17 7 21 21 21 18 42 49  
HOSLEM 50 33 33 34 34 25 33 31 34 31 42 33 33
HouseVotes 5 4 3 3 3 2 5 3 4 4 8 4 3
InsuranceFraud 27 18 31 29 27 5 17 19 24 28 32 24  
Ionosphere 9 7 6 5 10 8 10 10 9 14 20 14 13
Iris 5 3 3 3 3 3 4 4 2 2 4 15  
Labor-neg 12 10 5 7   0 22 12 7 10 15 10 10
Lenses 25 17 42 12 37 4 33 21 17 12 21 21  
Letter-recognition 14 4 0 2 95 2 31 20 14 30 5 44  
LibSvmVehicle 25 16 16 20 17 17 17 17 20 18 21 19  
LiverDisorder 32 29 26 29 30 30 28 29 30 30 41 30 34
LowBwt 36 34 34 35 30 30 29 29 31 31 37 30 36
LungCancer 50 50 87 47 44 6 61 56 62 50 53    
Lymphography 22 14 22 17 15 5 19 16 16 21 22 13  
Marketing 53 48 51 50 93 45 45 44 43 47 60 48  
Microchip 33 35 38 38 34 34 34 34 37 34 49 34 34
Mushrooms 0 0 0 0 0 0 0 0 0 0 0 0  
Musk 24 10 10 5 8 4 27 13 43 18 13 18 18
NLS 42 33 31 31 29 29 30 29 36 30 41 32  
Nursery 2 0 45 0 0 2 13 0 4 47 19 9  
NursingHome 20 7 6 16 7 4 5 5 5 6 25 6 16
OilSpill 12 7 18 10 3 3 4 3 4 3 16 3  
Optdigits 10 2 2 1 4 1 8 3 5 4 2 7  
Pageblocks 8 5 2 8 4 3 5 3 10 5 60 8  
PenDigits 4 2 1 0 3 1 6 2 2 11 1 12  
P.I.-Diabetes 25 24 26 24 23 16 23 22 24 23 29 23 26
PostOperative 58 42 43 51 34 29 39 37 39 38 41 33  
PrimaryTumor 74 59 68 60 79 55 55 52 53 55 70 52  
Reuters 11 5 4 3 3 11   7 13 23 12    
RingNorm 13 2 4 1 3 49 7 7 2 23 18 23 24
SalesPlan 65 59 61 56 63 63 63 59 64 61 66 60  
Satellite 15 8 8 8 11 8 14 12 10 16 10 24  
Segment 5 2 2 0 3 3 7 5 17 8 4 15  
Shuttle 1 1 1 0 0 0 5 0 21 6 21 13  
Smoking 65 49 41 66 31 31 32 32 32 32 55 31  
Sonar 24 13 13 13 21 1 28 26 26 24 16 24 26
SpamBase 7 6 5 6 7 9 11 7 39 10 34 9 7
Spectf 27 17 20 21 24 6 25 22 27 41 26 41 39
Splice DNA 5 4 35 3 5 3 7 5 4 5 10 5  
Spiral 47 42 47 8 60 24 52 57 24 51 23 51 51
SvmTumor 46 25 24 24 30 60 38 33 83 33 27 30  
Tae 51 47 61 51 60 48 56 54 53 50 44 52  
Thyroid (ANN) 2 2 3 3 5 2   1 1 6 56 7  
Tic-tac-toe 6 1 1 0 2 2 26 2 2 2 14 2 2
Tin 49 29 28 34 26 24 27 26 24 26 50 26 26
Titanic 21 24 22 21 21 21 22 21 21 22 30 22 22
TorchClassif 16 9 7 9 12 3 28 13 12 27 8 27 27
Twonorm 15 2 3 2 2 2 11 2 3 2 3 2 2
UTI 69 78 85 71 58 64 74 64 54 62 80 54  
Vehicle 28 24 24 14 19 25 27 22 15 22 35 24  
Vibration 60 60 60 61 50 48 50 49 48 51 68 51  
Vowel 17 7 3 2 5 0 20 11 7 28 1 37  
Waveform 22 15 15 13 13 15 16 13 14 14 15 14  
WBDC 7 5 6 4 36 6 7 7 6 7 14 7 7
Wine 8 3 2 1 2 0 3 2 1 2 24 2  
Zoo 12 7 7 4 5 1 5 4 6 9 4 4  
Average error 24.82 18.78 22.18 20.04 23.86 15.29 23.33 19.95 21.63 22.71 29.87 23.28 21.05
Median error 21.00 14.00 18.00 16.00 19.00 10.00 21.50 16.00 19.00 22.00 27.00 21.50 24.00
Num. times best 5 13 8 24 13 53 2 14 7 4 1 7 2
Num. times worst 14 0 8 2 9 1 4 2 7 4 37 11 0
Benchmark Single tree TreeBoost Tree forest SVM ANN PNN GMDH CCNN RBF LDA K-Means Linear Reg. Logistic reg.