The test was performed on a benchmark set containing 170 proteins with known structure. If experimental constraints from the PDBTM database would have been used, the test had resulted in 100% per protein prediction accuracy. Therefore, the topological constraints were not used in this test. Thus the real accuracy of CCTOP is above of these values.
| Method | Sensitivity | Specificity | MCC | AccTpg | AccTpl |
|---|---|---|---|---|---|
| CCTOP | 98 | 98 | 97.7 | 84 | 81 |
| TopCons | 97 | 97.7 | 97.2 | 79 | 75 |
| ScampiMsa | 97 | 96.7 | 96.8 | 76 | 72 |
| Pro | 96 | 97.3 | 96.5 | 75 | 70 |
| Prodiv | 96 | 94.8 | 95.5 | 75 | 69 |
| Octopus | 93 | 98.2 | 95.7 | 71 | 66 |
| HMMTOP | 95 | 94.7 | 94.8 | 69 | 64 |
| Philius | 95 | 97 | 95.9 | 71 | 64 |
| TMHMM | 93 | 97.2 | 94.9 | 66 | 59 |
| MetaTM | 94 | 97 | 96.0 | 67 | 58 |
| Phobius | 93 | 97.3 | 95.1 | 62 | 56 |
| Memsat | 94 | 98 | 95.8 | 66 | 53 |
| MemBrain | 92 | 97.4 | 94.7 | 62 | 0 |
We estimated the helix boundaries accuracy by BLAST searching all sequences in the OPM database against itself, and compared the transmembrane segment boundaries of proteins, which have 40% or larger sequence similarity to each other. By using the same method, we measured the transmembrane segment boundaries difference between OPM and CCTOP, and got the same result (3.3±0.5 residues). Thus, CCTOP predicts helix boundaries as accurate as it can be determined from the 3D structures by biophysics based method. For comparison, we determined the helix boundaries prediction accuracies of the used methods as well:
| Method | Average distance (residues) |
|---|---|
| OPM | 3.3±0.5 |
| CCTOP | 3.3±0.5 |
| Octopus | 3.6±0.3 |
| Memsat | 3.6±0.6 |
| PRODIV | 3.7±1.3 |
| Scampi | 3.7±1.3 |
| Phobius | 3.8±1.1 |
| Philius | 3.8±0.9 |
| PRO | 3.8±1.1 |
| TMHMM | 3.9±0.9 |
| Membrain | 4.0±1.0 |
| HMMTOP | 4.2±0.7 |