Community

ELIXIR 3D-BioInfo Community (3D-BioInfo)

ELIXIR 3D-BioInfo Community benchmarking activity on scoring and ranking of protein-protein interfaces and quaternary structure assignment, based on a curated benchmark of 1677 homodimer crystal structures.

protein-protein interactionquaternary structureinterface scoringhomodimersstructural bioinformaticsAlphaFold-multimer

Benchmarking Event

Benchmarking of scoring functions for physiological vs. crystallographic protein-protein interfaces

Created: 2023-05-15 · Modified: 2023-05-15

Reference: https://github.com/vibbits/Elixir-3DBioInfo-Benchmark-Protein-Interfaces

Challenges

PPI-PhysioClassification — Discrimination of physiological vs. non-physiological (crystallographic) homodimer interfaces

Implementing the PPI Benchmark as an OpenEBench Experiment

ELIXIR 3D-BioInfo Community · PPI-PhysioClassification challenge · overview

This page implements the physiological-vs-crystallographic protein-protein interface classification benchmark (Schweke et al.) as an OpenEBench-style Benchmarking Event, following the OpenEBench Scientific Benchmarking Data Model (community → benchmarking event → challenge → participants → metrics). It is a working preview, not a connection to the live openebench.bsc.es service.

The challenge

1671 homodimer entries (840 physiological, 831 non-physiological/crystallographic, after curation) are scored and ranked by 14 participant methods — DeepRank-GNN, Rosetta InterfaceAnalyzer, Voronota, DLScore-BC, IES normalized, NIPacking, QSQE, ISPRED4, GOAP, MaSIF, CP/COCOMAPS, shape_hydro_tails, ScanNet ProbaConsensus, and a cross-group Random Forest consensus — each evaluated on 5 metrics (AUROC, AUCPR, Balanced Accuracy, F1, MCC).

Leakage-safe cross-validation

Because many entries share homologous chains, metrics computed by simple resubstitution over-estimate real-world performance. This package instead provides 4 homology-aware CV splitting strategies (CATH/UniProt-based, this package's own MMseqs2 clustering, an authoritative blastclust reference, and a recommended combined union of the two), documented in CV_SPLITTING_STRATEGIES.md, so every participant's decision threshold is learned only from the other folds, never from the held-out one.

What's on this page

Results — the real OpenEBench Scientific Visualizer renders the AUC-ROC-vs-Accuracy leaderboard chart plus a full metrics table and a quartile/cluster ranking view. Datasets and Tools link to the underlying data files and participant tool registrations. Submit new predictions runs the exact same leakage-safe cross-validation engine (ported to JavaScript and numerically validated against the original Python implementation) entirely in your browser — no backend, nothing uploaded anywhere — to evaluate and rank your own scores against the 14 baselines.

OEB Bundle — submitting this benchmark to OpenEBench

The OEB Bundle link in the sidebar downloads oeb_ppi_benchmark.zip, the complete, self-contained OpenEBench submission package behind this page: the Community/Benchmarking Event/Challenge/Dataset metadata JSON files (OpenEBench Scientific Benchmarking Data Model), the 14 participants' and consensus predictor's submission files, the offline copy of the OpenEBench JSON Schemas used to validate them, draft Docker images for each workflow step, and this same interactive page for local use.

To actually import this data into OpenEBench after downloading:

Extract the zip, then run cd oeb_ppi_benchmark && python3 workflow/prepare_for_openebench_submission.py. The wizard replaces the placeholder Community ID and contact emails throughout every metadata file with your real ones, and re-validates each file against the bundled OpenEBench JSON Schemas (schemas/) as it goes.
Build and push the three Docker images under workflow/ (Dockerfile.validate, Dockerfile.compute_metrics, Dockerfile.consolidate) to a registry OpenEBench's VRE can pull from -- every benchmarking workflow step on OpenEBench's infrastructure runs as a container for reproducibility.
Register the participant tools and this benchmarking workflow with OpenEBench (or your ELIXIR community's OpenEBench liaison), providing the Community ID, the Git repository URL/tag, and the Docker image references from the previous two steps -- the wizard drafts a ready-to-send support-request email summarizing these for you.
Once OpenEBench ingests the metadata and runs the workflow, this benchmark becomes a real, citable Benchmarking Event on openebench.bsc.es, and anyone can submit new predictions there the same way this page's own Submit new predictions tab demonstrates locally.

Full detail on what remains to be filled in (real Community ID, registered Tool IDs, contact emails, built images) is tracked in OPENEBENCH_SUBMISSION_READINESS.md inside the bundle.

PPI-PhysioClassification challenge leaderboard

AUC-ROC vs. Accuracy-at-threshold · 14 participants

Community: 3D-BioInfo Challenge: PPI-PhysioClassification Metric X: AUC-ROC Metric Y: Accuracy@threshold

Full metrics table

AUROC and AUCPR are threshold-free ranking metrics. Balanced Accuracy, F1 and MCC are evaluated at each scoring function's published decision threshold, or at the data-driven optimal threshold (Youden's J statistic) when no threshold was published (applies to Voronota only). This is a fixed snapshot of the 14 published scoring functions -- see "Try it yourself" below to evaluate your own predictions with a declared, leakage-safe cross-validation procedure using the full, interactive version of this page.

Group	Participant	AUROC	AUCPR	Balanced Accuracy	F1 Score	MCC	Evaluation
Consensus (cross-group RF)	Random Forest consensus (top-221 features)	0.947	0.945	0.883	0.884	0.765	published threshold (resubstitution)
Bonvin	DeepRank-GNN	0.860	0.816	0.804	0.823	0.618	published threshold (resubstitution)
Schueler-Furman	Rosetta InterfaceAnalyzer	0.849	0.821	0.792	0.802	0.586	published threshold (resubstitution)
Guerois	IES (normalized)	0.821	0.843	0.768	0.811	0.537	published threshold (resubstitution)
Zou	DLScore-BC	0.817	0.802	0.747	0.756	0.496	published threshold (resubstitution)
SWISS-MODEL	QSQE	0.808	0.918	0.724	0.741	0.396	published threshold (resubstitution)
Venclovas	Voronota	0.807	0.768	0.750	0.752	0.501	data-driven optimal threshold (resubstitution)
Fernandez-Recio	NIPacking	0.791	0.745	0.766	0.766	0.532	published threshold (resubstitution)
Casadio (Bologna)	ISPRED4	0.769	0.747	0.720	0.716	0.440	published threshold (resubstitution)
Correia	MaSIF	0.766	0.747	0.708	0.732	0.417	published threshold (resubstitution)
Kihara	GOAP	0.754	0.713	0.693	0.708	0.387	published threshold (resubstitution)
Oliva	CP/COCOMAPS	0.749	0.712	0.693	0.695	0.386	published threshold (resubstitution)
MOBI	shape_hydro_tails	0.731	0.699	0.680	0.686	0.360	published threshold (resubstitution)
Wolfson	ProbaConsensus (ScanNet F_Network)	0.726	0.661	0.678	0.726	0.378	published threshold (resubstitution)

Quartile ranking

Each participant's per-metric quartile/cluster rank, computed independently for each of the 5 performance measures over the current set of participants (higher value is better for all 5 metrics). Follows the same Classification Method currently selected in the chart above (Square quartiles, Diagonal quartiles, or K-Means clustering): Square and Diagonal quartiles reduce to the same percentile-based split for a single metric (the 2D "distance-to-corner" score diagonals uses is just the metric's own normalized value when there is only one axis); K-Means runs real 1-D k-means (k=4) on each metric's values independently, grouping by natural breaks in the data rather than equal population percentiles, with clusters ranked by mean (Q1 = highest-performing cluster). Select "No classification" to hide this table, matching the chart's own behavior. Recomputed automatically whenever a submission is added or the Classification Method changes.

Select a classification method above (Square quartiles, Diagonal quartiles, or K-Means clustering) to populate this table.

Group	Participant	AUROC	AUCPR	Balanced Accuracy	F1 Score	MCC

Want to evaluate your own scoring function against this benchmark? Head to the Submit tab -- the whole evaluation (leakage-safe cross-validation, your choice of splitting strategy) runs directly in your browser, no server needed. The full OpenEBench submission package (metadata, Docker images, the submission-prep wizard) is also available here for those steps.

Datasets

Dataset type	# instances
aggregation	1
assessment	28
input	1
metrics_reference	1
participant	14
public_reference	1

Reference data used for evaluation:
gold_standard_labels.csv (physiological / crystallographic ground truth, 1671 entries) · cv_fold_assignment.csv (Strategy 1) · cv_fold_strategy2_mmseqs2_strict.csv (Strategy 2) · cv_fold_strategy2b_blastclust_reference.csv (Strategy 2b) · cv_fold_strategy3_combined_union.csv (Strategy 3, recommended) · full description of all 4 CV splitting strategies

Tools

Group	Participant tool	Internal ID	Submission file
Casadio (Bologna)	ISPRED4 physiological-score (MCC-optimized)	3DBioinfo:Bologna_ISPRED4	predictions.csv ↓
Bonvin	DeepRank-GNN	3DBioinfo:Bonvin_DeepRank-GNN	predictions.csv ↓
Correia	MaSIF descriptor-distance score	3DBioinfo:Correia_MaSIF	predictions.csv ↓
Fernandez-Recio	NIP (Normalized Interface Packing)	3DBioinfo:FernandezRecio_NIPacking	predictions.csv ↓
Schueler-Furman	Rosetta InterfaceAnalyzer (dG_cross/dSASA x100)	3DBioinfo:Furman_InterfaceAnalyzer	predictions.csv ↓
Guerois	IES homolog-normalized (10 seq, size-normalized)	3DBioinfo:Guerois_IESnorm	predictions.csv ↓
Kihara	GOAP statistical potential	3DBioinfo:Kihara_GOAP	predictions.csv ↓
MOBI	MOBI shape+hydrophobicity+tails integrated score	3DBioinfo:MOBI_ShapeHydroTails	predictions.csv ↓
Oliva	Contact-Propensity (CP) raw score	3DBioinfo:Oliva_CP	predictions.csv ↓
Consensus (cross-group RF)	Cross-validated Random Forest consensus classifier (top-221 features)	3DBioinfo:RF_Consensus_top221	predictions.csv ↓
SWISS-MODEL	QSQE (Quaternary Structure Quality Estimate)	3DBioinfo:SWISSMODEL_QSQE	predictions.csv ↓
Venclovas	Voronota interface energy (normalized)	3DBioinfo:Venclovas_Voronota	predictions.csv ↓
Wolfson	Consensus classifier probability (SOAP+Network+FireDock)	3DBioinfo:Wolfson_ProbaConsensus	predictions.csv ↓
Zou	DLScore-BC	3DBioinfo:Zou_DLScoreBC	predictions.csv ↓

Submit new predictions

Upload a scoring-function predictions file and see how it ranks against the 14 package participants, under a declared, leakage-safe cross-validation splitting procedure -- computed entirely in your browser (the file is never uploaded anywhere).

Every number shown here is a held-out, cross-validated metric: for each fold of the chosen splitting strategy, the decision threshold used by Balanced Accuracy / F1 / MCC is learned only from the other folds (Youden's J statistic), never from the held-out fold itself. This is different from (and stricter than) the plain resubstitution comparison against the manuscript shown in the Results section above. All computation happens client-side in JavaScript -- your file is read locally via the browser's File API and never leaves your machine.

Group name

Submission name

CV splitting strategy full description of all 4 strategies ↗

Predictions file (CSV)

Columns: entry_id (e.g. 1a17_6) and a score column named score, raw_score or raw_score_oriented -- higher value must mean MORE physiological. download a template ↓ or try the bundled demo file.