SEREBENCH: Adversarial robustness benchmarking for security and reliability event classification in networked railway cyber–physical systems

This paper presents SEREBENCH, an open-source security and reliability benchmarking framework for testing the adversarial robustness and federated resilience of AI-based attack and anomaly classification in networked railway Cyber–Physical Systems (CPS). In this work, SEREBENCH is instantiated and evaluated through a pneumatic braking subsystem case study, where pressure dynamics provide a physically interpretable basis for controlled robustness evaluation. The framework provides a simulation-based environment emulating a railway CPS system, enabling controllable class semantics and verifiable ground truth. This environment supports controlled adversarial robustness benchmarking by progressively stressing the decision boundaries of Machine Learning (ML) models. To this end, SEREBENCH includes a train simulator that produces telemetry, preserving explicit cause–effect relations among signals and operating modes, allowing perturbations to be interpreted as changes in system behavior rather than as arbitrary feature edits. Adversarial robustness benchmarking can be executed in multi-vehicle settings, allowing the study of federated resilience effects, i.e., how robustness-related behavior changes across vehicles under model aggregation. The experimental campaign considers multiple classifier architectures and federated aggregation strategies, targeting the classification of security and reliability events under controlled robustness evaluation conditions. Specifically, the study explores configurations combining federated learning, noise-augmented training, controlled Gaussian boundary stress, and HSJA-based hard-label black-box adversarial evaluation. The experimental campaign also includes a focused FL-flow sensitivity analysis in which controlled message loss and delivery delay are applied to uplink and downlink model exchanges. The obtained results show that SEREBENCH can support adversarial robustness benchmarking of ML models under controlled, braking-oriented railway CPS conditions.