A Serverless Architecture for Efficient and Scalable Monte Carlo Markov Chain Computation

This research project combines serverless computing, Markov Chain Monte Carlo (MCMC) methods, and astrophysical applications within a highly interdisciplinary framework. We developed a serverless architecture based on Amazon Web Services that efficiently adapts to the computational requirements of MCMC estimations to investigate problems presenting computationally expensive likelihood functions. We demonstrated in a test-case scenario the high potential for scalability of this architecture by increasing the number of random walkers, still considering the inherent limitation of parallelizing MCMC procedures. Remarkably, all chains returned computation results for the same iteration in approximately constant time, a result that can be exploited by taking advantage of communication among different chains at negligible costs. As an astrophysical application on real data, we presented an innovative analysis through a Bayesian Hierarchical Model on the pressure profile of a population of galaxy clusters using restricted cubic splines as a flexible modelization method. The population analysis follows a formally correct approach in modeling the statistical dispersion across clusters and provides accurate estimates. Our galaxy cluster pressure profiles are in agreement with those obtained by other authors. In the context of astrophysical analyses encompassing a large number of objects, a circumstance increasingly common due to the unprecedented volume of observations from new-generation surveys, our study demonstrates the potential to combine astrophysical insights supplied by a population analysis with the scalability offered by serverless MCMC.