Lead: Camille Goudeseune
Members: Benjamin Galewsky
Time frame:
AWS Lambda runs code in a way that is serverless, that is, without a server that you explicitly configure and maintain and pay for even while idle. The code consists of functions called lambdas. These are triggered in a fraction of a second by events such as http requests, new data in a table, and many AWS services. The system scales from a few events per day to a few per millisecond. Lambdas can be written in many languages, and even run native Linux binaries. Each lambda uses 128 MB to 3 GB of RAM, up to 10 GB of temporary storage, up to 15 cpu-minutes, and up to 250 MB of (uncompressed) storage between runs.
XRAS/XSEDE (Camille et al) is transitioning to lambas, as a way to collect and distribute HPC job requests.
XRAS's in-flux API is documented here.
Ben uses lambdas (via pyhf, histogram analysis) to make a serverless REST service using gateway for Materials Data Facility.
Ben works on funcX, which is functions as a service for HPC.
Ben's SciPy2021 paper: Distributed statistical inference with pyhf enabled through funcX
Chen Wang does a lot with lambdas for Shield's pipeline and for the social media macroscope, and with AWS Batch whose batch-mode wait time is typically not too long.
Michael Bobak and Dan Katz may also be interested in this.
Similar to lambdas is AWS Fargate, but you need to make your own docker image.
Migrating to alternative platforms:
It's worth understanding what would be involved, but XRAS doesn't yet need to actively plan for it.
Roughly, AWS Lambdas are built on something called Firecracker, and we'd need another way to auto-provision servers and/or micro-VMs.
Google, Oracle and Azure might have similar "serverless" offerings, but message queue code (at least XRAS/XSEDE's) would need a rewrite.