Fan Zhang

Title Ordinal-Optimized Scheduling for Large-Scale Cloud Data Processing

Processing large-scale cloud data requires the just-in-need compute resources. This becomes more important in processing IoT data in a 5G network, where optimized AI models need to fit into various edge computing devices, with restricted/limited computing capability. One of the major challenges would be the scheduling overhead, which usually takes extended amount of time to achieve satisfactory scheduling decisions. Simulation-based Optimization and Ordinal Optimization were initially proposed in the control science community, that have been proven to be useful for optimizing such workloads on very large cloud. In this talk, I will be presenting a series of innovative work I’ve done along the Ordinal-Optimized (OO) line, and their applications in large-scale cloud data processing. Specifically, the multi-objective OO scheduling for optimizing conflicting computing objectives, the iterative OO (iOO) scheduling for fitting the optimization approach to the multi-phase, time-series workloads, and the evolutionary OO (eOO) for further extending the methodologies to the simulation-based learning territory. This line of research has brought significant impact to the computing of the Laser Interferometer Gravitational-wave Observatory (LIGO) project, where processing terabyte of data per day from hundreds of distributed sensors on thousands of servers, has achieved 10X-100X speedup. They are being used daily in the LIGO pipeline in MIT and Caltech data centers for real-time LIGO data processing.


Dr. Fan Zhang is currently a Research Scientist with the IBM Massachusetts lab. He was a postdoctoral associate with the Kavli Institute for Astrophysics and Space Research at Massachusetts Institute of Technology. He received his Ph.D. in Department of Control Science and Engineering, Tsinghua University in Jan. 2012. From 2011 to 2013 he was a research scientist at Cloud Computing Laboratory, Carnegie Mellon University. An IEEE Senior Member, he received an Honorarium Research Funding Award from the University of Chicago and Argonne National Laboratory (2013), a Meritorious Service Award (2013) from IEEE Transactions on Service Computing, two IBM Ph.D. Fellowship Awards (2010 and 2011). His research interests include big-data scientific computing applications, simulation-based optimization approaches, cloud computing, and novel programming models for streaming data applications on elastic cloud platforms.