Congratulations to Team RooKeys for being a Top 5 Project of the The Erdős Institute’s Fall 2022 Data Science Boot Camp with their project Predicting Flight Departure Delays!

Mathematics Postdocs Anudeep Arora (University of Illinois Chicago) and Oussama (Sam) Landoulsi (Florida International University) teamed up with Physics Ph.D. Candidate Lalit Yadav (Duke University) to form Team RooKeys and create a flight delay predictor. The goal of their model was to predict—four hours ahead—if a flight will be delayed by more than 15 minutes. They combined an airline dataset from the Bureau of Transportation Statistics (BTS) reporting carrier on-time performance leaving JFK with a weather dataset they scraped from weather-underground. Initial exploratory data analysis showed that carrier delay exhibits seasonality and drift, an important relationship to capture in their model. To address this, they normalized delayed flights by percentage of flights delayed within a window of time, then used a cyclic embedding for date/time information and wind direction. Erring on the side of predicting delay—missing a delay is more costly to an airline—they optimized the F2 score. Ultimately, they found that random forest performed best, with a recall of approximately 82-83% and precision of 39%. Logistic regression matched precision, but had slightly lower recall.

They initially started a project in finance; however, in their first meeting with their mentor, they found out this was a common topic in interviews. They wanted to do a project that would set them apart, so they set out to pick a new topic that fit three main criteria: (1) data cleaning must be required for a challenge; (2) predictions must be possible, so something that could be labeled; and (3) more than one dataset to join, so they could get more data handling experience. Ultimately, these criteria led them to flight delay predictions, since the BTS had a publicly available dataset which was easily downloadable and they could add in weather data. Sam wrote a script to scrape the weather data and Lalit figured out how to merge the dataset on the timestamp. This was not as straightforward as one might suspect since the flights were recorded to the second and the weather data was divided into one measurement per hour. To solve this, he mapped the flights to their nearest weather interval to merge the data. Finally, they had to label their data to indicate whether or not a flight was delayed by more than 15 minutes.

Once they compiled the dataset, it was time for exploratory data analysis (EDA), which showed that carrier delay exhibits seasonality—both monthly and time of year—and drift over years (eg., staffing shortages one year for JetBlue). Furthermore, they found delays propagated through the day based on the previous hour and delays tended to occur at the same time as those of the previous day. Both Lalit and Anudeep greatly enjoyed this process of exploratory data analysis, plotting and analyzing their data to find the most predictive features and understand the cyclic nature of their data. Lalit added, where before “I would just plot x versus y [laugh], but now we are plotting two–double x’s and shifted histograms…I started to feel like I have now more grip in data engineering. I can handle different types of data and if someone shows me new data, then I can quickly find out what features are important.” Additionally, “this was also the first time working as a team, before now whatever I had to do I would just do myself, so coordinating, shifting, distributing work, so that experience was also quite rewarding for me.” Anudeep added, “working on the project gave me a good understanding and perspective of the dynamics between data science and business decisions. It helped me understand the workflow starting from researching to provide value to the business/company to data engineering by handling unstructured/imbalanced data to finally analyzing the data and simplifying the useful information via data visualization so that it can be leveraged by business executives.”

In the future, they would like to increase the scope of their model by increasing the number of origination airports and understanding the spread of delays through origination–destination pairs. From here, they suspect they could predict the evolution of a delay in real-time. They think their model could also be improved by incorporating weather information from the destination and topographical features surrounding both airports; they found that weather was a strong predictor of delay (eg., temperature near the dewpoint can lead to fog). Anudeep remarked that the simplest model was the best, but they did so much feature engineering that the base model also did well, so it was a challenge to improve. He wonders if given more time they could further improve on their results. After their first round of modeling they discovered that they could drop the four hour constraint and instead forecast 24 hours out with random forest at 82% recall. Looking forward, Lalit would like to expand their model to create an API or app to not only predict that a delay will occur, but also to predict how long that delay will be.

Anudeep and Sam knew each other through their Ph.D. advisors, and Lalit was a very compatible addition to their team. The team divided up responsibilities, but they were flexible and worked together to improve. Anudeep added that “the discussions in our group were pretty elaborate and receptive,” which helped ensure they were all on the same page. He also emphasized, for future participants, the importance of consulting advisors about project ideas early on in the boot camp, especially “since we don’t have the experience or background from industry and the second—or maybe the primary—goal is to choose a project which will help you get a job,” and they can help you understand “how important how not important [a topic] is.” He believes their project was selected as a top project primarily due to their project mentor’s (Shuvra Gupta) questions and guidance. Lalit encourages all future boot camp participants to get experience in data handling. “Once data is clean, then it’s just a matter of time…the real challenge is to prepare the data, so choose problems in which there is some difficulty in getting the data or cleaning it.”

Congratulations again to Team RooKeys for being a Top 5 Project of The Erdős Institute’s Fall 2022 Data Science Boot Camp!