Within weeks of data scientist Michael Draugelis arriving at Penn Medicine, he dove into building the first predictive analytics project at the $4.3 billion healthcare system: early detection of patients with severe sepsis and septic shock. The healthcare organization already had an early warning system based on heuristics to find those most at risk. Draugelis believed a machine-learning algorithm could do better.

The model performed well during a test run. But when deployed to three downtown Philadelphia hospitals, results were mixed. Patients were being transferred to the ICU three times faster, but mainly because care providers already knew who appeared susceptible.

“There was no great knowledge gap, as we had suspected in our original problem statement and key hypothesis,” Draugelis explained in a recent webinar, Lessons Learned in Advancing Predictive Analytics in Healthcare, sponsored by Comcast Business. “The care teams knew which patients were at greater risk and were providing them heightened care already,” he said.

That early entry into the world of predictive analytics may not have gone as planned, but it hasn’t dissuaded Penn Medicine from developing other data-driven systems to advance treatments and preventions in the years since.

“Using data to drive out a model of the world to make a prediction is really what we’re looking at,” he said. Though Penn Medicine’s terabyte of healthcare data pales in comparison to other enterprises’ petabytes worth, it’s providing the building blocks to more personalized screenings, diagnoses and treatments for patients. It may even change the course of care.

Why analytics is ‘big’ now

Big data applications, particularly within precision medicine, remain an abstract concept to some healthcare leaders. But predictive modeling’s promise to improve workflow efficiencies and more rapidly diagnose and treat patients is being realized, thanks to inexpensive storage and computing power, the rise of distributed processing and wealth of open-source software collaborations.

To work well, predictive analytics systems must use “clean” healthcare data (and lots of it), a connected enterprise, clinical expertise and readiness and, importantly, excellent project management. “The projects are very complex and it takes a huge group to work together to accomplish,” Draugelis noted. “So, the success of these projects is in large part due to a culture of project management.”

Penn Medicine, with more than 2,000 physicians across multiple medical facilities, uses an executive steering committee to determine the need and viability of proposed projects.  Committee members decide on several small projects and two to three major initiatives annually that align with the organization’s mission and strategic goals.

Traditional data analysts or business intelligence experts have long used analytics to support key decisions, such as where to build a new facility or unit. “There’s a lot of great work to be done on the data analyst side without these model-driven solutions,” he said, “but there isn’t a lot of work being done yet in deploying clinical decision support tools to forecast outcomes in a way that would allow intervention and change the course of care.”

Penn’s project planning

Penn Medicine prefers to build its models and algorithms, mainly because Draugelis sees few bulk solutions in healthcare that align well with complicated workflows and data feeds. Larger predictive analytics projects take six to 12 months to produce and include the following phases:

Exploration: An interdisciplinary team decides if a problem is worth tackling with predictive analytics technology. Is there a clear metric return for the patient and the health system? If yes, then a hypothesis is developed and the project is sent to the steering committee, which ranks its relevance.

Proof of concept: Once the project rises to the top, it’s time to build an algorithm that works in a clinical trial.

Final design, build and feasibility testing: This phase often is among the most challenging, especially when it involves different entities or multiple lines of business.

Deployment: Once in production, it’s time to measure, control and re-release as needed.

Challenges working in healthcare

“I knew this would be challenging, but I was surprised how challenging this is,” Draugelis said, referring to working with healthcare data that often had tremendous latency and/or data entry issues and errors for a variety of reasons. He recommends creating a data governance committee to develop and enforce data entry standards. “We also really need to drive towards automation. The data collection we have now is not random; it’s biased. Nurses tend to collect more data for patients they believe are sick.”

Another ongoing issue involves machine-learning systems in general. Penn Medicine’s data scientists found it difficult to create algorithms good enough for highly active clinical settings. For instance, early models had real-time dependencies: Every time there was an update to the data, the system produced a prediction, leading to millions of predictions.

“We discovered that the performance of the model in operation was different than when we trained because there were massive amounts of deleted data and massive amounts of corrected data. We couldn’t recreate the performance of the model in operations,” he detailed. “So, when we wanted to update the model, we had to run the model in silent mode to watch how it performed.”

Finally, there’s the challenge of managing expectations, given current limitations. “When data changes, when the concept that the model is trying to predict changes, either through populations shifts or even updates in care, there usually is not this daily monitoring of the system to catch those things,” Draugelis said.

This is where periodic A/B testing can be of benefit. “It’s a way to disentangle the effect of the algorithms with the care,” he explained. “For example, in healthcare, what if we’re trying to predict if someone will be readmitted, and there’s care provided so they are not readmitted. On face value, that is a false positive, but in fact that was actually an outcome we wanted.” That’s where random controls testing, or A/B testing, comes in. 

This brings us back to that initial predictive system Draugelis’s team developed to find patients with severe sepsis or septic shock upon admittance to a Penn Medicine hospital. “We were thinking that we were finding some unsuspected decline or late detection that was causing that outcome to the patients,” he recalled. “So, if we could improve this knowledge gap for the care team, they could act in a way they otherwise wouldn’t be unable to do.”

But, they discovered, no such gap existed. Clinical response teams were already aware of high-risk patients and responding with a higher level of care. Of the 50 to 60 patients involved during that first month’s deployment, 22 were admitted directly to the ICU. Of the 34 that remained, 22 came into the hospital showing sepsis symptoms. That left only 12 who didn’t present symptoms during admittance.

“So that was our opportunity,” Draugelis determined upon reassessment. “We would have to wait two years to know if we were achieving success with this method.”

In the meantime, he learned to clearly identify and update the clinical opportunity throughout a project so that the system performed as intended: to alert care teams when an incoming patient may be prone to sepsis despite showing no signs their body was reacting to a serious infection. He also learned there’s tremendous promise in building algorithms that can move the needle in personalized care and precision medicine.

Read this article in PDF version.

Produced by HIMSS Media
HIMSS Media is the fastest growing B2B media group focused exclusively on healthcare and technology markets. Through its suite of market-leading brands, such as Healthcare IT News, Healthcare Finance and MobiHealthNews, HIMSS Media delivers news, analysis and must-have information to an audience of senior healthcare and technology influencers. HIMSS Media is also the leading producer of important live events, such as the Privacy & Security Forum, Patient Engagement Summit, Revenue Cycle Solutions Summit and Big Data and Healthcare Analytics Forum.