Gennaro Maida, CEO and Founder at Denovo Innovations Group

 [00:00:00] Welcome back to Global Trials Accelerators, the podcast where we dismantle the barriers to clinical innovation and explore the strategies driving the next generation of life-saving therapies. Every wearable device on the market claims to measure something clinically meaningful: heart rate variability, blood oxygen, arrhythmia risk, to name a few.

But there's a wide gap between a sensor reading and something a clinical trial or an FDA will actually accept as evidence. Closing that gap is its own discipline, sitting squarely between signal processing, clinical medicine, data engineering, and regulatory strategies. Our guest today lives in that exact gap.

Renato Meira is the founder and CEO of the Noah Innovation Group, where he helps med tech and digital health

 [00:01:00] companies build AI architecture, navigate regulatory strategies, and hit the mark. He also serves as the Chief Medical Information Officer at Sena Research Institute, leads clinical products at Tactical Rehabilitation, and sits on the American's Heart Association Health Tech Earning recognition as Most Influential Healthcare Leader by CXO Outlook in 2023.

With a biomedical engineering degree from Duke University and a master's focused on heart surgery outcomes, Renato Maeda bridges the gap between raw optical data and clinical evidence. His work centers on secure remote data collection, making complex health records accessible to AI, and tackling what he calls the ground truth problem, keeping AI 

[00:02:00] estimates strictly separate from verifiable clinical facts.

Renato, it's a pleasure to have you with us today. Welcome to Global Trials Accelerator's show. Thank you, Jesus. I greatly appreciate it. I'm excited to have a discussion with you. Uh, it's a very interesting topic that a lot of people are trying to tackle right now Yes, indeed. It's a hot topic. And, and but before we, we tackle that, um, you know, the present day of clinical innovation, I would like to stay-- take a step back and, and visit your, your early days.

You trained as a biomedical engineer at Duke University, then went on to a master's degree focused on cardiac surgery outcomes. What was it about the early clinical exposure to cardiac surgery specifically that provi- pr- pointed you towards, uh, signal processing and AI rather than staying 

[00:03:00] on a purely clinical or surgical track?

It, it's a great question. I'll step even back a little bit further. I ha- grew up in a family of physicians, and they range from ophthalmology, cardiothoracic surgery, pulmonology, and then my grandfather was one of the fellas that ran around with a black bag in his neighborhood in New York, going from door to door.

Over the last years, we have actually lost that personal-- personalization in medicine, and what I started to see is ways to bring the black bag back through technology, being able to gain these-- u-use the data that's out there and being able to personalize medicine. So that led me to the biomedical engineering to really understand the systems, and biomedical engineering is a lot of fun because you cover everything from material science to electrical engineering, so and some biochemistry, and you get to see engineering across the board from biomechanics and beyond.

And then layering on the master's, where we focused on, uh, cardiothoracic 

[00:04:00] surgery, CABG surgery, uh, outcomes, and then the clinical medicine. And then around my third year, I said, "You know, I really, I really enjoy the building." So I stepped away and spent my next few years in infrastructure, understanding how the sensors, the electrical, the networks all work together to gather this data, and then put the data science on top of it to pull the medicine, the research, and the infrastructure together to try to solve these problems That's fascinating, and I, I agree.

It's, it's a discipline that opens up many possibilities and, and can support many new, uh, avenues of research and, and opportunity for healthcare. Uh, but transitioning from a static outcome, uh, to the real-time computer vision, um, is, is a significant leap. Uh, you founded the Novo Innovation Group during COVID-19 specifically 

[00:05:00] to solve problems in remote medicine.

Um, and I think that aligns well with, with that interest that put you down this path. Um, and furthermore, uh, on your early-- one of your early projects was co-founding Vital Intelligence, uh, extracting vital signals like heart rate straight from ordinary video computer cameras and, and, um, using computer vision to, to, um, extract that biomarker.

Uh, what did that project teach you about how fragile or how robust a biomarker actually is when it, when it's pulled from a consumer camera instead of a clinical-grade sensor? Well, what I think one of the biggest lessons I learned, it's not that what they would call reflective image photoplethysmography.

It's the same technology that's in your watch or in an O2 sensor on your finger. We just use the light coming off the face. 

[00:06:00] If you had done this 15 years ago, it would've cost $100,000 to build the system. The cameras weren't there, the connectivity, et cetera. By the time we did this in 2020, the Logitech C120 was powerful enough to do it.

The lap-- we did it on an Intel NUC with a RTX 2070 from Nvidia, and a Core i-- I believe it was a Core i5 with, uh, six core 12 threads. So suddenly, all of this technology has moved along to make some of these things a reality. So it's not the, that they haven't been done in the past, it's just the technology's starting to catch up with some of these things.

We're seeing it with AI, high-powered computing, and all of the different problems being solved across the board, whether it's memory bandwidth, connections between the different RTX cards or distributed computing, better sensors, and gathering all of this data in real time. So I think it's a convergence of a lot of technologies all at once, whether it's the cell phone, 5G, Starlink, eventually quantum computing, AI, new 

[00:07:00] algorithms, and the list goes on and on, and they're starting to come together, and then you have AI that's helping to improve all those technologies at a faster and faster rate.

So with Vital Intelligence, I was asked if you could detect infection from a drone, and there were some crazy ways to do that, hanging antibodies and some other, uh, interesting ideas, but they weren't cost-effective. So I stepped back with the clinical training and said, "Can you detect symptoms of infection from a drone?"

And suddenly the question changed. You started to look at microwave, uh, millimeter length radar, and all the way through, uh, infrared and all the way through, uh, RGB cameras, and I found a team out of South Australia led by Dr. Jivan Schall that had actually done it. And, uh, so we did a tech transfer from there, and it turns out they have worked closely with DARPA on some of the signal processing for some of the drones that are out there.

So they were very advanced, uh, with this, and they actually did IRB trials and were able to do it on neonates in the intensive care unit using

 [00:08:00] cameras. So we adapted it and used it for COVID detection, and we ran into all kinds of interesting, uh, issues there as well. But it's really a convergence of all of the technologies making it possible now, being able to process this data in real time that might have been possible 10 years ago, but the cost would've been astronomical.

So that development on the commercial grade, uh, sensor technology has allowed that distance between the patient and the recording device to increase from what you would find in a clinical setting where, where the device is practically on top or inside the patient even to a greater distance and, and to a broader, um a broader range of situations, uh, and ultimately that- Yeah

that, you know, brings flexibility and accessibility to the system. 

[00:09:00] Ab- absolutely, because you very often see new technologies come out, and they're very expensive, whether it's a drug or a new device, and a lot of that money's going towards the research and development of that device, and the only customers that can initially afford it are big pharma or large enterprises.

And over time, you start to see those c- those different devices start to show up in the consumer world, and they start to become cost-effective. A great example is the thermography camera that we use, the FLIR A400, uh, has an MSRP of roughly $20,000, and it was the only one FDA approved to do thermography.

Now I'm starting to see different groups use different cameras from FLIR that are smaller, still quite accurate, uh, not to the level of the FDA, but you start to get a camera that's in the $500 range rather than $20,000 range. Mm. So it depends on what you're trying to do, but these technologies start to change prices and make them accessible on consumer level hardware as they shrink and get iter- iterated over time.

That adoption is, is 

[00:10:00] crucial. But, and it's funny that you mentioned the, the economic side of things or, or the cost side of things, because beyond clinical signals, you've also applied AI to complex regulatory and coverage policy side of, uh, of the side of healthcare. Uh, like building a conversational AI workflow to navigate complex reimbursement decisions that traditionally take weeks to resolve.

So in, in that experience, what, what made you tackle healthcare policy and- ... and, yeah, the device building, um, to, to the billing side of things, what makes those administrative workflows, uh, surprisingly trick... Because those workflows are, are surprisingly tricky to automate with AI. It, it needs to be consistent, needs to be, uh, explainable even.

So, uh, that, that's an interesting leap from the 

[00:11:00] patient-focused, uh, device measuring to administrative workflows that relate to, to healthcare. Uh, and that's an excellent question. Uh, you know, part of it's working in startups. You end up... People ask you, "Can you do it?" And I have this problem where I always say yes.

Uh, so, you know, I've built a lot of things that have been very interesting. And when you're in the med tech world, eventually, as you find out in engineering, I started over here and said, "Hey, everyone's gonna love this device. It's great." And the reality is there's an economic side of it. And whether it's F- FDA approvals and reimbursement and third-party insurances, you have to touch on it at some point.

And I work with a group, uh, Tactical Rehabilitation, that's a durable medical goods company that works primarily with active duty military. And they asked me to bring them new devices that they could bring into TRICARE to get reimbursed. And every time I would bring a new device in, going through the questions, asking Humana, TriWest, all these different groups, "Can you actually bill this?"

Would take 

[00:12:00] a long time. And more, more often than not, the answer was, "No, we can't." And somebody asked as, sort of as a side question, "Can you build this as a chatbot?" And I said, "Sure, this will be easy." And, uh, then I found out that was my first real experience building something to this scale because, you know, uh, the TRICARE was a great example, uh, because there's four man- four main manuals, there's hundreds of revisions.

And then you have the DHA overseeing it. You've got TriWest, Humana, TRICARE Overseas, ISOS, and it's supposed to be based on CMS on a state-by-state basis, but there's a government do not cover list. And then just because TRICARE, uh, approves something doesn't mean that the contracts with Humana and TriWest actually adopt it.

So there's still human in the loop questions 'cause some of this stuff is still unverified. And what it ended up really being was a huge data engineering problem. A great example would be if you ask a question and it's repeated in the first 252 revisions, and only in 253 does it change to the right answer.

[00:13:00] You do traditional vector matching, it matches 252 times, and then one time here. So your correct answer is buried. How do you account for all the repetition? 99% of the manuals are repeated. There's only a very small amount that's updated on each revision. And trying to account for things like that, or if somebody asks you a question, uh, "Can this person get reimbursed for hearing aids?"

It depends. So you might get a match and it'll say on the one paragraph, "Yes, they're reimbursable." The previous answer said, "Well, if you're in North Carolina," and the previous section said, "If you're under 22." And traditional vector matching will miss that, so suddenly you have to expand. And then being able to expand it into the DHA rules, CMS rules, and ul- and ultimately you solve a lot of the problems with the context side of it.

The other side is the process, and working with groups that actually do this for a living and saying, "Well, I'm going to go here and look at this, and then here, 

[00:14:00] and then we're gonna go ahead and cross-reference this. And eventually we do need to pick up the phone here." And understanding that there might be that section.

And as you're pulling each of these pieces of data, you keep the provenance where it came from, what revision, what date, what manual, what was in effect. So you start to look at the hierarchy, you look at the temporal, you look at the contextual. And then they also threw in tables, pictures, and of course online when you look at the TRICARE manual, some of the images are upside down, the HTML is broken.

So it turned out that one, you know, you start to see that is 95%, and a lot of these are 95% context engineering and data engineering. And if you Bring the right context in, even smaller models are much more accurate. And it's really a difficult part of it. You know, and you start to see this with the large language models where it was trained on everything on the planet, not all of it was great data.

And now as they're starting to bring that in and start to try to clean the data and bring it in standard manners, they're

 [00:15:00] starting to find out that those language models are much more powerful, and the smaller language models are much more powerful with clean data that allows you to give the next predicted answer, which is ultimately what they do that's correct.

So it was a side question that somebody asked me, and initially I thought it would be relatively easy. And the more I dug into it, the more problems were uncovered on how difficult a problem it actually is trying to solve it, whether you're using vector databases, vector matching graph databases, how to organize the data, how do you hop between the data, how do you pull what's-- pull, pull it in while keeping it in a sm- keeping the context fine-tuned enough that the answers will come out, let's call it correct.

There will be fewer hu- hallucinations, and it'll actually answer the questions that you're asking. Hmm. And isn't that often the case that th- the seemingly simple questions end up bringing about, uh, a plethora of opportunities 

[00:16:00] and, and fine details that help fine-tune other systems or, or better understand problems?

Uh, uh, but I'm curious, across all those roles, De Novo, Sena, uh, Tactical, Rehabilitation, is there a single operational principle you carry into every one of those? When I'm working with AI, and I'm working on a project now trying to fine-tune the systems, is to make sure everything has a provenance and is based in ground truth.

You know, I'm working-- The problem I'm trying to solve is we want humans to work beyond their skill level, and we're being guided by a system that could lie to you. So how do you actually align those? And I'm ev- I'm, I'm actually currently working on a system for memory that every section of memory is actually validated in ground truth.

Unless it-- And it can actually state that it's unverified, trying to get it to be honest. And being able to look whether it's making sure you're looking at the right manual and time, 

[00:17:00] or you're actually looking at your deployed infrastructure is it's actually what's correct, or your errors, and being able to look at the logging.

And to making sure each question has some type of provenance to be able to answer the question correctly. The one I'm having probably the most difficulty with is, are temporal questions. In, in memory, or ChatGPT is a great example, I have every chat saved for all time. Sometimes I'll ask a question and it'll base it on something I did two years ago, which is not the answer I need.

I need it based recently. And you see this in research. Research develops over time, and when you ask AI a question and it has access to the full time, it might give you an, a question that's biased towards old answers. So being able to look at, and again, it's the provenance across the ground truth what's going on, but also making sure that it's focused on the correct points in time, whether a decision has been superseded, if it's stale after a pers- a 

[00:18:00] specific time.

And when you start to look at large documents, each section might be slightly different. A section might be superseded, but one might still be in, you know, enactment. Google's working on something with like this right now called the Open Knowledge Format that's on, I think, 0.2, and they're on- only just starting to look at the temporal side.

So everything I try to do, I'm trying to build systems that are based in truth and real data in real time And that's the ground truth problem as you've described it. And it, it has its, uh, shall we say, uh, mirror, uh, or it maps to, to biomarkers, uh, similarly but slightly differently. This is, this is my understanding.

The gap between what a wearable algorithm infers and what actually is being, uh, clinically validated. Um, that distinction between an algorithm guessing a state versus, uh, [00:19:00] measuring a signal is where so many trials sponsors burn capital jumping that gap. Uh, so for a sponsor sitting on a pile of raw data, how do you tell them to separate algorithmically, um, what is inferred from what they need to def-- what they can defend clinically, uh, before submitting their, their outcomes to a regulator?

How do you find the ground truth? And this is, this is the tough one for, you know, startups trying to break in with wearables because really you want to be able to-- like any clinician will come in, and I-- you know, my wife even does it. When I bring her a new piece of, uh, evidence, you know, or a new device, she goes, "Where's the evidence?"

And there's companies out there like Compatica that have been used in hundreds and hundreds of trials and are heavy, heavily validated, so you know that their, uh, algorithms are pretty solid. They're based in ground truth with 

[00:20:00] actually peer-reviewed journals and used in a lot of studies. And it's tough for a new company coming out that's had a-- that has a device and is trying to get that first pilot.

So, and it's-- this is not-- they're not newer companies, but if you're doing a real world trial, you want something that has some ground truth behind it in peer-reviewed journals. And The catch with wearables is a lot of the data's inferred. Uh, the only actual real data is the original PPG signal that's coming from the actual red, green infrared sensors.

Beyond that, everybody's using different peak detection algorithms. They're looking at morphology. They're looking at dif- distances from the point of highest peak acceleration to highest peak acceleration, different algorithms to kick out, uh, bad waveforms. And once you get to that, you're already inferred.

And you'll see devices that come in, and there's some on the market right now that only supply the PPG and leave it up to you. That's a notoriously difficult problem, uh, as I found out in the 

[00:21:00] past, and getting it FDA approved. Or you come all along to the other side with groups like Compatica that have done tons of studies and in, in fact, I think did the first use of the predetermined change control plan with the FDA, showing how algorithms can change over time.

And then there's levels in between. So making sure that you pick your device right, look at what data you're actually trying to pull from it. Is it heart rate variability, oxygen saturation? Uh, sleep is starting to be pretty well developed, uh, you know, at least on the primary, when you fall asleep, when you wake up, sleep efficiency, REM.

And different devices have different specialties. Aura, Oura Ring is incredibly good at sleep. You know, uh, different devices are in- incredibly good at oxygen saturation. Sometimes you might actually want two devices, depending, and be able to compare them. The ring versus the watch are slightly different.

Uh, you get transmissive reflecting versus, uh, well, reflective and transmissive. The ring works pretty well because the light actually goes through as well as 

[00:22:00] reflects, and you get different pieces of data from different depths. So it's a complex question, but you want to look at devices that have validation and devices that have-- that are focused on the specific biometrics that you're looking at.

And I think that's probably the easiest way to look at it. And the more processing you go through, unfortunately, oxygen saturation is derived, it's a ratio of ratios. You know, and you go all the way through, and all of them are calculations And ultimately, you want to look to see how close their outcome is.

Is it one beat per minute? Is it one respiratory rate per minute? Well, excuse me, one breath per minute. Are you plus or minus 1% with the oxygen saturation? Uh, of course, that's a little bit of an exaggeration. And then blood pressure is, you know, that one people, a lot of people are working on, you know, and there's a lot of questions there.

What is it exactly that you're looking for? Are you moving? Are you not moving? And there's different groups working on a lot of these different problems. So I know the answer was a

[00:23:00] little bit long-winded, but it goes to what you s- what you hear from physicians a lot. Where's the data, and what is it specifically we're looking at, and is it FDA-approved?

Is it on track for an FDA approval? Uh, is it on a 510? You know, is it something that's being built on top of something that's already been validated as well? So you have different levels there, and the ones that are further along, obviously, to use in a study are much more expensive. So there's also, uh, you know, again, cost.

What's the best device we can get to look at the biomarker that we need based on what stage we're at? And, and I think the validation problem grows even more challenging and complex, uh, when you get-- when you're trying to run multiple site trial across heterogeneous IT environments. Uh, what does it actually take to build a secure audit-ready remote data capturing

[00:24:00] system to hold up scrutiny across, across sites?

Uh, specifically when dealing with, mm, integration of, of different standards, EHR, SMART, or F-FHIR, uh, to, to name a few protocols, um, or even enterprise hospital systems. How, how do you navigate such variety in, in the IT environment to arrive to that validation that is, is key to demonstrate or, or key to, um, you know, transcend the, the regulatory, uh, landmarks?

Actually, I just worked on this problem, uh, with Cena, uh, while I was with them, and they're doing a couple of smaller trials, and that's the first thing that comes up is the size of the trial. Really, you wanna offload

[00:25:00] as much of the compliance as you can to one of the softwares out there like, you know, Greenlight Guru's come up, uh, Castor is one of them, and they allow you to actually set up sites globally, and they actually automatically pull in the data, and they can work with you to be able to pull data from a variety of different devices.

And that way the HIPAA compliance is built into their systems. They have the disaster recoveries, the cybersecurity is on, is on them. To build everything from scratch is notoriously difficult, but with Cena we still ran into the issue you have to have the patient list of the de-identifiers. So how do you store that correctly?

And where we-- with Cena, we started to run into some interesting questions because we had a multi-tenant platform, and down the road there was potential SEC, and then we also had some HIPAA compliance with the research side, and then layering AI on top of it. And how do you actually build 

[00:26:00] that? And you have to make sure that everything is specifically isolated.

And a lot of this is well-defined, you know, using the intro roles and the different, uh, the different things built into Microsoft already. But being able to actually map it to the risk assessments and the, the threat assessments based on each HIPAA rule, and actually being able to point at where you solve that in your actual Microsoft tenant or in your Azure blobs, how are you actually offloading what to Castor?

And actually the AI is able to pull the ground truth. It can actually pull your Microsoft graph and information and go ahead and start mapping these different things to build out your threat assessments. And it differs based on how many people are actually working in your organization and the size, the size of the trial.

You know, big pharma has it good. They can just go and buy this gigantic platform that's two hundred thousand a year and it's not a problem, and ha- and offload everything. Smaller groups generally can't do that, but there's comp-- there's companies out there like Castor and other that have

 [00:27:00] smaller, uh, offerings that allow you to really leverage their expertise and offload a majority of it and focus on building your platform focused on exactly what you need.

What I did find out is now, you know, with the amount of cybersecurity issues going on, whether it's Striker getting hacked or smaller groups getting hacked, it's almost easier to come in and set everything to the maximum, you know, when you're looking at, say, HIPAA or others, and it doesn't cost that much more to go ahead and do things across the board.

It used to be an art to be able to do things at a minimum level, and you'd be able to get away with it because of cost, and there were not that many cybersecurity incidents. But anyone who codes right now sees that there's different package CVEs coming up every single day, and you have to stay on top of it.

It used to be a couple a month. Now I get two per day. So you start to see these cybersecurity incidents happening more and more and more, so it's a- almost easier to come in at a top-level security posture and

 [00:28:00] come in from that angle. And of course, there's always issues that come up when you're fully locked down.

But on, you know, what we're starting to see with AI, with Fable and the, you know, the AI going rogue, we're starting to see some things, uh, some pretty interesting things. So I always try to take a security-first posture when I'm building stuff, and I always work with people that are experts in that area to be able to guide me on what I'm looking at and what I'm trying to do, and what the most important information is to provide.

And this, this conversation gets even more layered when you introduce the, the new trend of AI autonomous agents. Um, and as a major focus o- of your work being data engineering, making massive, complex healthcare data sets structured and accessible to, to agents, it's, I would imagine, a, a key component of, of the new generation or the next generation of challenges you need 

[00:29:00] to navigate when, when establishing how that infrastructure, how that pipeline, uh, connects and how it's architected.

Uh, so how do we architect clinical data pipelines so AI agents can deliver personalized insights across different geographies, health systems, race, uh, socioeconomic factors, without introducing bias or losing data integrity? That, that is an excellent question. I mean, and I think the answer to that's gonna almost be generational, or at least over the next 10 years.

People have been working on that, and I think blockchain technology is gonna come to the forefront on that with data privacy. Right now- Sorry about that. Uh, right now, one of the things, uh, let's see. We, uh, you know, there's been the Healthcare Interoperability Act. Within one year, everybody was supposed to be interoperable.

That was 10 years ago. Never happened.

 [00:30:00] There's companies working on that. I work with a, uh, strategic advisor for a large AI company up in New York. They've integrated roughly 150 different systems. There's 17,000 EHRs worldwide. Some are very small, some are larger. So how do you start to tackle that problem?

And you need to really have a data standardization. And I know there's groups out there, uh, there's one company I've wor- I work with, Wish Kanish, and then there's, uh, Heather Flannery that works with AI MindSystems, and they're trying to solve these problems with smart contracts, AI being able to translate the data, well, algorithmically and AI, to be able to make sure that the data's correct, everything's standard, to be able to put it on the blockchain, and then return ownership to actual pe- to the actual patient.

So any system in the world would be able to access it, and there would be permissions built in. That way, eventually when you have these AI systems that are actually crawling, let's call it the Web3, they'll be able to look at some things and not look at others, and they'll be able-- When they're autonomous, it'll be more controlled.

You know, whether or 

[00:31:00] not people agree with me on that, I think that that's probably where it's headed. The most robust technologies out there are blockchain, but there's still gonna be a huge, well, hump of adoption to get over. People have been working on it for 10 years, but I think the technology, processing speed, connectivity is getting to the point where we'll start to see some things move there.

Now, now for a startup, you know, you can leverage companies like Redox that have focused on building these interoperability standards and help. It's sort of a fast-forward button if you have the money. You can focus on one or two EHRs that you need to integrate with immediately and build those out, or work with people that have built them.

'Cause it doesn't become cost-effective building them all from scratch across the board. It gets harder and harder. So leveraging companies, uh, technology companies that have built them and using the FHIR, SMART on FHIR, HL7. We're starting to see AI come in because it's able to handle some of the, let's call them translation problems between one [00:32:00] system to another, that it's been hard to do algorithmically.

Whether it's a three-line address, two-line address, the, you know, the birthdays being switched. How do you identify that? Duplicates. And a lot of the notoriously difficult problems are something that large language models are able to handle a little bit more natively

In working with the advisory body like AHA, where do you see the boundary between accelerating data accessibility and AI-- accessibility for AI and, uh, maintaining strict clinical validation standards? How, how do you, how do you see those traditional strict, uh Um, evidence-driven bodies, uh, adopting AI and, 

[00:33:00] and its, um, advantages and disadvantages.

Sure. You know, and, you know, American Heart is specifically working on some of those problems 'cause the ultimate goal is to have personalized medicine. And everyone knows that's for years, here's your blood pressure, it's high or low. And that's a bit naive. Some people tend to run a little bit high and it's fine.

Some people tend to run a little bit low and it's fine. And, you know, American Heart has care plans or care paths that they supply to physicians to be able to supply to patients. And people understand that the-- it might vary differently, uh, between patient to patient or based on geographic area or even they're finding out even within the hospital systems are slightly different is what...

And so you can look at the geographic, socioeconomic, uh, you know, race, age, sex, all of that fun stuff, and start to be able to look at how these things affect that care plan. And then ultimately you really need to look whether it's on a

 [00:34:00] genetic level when we get to the point where we can process that with speed and what's going on with a specific person.

And what we're finding is a lot of the data right now is for the old internet and older algorithms. So a lot of the work now is translating these huge compendiums of data into something that's accessible by AI, whether it's through MCP or other different connectors to be able to pull that data and then look at the biometrics that are out there from a patient and being able to fine-tune and move that forward.

You know, and you see a lot of issues with bias, uh, and there's a lot of work being done. You know, I g- um, one of the white papers I'm working on now is being able to try to quantify bias when you look at the solutions on these large language models or with any of the, I guess let's call them AI models, even though they're deep learning computer vision.

And a model might be specifically biased for a specific group. That's fine, just be aware that it's specifically

[00:35:00] biased for that specific group when you use it there. There was a model that was trained across Mayo, uh, Mayo Clinic information, and it worked fantastically well in the Mayo Clinic, but when you applied it to another hospital system, it didn't work as well.

So you're starting to find out what is the bias and just making sure that you pick the right, and which is an additional bias. You know, what are you looking at and testing it for to make sure it's biased? Because no matter where you look, there's always another level of bias, whether it's the architecture of the actual language model, the initial starting point, the weights, and then what data you put in.

And then-- And it starts to keep going down the road, and you start to get, getting down to this, you know, infinite loop. But at some point, you actually get to a level where you can quantify it. It's just extraordinarily computationally heavy. And then trying to decide what are the most important factors you're looking at to figure out where that bias points.

Is it race? Is, is it age? Is it genetic? Is it hospital systems? Is it the fact that you're in the Mississippi

 [00:36:00] Valley, and it might be histoplasmosis you're looking at from the pigeons? You know, one of those board questions from, uh, med school. So what is the most important thing that's creating a lean in that model?

And then how do you leverage that lean? You end up with a set of models that are good for spec- very specific groups, which helps to personalize and eventually, potentially training individual models on a person. Who knows, medicine down the road might be language models, small language models trained on your individual data, and that update, oh, each day.

And take a look at your data and update to make sure you're staying on track and have an underlying model that looks at it monthly, yearly, and you can actually start to look at these trends, which actually start to become important. So I think it's pretty interesting, and it's opening up and developing, and there's a lot of different groups trying to solve the bias problem, you know, as well as these interoperability problems, how to translate all the data we have into something that's usable to AI, making sure that the data is actually good, you know, and, and is actually useful as well.

[00:37:00] 

And trying to find out, you know, from a research side, when you start to see, is this something that has been researched before? Is the data actually out there or a paper written? What area is actually the most beneficial for us to start to move forward in? And building engines to actually start to look at that sort of stuff to be able to move things forward at a rapid pace as well.

And I imagine this is the type of thing you, you deal with with your clients at the Novo Innovation group. Wh- when you're trying to prove a clinical claim rather than just showcase a feature, the underlying architecture has to, has to change. So when a digital health or med tech company comes to the Novo, um, needing an AI architecture that can eventually stand up to a regulatory evidence, uh, scrutiny from a regulatory body, where do you tell th- them to start?

Wh- wh- where's the s- the starting point to 

[00:38:00] building the infrastructure needed for, for evidence to speak for itself? Depends how big the company is, uh, what their long-term goals are. You know, uh, and just a good example is when you start to look at, let's take Claude, for instance. You can buy an enterprise that, uh, membership that's supposedly HIPAA compliant, BAA compliant, but you need a minimum of 20 seats.

It's not cost-effective for a small company. So the next question is, do I build it myself? Do I put that on AWS Bedrock and start to build that out? And then the next question from there is all of a sudden, well, then I need to build out my own memory. I want that to be secure. And then you have to build out the harness to make sure that you're going in different directions.

And then ultimately, one of the things I'm working on now is to take some of the decision or all away from the large language model. It's handle, it handles the inference. But when it comes down to which tool I'm going to choose and which piece of data can I 

[00:39:00] access, there's a- an actual mechanistic router that makes that decision that the AI can't go around.

Because we've als- we've all played with Claude, and all of a sudden it's making a decision. Why did you do that? Oh, you're right. I didn't mean to do that. I'm not sure why. You know, so trying to actually control that a little bit better. And I've created a set of tools that I use when I start to work on a new project.

I do it, I work on a documentation first, and I describe the project from an architectural standpoint and the general idea behind the project. And then I have my tools actually pull in from a set of tools that I built that are focused on that type of project. And they come in before writing any lines of code or working on, uh, you know, infrastructure projects like Microsoft or AWS, that I try to give any AI a complete understanding end-to-end, and the tools to be able to pull the right data off the bat before I even start writing a configuration or a line of code.

And then it's focused on that 

[00:40:00] particular company, that particular problem, and then you iterate from there. And make sure always every time you build something, it checks to make sure it's been achieved, to make sure it's valid, and to make sure it meets the set of rules. I have, uh, you know, in my Claude, I have my rule zero is don't make assumptions, always base everything in ground truth, and double-check everything and validate everything.

And that's the first three lines of a- any... And even then I still have to repeat it. Don't make assumptions. Oh, you're right, I'm making assumptions. You know, and it has to go back. But I focus very hard on keeping it a tight rein on what data's available to any of the coding partners. And I also have validation.

I have Codex as well as Gemini that are used as reviewers, and they actually work as a three-party, let's call it a discussion, where they work with each other and then I get to see what their thoughts are, and then I actually get to make a final choice, getting different inputs from different AIs that have different blind spots.

So I cover as many as I can, 

[00:41:00] and based on, you know, MCP polls from, say, Railway or Redis or AWS, Microsoft, and it pulls live data so I understand where everything is and make decisions based on that. Hmm.

And okay, wow, that's, that's a huge Topic and, and- Big topic. It, it, it really requires a lot of, uh, nuance and, and a global understanding of the process to, uh, connect, wire those, all those seemingly, um, yeah, all those tools that are very good at, at looking like they're telling you what, what is true, but not necessarily so.

Um, and I wanna engage that, that same perspective, but now speaking of how you design a,

 [00:42:00] a clinical trial to accommodate for those nuances. How do you work on making a large data compendium accessible for AI, and how do you-- how does that shape the way you, um, you use traditional l-for to, for this example, let's, let's talk about traditional cardiovascular study data.

Um, are you able to, to use data that was collected years past, um, by... that was collected in a completely different IT environment, be relevant to, to what AI needs or, or expects in order to derive that, um, clinical, uh, value? And move -- in looking forward, how do you design your, your clinical trial protocols to accommodate 

[00:43:00] for AI to, to maximize the outcome derived from this technology?

Hmm. As a, you know, that, it's a very interesting problem. It's something I came up, I came across when I was working with Sena, and it's something I call, let's call it the epistemic clean room. A fancy word for a researcher has access to research data, but also has access to capital and commercial data. A researcher can say, "I have a disclosure.

I'm not going to take this into account." But when I layer AI on top of it, it doesn't have a disclosure built in. So you wanna make sure that it doesn't make a decision on research based on the fact that you just raised ten million in capital. So being able to separate those. But it's also interesting if you can give it access to just research and then access to both, being able to say, "Were my past decisions biased by that capital information now that you have access to it?"

And you see this on a temporal basis as

 [00:44:00] well. Decisions that were made early on in a trial aren't necessarily applicable to current, uh, to where you are currently in a trial. But this, I think, goes more to a data provenance question, which really comes down to the FDA. It depends how that data was collected.

From an AI perspective, could it be beneficial internally to researchers to make decisions? Probably. Would it be valid to be able to present to the FDA to be used in a trial? That depends. It depends how it was connected. You know, one of the things that we're seeing, uh, you know, Julio works in Latin America, and depending on which country you're at, do they have art-- data collection requirements that meet the requirements of the FDA?

Most groups go in there and talk to the FDA and say, "Hey, we're collecting the data anyway. Let's make sure we connect it-- collect it to the level that we need to save the trouble of having to redo it when we get to the US." And, you know, so you can actually use some of that data and make it reusable. So the question is how it was collected.

[00:45:00] And for startups, I always say talk to, talk to the FDA early. Build that relationship. They have their doors open, the conversations are free, the pre-submissions are free. And to be able to leverage their knowledge and say, "This is how we collected the data. Does this work?" And whether, you know, they'll tell you yes or no.

And so sometimes it depends what you want to use the data for, and internally being able to guide which way you might go in the future is pr- is something that's generally valid for that. But whether or not it's actually usable for trial data is up in the air, and that's an open-ended question depending on how it was collected.

Hmm. But let's say you're advising a founder who's, who has a genuinely promising wearable device. Mm-hmm. Um, but he doesn't yet have a clear platform or plan for validating it against a clinical fact. What are the two or three things they could do 

[00:46:00] or, or need to, to fix from their data collection pipeline before they take that data anywhere near a sponsor or FDA?

Rule number one, gather everything. You can't go back and gather data you missed. So- Mm ... when you have that opportunity, make sure you're collecting every piece of information you can from that wearable. Uh, what might happen is at the time you're collecting it, it might have not been validated. That particular, let's take O2 saturation, might be go- undergoing the FDA, so you don't know if it's usable.

But down the road, they get the 510K, they get the approval or authorization, and then it clears. But if you didn't collect it, all of a sudden you missed out on that whole piece of data. And it doesn't mean that you can't use that to be able to influence your choices, and it's becoming more commonplace in that it's a, you know, it's a gray area, but there's a lot of wearables out there.

Say Fitbit, they-- Last I checked, they had a website up and they had over 

[00:47:00] 700 studies based on Fitbit. But at the time, it wasn't clinic-- it wasn't FDA approved. So the question, is there clinically validated versus FDA approved? They're different things, and a lot of the wearables that are out there and the datas that's being, the data that's being used and gathered, it has been clinically validated.

There's been a ton of papers to show that it's pretty accurate. And is that use- usable? That ends up being, again, an FDA question. Are they going to let that slide through? Has that device been used before? Are you building a completely novel device? In which case you want to have a device that's validated while you're wearing your device to be able to validate it.

Make sure that you're getting some sort of ground truth data. If you're looking at heart rate variability and peak detection on a PPG, make sure you're getting an EKG if you can, because the peaks are sharp. It's really good to be able to pull those peak detections. Using a clinically validated oxygen saturation, uh, you know, 

[00:48:00] blood pressure cuff, uh, and you know, whether or not you actually need to do, you know, arterial pressure to be able to validate blood pressure, uh, you know, for a patch.

There's different levels. But find some way to get ground truth from a well-known device that's out there or a heavily validated device that you can base yours off of. Otherwise, your data- A gold standard ... is just sitting there with nothing to attach it to. Hmm. And gather all of it. Have the gold standard on hand to, to compare to and, and point towards, uh, you know, closeness in, in results.

Yes. And looking ahead, where do you see the ground truth problem in digital biomarkers actually getting solved? Is it gonna be better sensors, better AI validation standards, or maybe even a regulatory shift? Yeah, that's a good question. Part of it is, I think, regulatory shift, which we're seeing with the predetermined change control plan that allows you to say, "Hey, here's gonna be our next 10

 [00:49:00] changes with AI."

I think there's going to be improvement on the small language models that will be focused because, uh, ultimately when you look at a biometric, it looks like the stock market. It looks-- It's a piece of data. It's a language in and of itself, so you can train it, and if you look at the body, you have all these different lines.

It's almost as if there's a symphony, and can you get something that's smart enough to be able-- Smart enough. Robust enough, being able to process fast enough to make all those connections. You're starting to see it with digital twins, where you build the structure of the heart and then you're layering the electrophysiology, and you're layering on top of that the s- uh, the cellular effects of the biochemistry.

For instance, if there's a calcium channel blocker, it's going to change the electrophysiology, which will change the structure, and building these layers upon layers. So part of it's gathering data and finding ways to attach it, and then being able to s- look at

[00:50:00] connections in between, in the case of biometrics, in between the different biometrics.

Because ultimately AI, its pattern recognition is very, very good at, and making sure you're finding the right patterns, being able to discard that those that aren't. And what additional piece are we missing? Is it genetics? Then you start to get to the epigenetics, which gets more difficult. So I think down the road it's going to be da-data gathering on a personal level and being able to get enough data.

Actually, it's not even more data, it's the right data, and trying to figure out what it is. You might have, you know, p-petabytes of data and it turns out there's only two biometrics and maybe one genetic that turns out to be connected. So learning what the right data is out of that gigantic pile of data is probably the most difficult part Jenaro, the through line today is one I believe every trial sponsor, med tech founder needs to hear.

[00:51:00] A wearable signal isn't clinically evidenced until someone has done the hard work of proving it. Skipping the ground truth validation and data engineering groundwork does not just invite regulatory rejection, it burns precious, uh, trial capital, data, and time. Thank you for walking us through what works in real life and actually brings about innovation and moving the boundary of technology further.

So no, I wanna thank you, Jesus, for your time as well and inviting me along, and by extension, Julio as well. It's been, it's been a lot of fun having an opportunity to discuss a lot of the changes we're seeing in AI and the difficulties applying them in studies and biometrics and making sure they're really grounded in reality.

Again, making sure that we're not hallucinating and making things up. Thank you for tuning into "Global Trials Accelerators." If this episode challenged how [00:52:00] you view your trial strategy, share it with a colleague or a clinical lead who needs to hear it. Subscribe where you listen to the podcast, and we'll see you on our next episode.

Until next time, keep accelerating.