Ensuring Responsible and Compliant Use of AI in Healthcare: Addressing Explainability, Data Security, and Risk Management in Patient-Facing Applications

SACRAMENTO, Calif. — A judge has initiated a federal court takeover of California’s troubled prison mental health system by naming the former head of the Federal Bureau of Prisons to serve as receiver, giving her four months to craft a plan to provide adequate care for tens of thousands of prisoners with serious mental illness. Senior U.S. District Judge Kimberly Mueller issued her order March 19, identifying Colette Peters as the nominated receiver. Peters, who was Oregon’s first female corrections director and known as a reformer, ran the scandal-plagued federal prison system for 30 months until President Donald Trump took office in January. During her tenure, she closed a women’s prison in Dublin, east of Oakland, that had become known as the “rape club.” Michael Bien, who represents prisoners with mental illness in the long-running prison lawsuit, said Peters is a good choice. Bien said Peters’ time in Oregon and Washington, D.C., showed that she “kind of buys into the fact that there are things we can do better in the American system.” “We took strong objection to many things that happened under her tenure at the BOP, but I do think that this is a different job and she’s capable of doing it,” said Bien, whose firm also represents women who were housed at the shuttered federal women’s prison. California corrections officials called Peters “highly qualified” in a statement, while Gov. Gavin Newsom’s office did not immediately comment. Mueller gave the parties until March 28 to show cause why Peters should not be appointed. Peters is not talking to the media at this time, Bien said. The judge said Peters is to be paid $400,000 a year, prorated for the four-month period. About 34,000 people incarcerated in California prisons have been diagnosed with serious mental illnesses, representing more than a third of California’s prison population, who face harm because of the state’s noncompliance, Mueller said. Appointing a receiver is a rare step taken when federal judges feel they have exhausted other options. A receiver took control of Alabama’s correctional system in 1976, and they have otherwise been used to govern prisons and jails only about a dozen times, mostly to combat poor conditions caused by overcrowding. Attorneys representing inmates in Arizona have asked a judge to take over prison health care there. Mueller’s appointment of a receiver comes nearly 20 years after a different federal judge seized control of California’s prison medical system and installed a receiver, currently J. Clark Kelso, with broad powers to hire, fire, and spend the state’s money. California officials initially said in August that they would not oppose a receivership for the mental health program provided that the receiver was also Kelso, saying then that federal control “has successfully transformed medical care” in California prisons. But Kelso withdrew from consideration in September, as did two subsequent candidates. Kelso said he could not act “zealously and with fidelity as receiver in both cases.” Both cases have been running for so long that they are now overseen by a second generation of judges. The original federal judges, in a legal battle that reached the U.S. Supreme Court, more than a decade ago forced California to significantly reduce prison crowding in a bid to improve medical and mental health care for incarcerated people. State officials in court filings defended their improvements over the decades. Prisoners’ attorneys countered that treatment remains poor, as evidenced in part by the system’s record-high suicide rate, topping 31 suicides per 100,000 prisoners, nearly double that in federal prisons. “More than a quarter of the 30 class-members who died by suicide in 2023 received inadequate care because of understaffing,” prisoners’ attorneys wrote in January, citing the prison system’s own analysis. One prisoner did not receive mental health appointments for seven months “before he hanged himself with a bedsheet.” They argued that the November passage of a ballot measure increasing criminal penalties for some drug and theft crimes is likely to increase the prison population and worsen staffing shortages. California officials argued in January that Mueller isn’t legally justified in appointing a receiver because “progress has been slow at times but it has not stalled.” Mueller has countered that she had no choice but to appoint an outside professional to run the prisons’ mental health program, given officials’ intransigence even after she held top officials in contempt of court and levied fines topping $110 million in June. Those extreme actions, she said, only triggered more delays. The 9th U.S. Circuit Court of Appeals on March 19 upheld Mueller’s contempt ruling but said she didn’t sufficiently justify calculating the fines by doubling the state’s monthly salary savings from understaffing prisons. It upheld the fines to the extent that they reflect the state’s actual salary savings but sent the case back to Mueller to justify any higher penalty. Mueller had been set to begin additional civil contempt proceedings against state officials for their failure to meet two other court requirements: adequately staffing the prison system’s psychiatric inpatient program and improving suicide prevention measures. Those could bring additional fines topping tens of millions of dollars. But she said her initial contempt order has not had the intended effect of compelling compliance. Mueller wrote as far back as July that additional contempt rulings would also be likely to be ineffective as state officials continued to appeal and seek delays, leading “to even more unending litigation, litigation, litigation.” She went on to foreshadow her latest order naming a receiver in a preliminary order: “There is one step the court has taken great pains to avoid. But at this point,” Mueller wrote, “the court concludes the only way to achieve full compliance in this action is for the court to appoint its own receiver.” This article was produced by KFF Health News, which publishes California Healthline, an editorially independent service of the California Health Care Foundation.  If you or someone you know may be experiencing a mental health crisis, contact the 988 Suicide & Crisis Lifeline by dialing or

Imagine you’re building your dream home. Just about everything is ready. All that’s left to do is pick out a front door. Since the neighborhood has a low crime rate, you decide you want a door with a standard lock — nothing too fancy, but probably enough to deter 99.9% of would-be burglars. Unfortunately, the local homeowners’ association (HOA) has a rule stating that all front doors in the neighborhood must be bank vault doors. Their reasoning? Bank vault doors are the only doors that have been mathematically proven to be absolutely secure. As far as they’re concerned, any front door below that standard may as well not be there at all. You’re left with three options, none of which seems particularly appealing: Concede defeat and have a bank vault door installed. Not only is this expensive and cumbersome, but you’ll be left with a front door that bogs you down every single time you want to open or close it. At least burglars won’t be a problem! Leave your house doorless. The HOA rule imposes requirements on any front door in the neighborhood, but it doesn’t technically forbid you from not installing a door at all. That would save you a lot of time and money. The downside, of course, is that it would allow anyone to come and go as they please. On top of that, the HOA could always close the loophole, taking you back to square one. Opt out entirely. Faced with such a stark dilemma (all-in on either security or practicality), you choose not to play the game at all, selling your nearly-complete house and looking for someplace else to live. This scenario is obviously completely unrealistic. In real life, everybody strives to strike an appropriate balance between security and practicality. This balance is informed by everyone’s own circumstances and risk analysis, but it universally lands somewhere between the two extremes of bank vault door and no door at all. But what if instead of your dream home, you imagined a medical AI model that has the power to help doctors improve patient outcomes? Highly-sensitive training data points from patients are your valuables. The privacy protection measures you take are the front door you choose to install. Healthcare providers and the scientific community are the HOA.  Suddenly, the scenario is much closer to reality. In this article, we’ll explore why that is. After understanding the problem, we’ll consider a simple but empirically effective solution proposed in the paper Reconciling privacy and accuracy in AI for medical imaging [1]. The authors propose a balanced alternative to the three bad choices laid out above, much like the real-life approach of a typical front door. The State of Patient Privacy in Medical AI Over the past few years, artificial intelligence has become an ever more ubiquitous part of our day-to-day lives, proving its utility across a wide range of domains. The rising use of AI models has, however, raised questions and concerns about protecting the privacy of the data used to train them. You may remember the well-known case of ChatGPT, just months after its initial release, exposing proprietary code from Samsung [2]. Some of the privacy risks associated with AI models are obvious. For example, if the training data used for a model isn’t stored securely enough, bad actors could find ways to access it directly. Others are more insidious, such as the risk of reconstruction. As the name implies, in a reconstruction attack, a bad actor attempts to reconstruct a model’s training data without needing to gain direct access to the dataset. Medical records are one of the most sensitive kinds of personal information there are. Although specific regulation varies by jurisdiction, patient data is generally subject to stringent safeguards, with hefty fines for inadequate protection. Beyond the letter of the law, unintentionally exposing such data could irreparably damage our ability to use specialized AI to empower medical professionals.  As Ziller, Mueller, Stieger, et al. point out [1], fully taking advantage of medical AI requires rich datasets comprising information from actual patients. This information must be obtained with the full consent of the patient. Ethically acquiring medical data for research was challenging enough as it was before the unique challenges posed by AI came into play. But if proprietary code being exposed caused Samsung to ban the use of ChatGPT [2], what would happen if attackers managed to reconstruct MRI scans and identify the patients they belonged to? Even isolated instances of negligent protection against data reconstruction could end up being a monumental setback for medical AI as a whole. Tying this back into our front door metaphor, the HOA statute calling for bank vault doors starts to make a little bit more sense. When the cost of a single break-in could be so catastrophic for the entire neighborhood, it’s only natural to want to go to any lengths to prevent them.  Differential Privacy (DP) as a Theoretical Bank Vault Door Before we discuss what an appropriate balance between privacy and practicality might look like in the context of medical AI, we have to turn our attention to the inherent tradeoff between protecting an AI model’s training data and optimizing for quality of performance. This will set the stage for us to develop a basic understanding of Differential Privacy (DP), the theoretical gold standard of privacy protection. Although academic interest in training data privacy has increased significantly over the past four years, principles on which much of the conversation is based were pointed out by researchers well before the recent LLM boom, and even before OpenAI was founded in 2015. Though it doesn’t deal with reconstruction per se, the 2013 paper Hacking smart machines with smarter ones [3] demonstrates a generalizable attack methodology capable of accurately inferring statistical properties of machine learning classifiers, noting: “Although ML algorithms are known and publicly released, training sets may not be reasonably ascertainable and, indeed, may be guarded as trade secrets. While much research has been performed about the privacy of

SQA Regulatory Surveillance Summary for March and April 2023 By: Laurel Hacche, Rocio Cabeza, and Debra Cortner Agência Nacional de Vigilância Sanitária (ANVISA) ANVSA Recommends Intensification of Actions Against Arboviruses, 22 March 2023 ANVISA has published Technical Note 12/2023 which recommends the intensification of measures or the reduction of mosquito breeding sites and control of adult vectors, airports, and border areas, which are points of entry and exit of the country. The objective is to reinforce actions for the prevention and control of diseases such as dengue and chikungunya, arboviruses transmitted by the Aedes aegypti mosquito.  As recommended by the International Health Regulations (IHR), there must be a vector-free zone at seaports, airports, and land crossing and within a perimeter of 400 meters around these entry points. For this, it is necessary to maintain regular active surveillance and vector control so that the risk of disease transmission is reduced. ANVISA Resumes Inspections in the Area of Technovigilance, 29 March 2023 In partnership with local Health Surveillance, ANVISA technical teams carried out in March 2023 two (2) medical device manufacturer inspections with an aim to strengthen the monitoring of these products in Brazil, ensuring their safety, performance, and quality. The technovigilance inspection consists of: Evaluation of compliance with technovigilance standards by companies Monitor the behavior of products in the post-marketing phase Evaluate the effectiveness of filed actions that affect medical devices Collect data and information of adverse events and technical complaints, as well as evaluate the investigative process triggered by the registration holder in the face of such situations The selection of companies for inspection includes the following criteria: silent companies (absence of notifications of adverse events, technical complaints, or field actions), quantity of products regulated by ANVISA and the risk class of these products. ANVISA Warns of New Cases of Botulinum Toxin Counterfeiting, 05 April 2023 ANVISA alerts health professionals and the population to the fact that new cases of adulteration/falsification of the drug Botox® 100U (botulinum toxin A), Batch Number C6835C3 have been identified. ANVISA has intercepted international shipments of Batch Number C6835C3 which presented a false description of content and contained bottles in packaging in the Turkish language with expirations dates 10/2024 (bottle) and 12/2024 (secondary packaging). The company that holds the registration of the drug Botox, Allergan Produtos Farmacêuticos Ltda., confirmed to ANVISA that the original Batch Number C6835C3 has a shelf life of 12/2023 and was marketed only in Turkey, and has not been imported into Brazil by official means. Thus the seizer and prohibition of marketing, distribution, and use of Batch Number C6835C3 was determined by means of Resolution-RE 796 of March 9, 2023. China: National Medical Products Administration (NMPA) China Update: Announcement No. 124 of 2022; Draft Guidelines for the Registration of Implantable Medical Devices, 06 January 2023 The NMPA and Center for Medical Device Evaluation (CMDE) published new rules for the supervision and management of enterprises implementing medical device safety, as well as draft guidelines for the registration and review of implantable medical device batteries. Following the draft consultation in November 2022, the NMPA announced the issuance of the Regulations on the Supervision and Management of Enterprises Implementing the Main Responsibility for the Quality and Safety of Medical Devices, which will come into force on 01 March 2023. This requires medical device registrants and record filers to fulfill the main responsibility for medical device quality and safety. It also includes provisions that emphasize the implementation of responsibilities for medical device registrants, filers, and entrusted manufacturers and personnel in key positions of quality and safety in medical device production and operating enterprises.  The Interpretation of Regulations on the Supervision and Management of Enterprises Implementing the Main Responsibility for the Quality and Safety of Medical Devices further discusses the provisions of the Regulations. 2022 Recap & 2023 Outlook: China Pharmaceutical Regulatory Updates, 28 February 2023 In 2022, China issued and implemented a series of drug regulations, including new rules for marketing authorization holders to manage drug quality and guidelines for pharmacovigilance inspections. Last year also saw China’s first regulation permitting online sales of prescription drugs nationwide. The regulations impact how pharmaceutical companies apply for marketing authorization, sell drug products, ensure drug safety, and fulfill other obligations. ChemLinked BaiPharm Team held this webinar to review the significant regulatory updates, which can help stakeholders keep compliant with current regulations and prepare for the upcoming ones. Key points for the webinar include: China’s Drug Regulatory System 2022 Regulatory Updates 2023 Outlook European Commission (EC) New European MDCG Guidance on Medical Device Vigilance Requirements, 17 February 2023 The long-awaited Medical Device Coordination Group (MDCG) Guidance on Vigilance has been published to the European Commission website. MDCG 2023-3, Questions and Answers on Vigilance Terms and Concepts as Outlined in the Regulation (EU) 2017/745 on Medical Devices, provides additional information on key vigilance terms. Interestingly, the guidance only covers devices under the scope of the Medical Devices Regulation (MDR) and not those under the scope of the In Vitro Diagnostic Medical Devices Regulation (IVDR). The guidance clarifies key differences in vigilance reporting between MEDDEV 2.12/1 rev.8 and the Regulations: The guidance features a flowchart that illustrates the process of analyzing complaints to determine reportability under the MDR. The difference between the term “incident” and the new term “serious incident” introduced in the Regulation are also discussed in detail. The deadline for reporting serious incidents not constituting serious public health threats, deaths, or unanticipated serious deteriorations in state of health was reduced to 15 calendar days. The reporting period begins on the day after the awareness date of a serious incident. The awareness date may change if, during the investigation of an incident, the manufacturer receives additional information which subsequently changes the determination of the report to a serious incident. Although the original awareness date in Section 1.2c of the Manufacturer Incident Report (MIR) should still be identified, an explanatory comment may be placed within Section 5 of the MIR. It can be challenging to understand when the MIR report type, Final (non-reportable), may be used.

Exercise-induced bronchoconstriction (EIB), previously EIA, results in airway constriction during or following exercise, and wheezing, coughing, and breathlessness. Cold, dry air, allergens, pollution, and vigorous exercise are typical precipitants; swimming is frequently less of a problem because warm, humid air is breathed. Warming up, the use of a rescue inhaler, the selection of appropriate activities, nasal breathing, and air quality monitoring are all management options.  Ever felt like you are gasping for air after a workout, even when you are in good shape? If your breathing gets tight, wheezy, or downright difficult during or after exercise, you might have exercise-induced asthma (EIA), now called exercise-induced bronchoconstriction (EIB). But do not worry, it does not mean you have to give up your favorite activities. With the right game plan, you can keep moving without gasping for air! What Exactly is Exercise-Induced Asthma EIA happens when physical activity causes your airways to narrow, making it harder to breathe. Symptoms like wheezing, coughing, and shortness of breath usually kick in a few minutes into exercise, peak shortly after stopping, and can last up to an hour. It is more common in cold, dry air or when you are exposed to triggers like pollen, pollution, or strong smells. Unlike regular breathlessness from being out of shape, EIA comes with a tight, wheezy chest feeling that just would not go away easily. The culprit? Fast breathing through the mouth. Normally, your nose warms and humidifies the air before it reaches your lungs, but when you breathe hard through your mouth, especially in cold weather, you skip that built-in filter, and your airways react by tightening up. What Triggers Exercise-Induced Asthma EIA can be triggered by: Cold, arid air, typical of winter sports such as skiing or racing in cold climates. Allergens such as pollen, pet dander, or dust may exacerbate symptoms. Smog & pungent smells of automobile exhaust, smoke, or perfumes can irritate respiratory passages. Interestingly, swimming is usually easier on the lungs because of the warm, humid air around pools, while activities like running or cycling in cold air tend to be tougher. Controlling and Staving Off Symptoms The best part? EIA is completely controllable with a few intelligent tactics: Gradually warm up first and begin slowly with stretching or a gentle jog to help your lungs get underway. Inhaling a rescue inhaler along with medications such as Albuterol prevents symptoms if done prior to exercise. Selecting the appropriate exercises such as swimming, walking, and yoga are excellent choices less likely to trigger EIA. Breathe in through your nose that serves to warm and filter the air before it reaches your lungs. Monitor the air quality for pollution or pollen levels, if it is high, opt for indoor exercises. Staying Active with EIA Just because you have EIA does not mean you have to sit on the sidelines. Opt for sports with short bursts of activity (like baseball or golf) instead of endurance-heavy exercises. If running is your thing, try intervals with walking breaks. And if cold air is a problem, covering your mouth with a scarf or mask will trap in moisture and warmth. Most of all, see your doctor if the symptoms persist. He may suggest a change in medication or some breathing exercises to keep your lungs at peak performance. By doing it the right way, you can stay active, stay healthy, and breathe easy—no matter your exercise of preference!

Hey! Experiencing shortness of breath, wheezing, or persistent coughing? Wondering if it is just allergies or something more serious? Asthma affects millions, yet many people struggle to identify its triggers and symptoms. Understanding them can help you manage the condition effectively and improve your quality of life. Asthma is a chronic respiratory condition that various factors can trigger, yet misconceptions often prevent proper management. Recognizing common triggers and symptoms is essential for better control and treatment. Myth 1: Asthma Only Affects Children. Fact: Asthma can develop at any age. While it is common in children, many adults are diagnosed later in life. Myth 2: You Can Outgrow Asthma. Fact: While some children see an improvement in symptoms as they grow older, asthma is a lifelong condition that may return or worsen over time. Myth 3: People With Asthma Should Avoid Exercise. Fact: While exercise can sometimes trigger symptoms, regular physical activity can actually strengthen the lungs. Proper management and medications can help individuals with asthma stay active. Myth 4: Asthma Attacks Only Happen Due to Allergens. Fact: While allergens like pollen and dust can be triggers, asthma attacks can also be caused by cold air, stress, respiratory infections, and even strong odors. Myth 5: Inhalers Are Addictive. Fact: Inhalers are a safe and essential part of asthma management. They are not addictive, but they must be used correctly as prescribed by a doctor. Myth 6: If You Are Not Wheezing, You Do Not Have Asthma. Fact: Not all asthma patients experience wheezing. Symptoms can include coughing, chest tightness, and difficulty breathing, which may be mistaken for other conditions. Understanding asthma and separating myths from facts can help individuals manage symptoms effectively and reduce the risk of severe attacks. Don’t let myths cloud your understanding—know the facts, control your asthma, and breathe easier! By spreading awareness, we can support those living with asthma and encourage better respiratory health.