The Delicate Balance of Generative AI in Patient-Facing Healthcare

Canada’s healthcare system has continued to face significant strain in recent years, as growing demand places increasing pressure on an already overstretched system. Recent public opinion data indicate that the proportion of Canadians who either do not have a family doctor or struggle to access one has increased significantly over the past decade. In 2015, approximately 40% of Canadians reported difficulty accessing primary care; by 2025, that figure had risen to nearly 50% [1]. At the same time, waiting times for primary care and specialist appointments continue to stretch from weeks to months. Faced with these systemic bottlenecks, many patients are increasingly turning to alternative sources of healthcare advice, including virtual care platforms and consumer-facing digital health technologies, to bridge the gap.

Across Canada, both public and private healthcare providers are adopting digital solutions to improve accessibility and efficiency. For example, Newfoundland and Labrador Health Services offers province-wide virtual primary care appointments to residents without a family physician or nurse practitioner, after they are registered through Patient Connect NL [2]. Meanwhile, the rapid expansion of private virtual care providers around the country has raised important concerns regarding the collection, storage, and commercialization of sensitive patient information. Recent investigations have highlighted that some commercial virtual care platforms may expose patient data to unnecessary privacy risks [3].

An even more significant and potentially riskier trend is the growing reliance on generative artificial intelligence (AI) for health and wellness consultations. Around the world, individuals are increasingly using general-purpose large language model (LLM) applications to seek advice on medical conditions, mental health, nutrition, and other health-related concerns. Unlike regulated healthcare services, these AI systems operate largely outside established clinical governance frameworks. Although generative AI has demonstrated remarkable capabilities, current models remain susceptible to hallucinations, persuasive but inaccurate recommendations, and unpredictable changes in behavior following model updates. Recognizing these limitations, major AI developers, including OpenAI, have strengthened their policies to discourage the provision of personalized medical advice without appropriate professional oversight [4].

Despite these safeguards, public adoption continues to grow. The market has also witnessed a rapid increase in specialized AI-powered mental health and wellness applications, including many developed within Canada. While these technologies have the potential to improve access to support, they also raise significant concerns regarding the handling of Personal Health Information (PHI).

In response, organizations such as the Mental Health Commission of Canada have developed data and privacy standards to promote responsible digital mental health services [5]. Nevertheless, application marketplaces remain saturated with both vetted and unvetted applications. Many commercial health and wellness platforms collect highly sensitive behavioral, emotional, and physiological information that often falls outside traditional healthcare regulatory frameworks, leaving users vulnerable to privacy breaches, algorithmic profiling, secondary data use, and third-party tracking.

Canada’s evolving AI governance framework presents an opportunity to proactively address the intersection of digital health, artificial intelligence, and data protection. Attempting to restrict innovation or limit the development of AI-powered healthcare applications is neither practical nor desirable, particularly when these technologies can expand access to care for underserved populations. Instead, policymakers should prioritize robust privacy protections, transparent data governance, accountability requirements, and security-by-design principles for patient-facing AI systems. 

Equally important is improving public awareness. Users should understand what personal information an application collects, how it is used, whether it complies with Canadian privacy regulations, and whether qualified healthcare professionals oversee its recommendations.

One practical step forward would be the introduction of a national privacy-by-design certification for patient-facing AI applications operating in Canada. Like nutrition labels on food or energy efficiency ratings on appliances, AI health applications could display a standardized trust mark indicating that they meet minimum Canadian requirements for privacy, cybersecurity, transparency, and responsible AI governance. Certification should assess how Personal Health Information (PHI) is collected, stored, shared, and retained; whether user data are used to train AI models; whether data is transferred outside Canada; and whether meaningful human oversight exists for high-risk recommendations. Such a framework would not stifle innovation or create unnecessary barriers for developers. Instead, it would establish a baseline level of trust that empowers Canadians to make informed choices while encouraging responsible innovation. 

As generative AI increasingly becomes a first point of contact for health-related questions, protecting patient privacy should not be viewed as an obstacle to innovation, but as a prerequisite for trustworthy digital healthcare. Achieving this delicate balance between innovation, accessibility, and privacy will determine whether generative AI strengthens—or undermines—the future of Canada’s healthcare system.

References:

[1] Angus Reid Institute, Health Care Access: Half of Canadians Either Don’t Have a Family Doctor or Struggle to See the One They Have, Jan. 2025. [Online]. Available: https://angusreid.org/canada-health-care-family-doctors-shortage/ 

[2] Newfoundland and Labrador Health Services, “Province-wide virtual primary health care appointments available to eligible individuals,” 2024. [Online]. Available: https://nlhealthservices.ca/news/province-wide-virtual-primary-health-care-appointments-available-to-eligible-individuals/

[3] CBC News, “For-profit virtual care companies putting patient data at risk, new study finds,” Feb. 2024. [Online]. Available: https://www.cbc.ca/news/health/virtual-care-for-profit-patients-data-1.7109278

[4] OpenAI, Usage Policies, 2025. [Online]. Available: https://openai.com/policies/usage-policies

[5] Mental Health Commission of Canada, “Data and Privacy Standards,” 2023. [Online]. Available: https://mentalhealthcommission.ca/emh-content/data-and-privacy-standards/