What are the challenges of using AI in the healthcare sector?
India’s healthcare industry is on a meteoric rise, projected to soar to $650 billion by 2025, driven by an impressive 22% compound annual growth rate since 2016. At the forefront of this transformation is artificial intelligence (AI), with its integration into healthcare set to skyrocket to a $1.6 billion market by 2025, revolutionizing diagnostics, treatment, and patient care like never before, growing at 40%.
The healthcare sector is set to receive 2.5% of India’s GDP as government investment, and AI in healthcare could add $25-30 billion to India’s GDP by 2025. Tech giants like Google (partnering with Forus Health) and Tata Group (through Tata Elxsi) are all set to drive innovation in the healthcare sector. The sector itself is ripe for innovation and needs to tackle barriers like poor connectivity and a lack of healthcare professionals.
India’s healthcare ecosystem faces several challenges, including limited access to medical professionals in rural areas, inadequate infrastructure, high healthcare costs, and connectivity issues in remote regions. AI-powered solutions—such as predictive diagnostics, automated medical imaging, virtual healthcare assistants, remote diagnostics, error-reduction systems, and healthcare delivery optimization models—can help bridge these gaps by improving quality, reducing costs, and enhancing accessibility for all. While AI presents unprecedented opportunities to enhance healthcare delivery, there is an urgent need to assess the existing regulatory framework for AI implementation in this sector.
Existing legal structure
The advancement of AI depends on its ability to process and analyze vast amounts of data. Machine learning models require extensive datasets for training, which can either be manually provided by users or gathered incidentally as AI performs tasks. In healthcare, this data often includes sensitive personal information such as medical history, diagnostic test results, and genetic details, all of which are essential for delivering effective medical services. The Digital Personal Data Protection Act, 2023 (DPDPA), India’s first comprehensive law on the subject, does not explicitly address AI processing, and all obligations are tied to human actors.
The Indian government is working towards establishing a National Digital Health Infrastructure (Ayushman Bharat Digital Mission), to create a nationwide digital ecosystem that integrates healthcare service providers and patients through unique health IDs, with the use of emerging technologies. This will likely result in large pools of medical data being integrated into AI systems for the purpose of at least organizing and indexing, if not analysis and decision making.
The eventual implementation of the DPDPA will have to be considered in respect of the healthcare sector. The provisions of health services necessarily involve several checkpoints for data collection and processing (including data of a patient’s family, guardians, and visitors), and it will be difficult to impose any liability on personal data processing by AI, in the absence of clear guidelines.
What are the existing guidelines?
The Indian Medical Council (Professional Conduct, Etiquette, and Ethics) Regulations, 2002, which govern the professional conduct of healthcare providers in India, have been further supplemented by the Indian Council of Medical Research (ICMR) Guidelines on the Ethical Application of Artificial Intelligence in Biomedical Research and Healthcare. These guidelines currently mandate a ‘human-in-the-loop’ approach, requiring healthcare practitioners to review and validate AI-generated results before they are communicated to patients. This ensures human oversight, maintaining the accuracy and reliability of AI-driven decisions in patient care. However, as AI capabilities continue to evolve, the need for dedicated legislation becomes increasingly crucial to prevent potential risks and unintended consequences for end users.
The government’s National Digital Health Mission (NDHM) aims at creating a robust digital healthcare ecosystem by integrating the use of AI into healthcare data management. The NDHM also emphasizes on creating a secure, unified health data platform and trustworthy clinical decision support systems while establishing clear guidelines for data interoperability, privacy and security.
Regulatory gaps and challenges
Importantly AI algorithms can exhibit biases due to non-diverse training data, potentially leading to misdiagnoses, especially for underrepresented populations. This challenge is particularly significant in India’s diverse demographic landscape, raising ethical concerns about the fairness and accuracy of AI-driven healthcare solutions. The issue is further compounded when AI models trained on foreign or predominantly urban medical data are applied across India. For instance, an AI algorithm designed to predict breast cancer risk may inaccurately classify women of color as ‘low risk’ due to the lack of diverse training data, resulting in potential disparities in diagnosis and treatment.
There is a lack of a comprehensive regulatory framework governing AI, while the responsibility for any AI tool oversight is unclear and largely depends on contractual arrangements. Further, the testing requirements for diagnostic AI tools are insufficient, and the liability attribution in cases of AI misdiagnosis is also ambiguous. This could result in inaccurate results and consequently unreliable diagnosis, jeopardizing patient safety.
Potential legal framework
To ensure accurate and fair AI outcomes, it is essential to mandate diverse dataset requirements that account for ethnic diversity, geographic representation, and socioeconomic variation in AI training. Statutory obligations should include regular audits of AI systems and the implementation of continuous monitoring protocols. A key priority is the establishment of a dedicated regulatory authority for healthcare AI, responsible for developing comprehensive testing and approval processes, defining clear liability frameworks, and implementing effective grievance redressal mechanisms.
