In pandemics like Covid-19, the use of autonomy and machine learning technologies are of high importance. The Internet of Things (IoT)-enabled autonomous transportation system (ATS) envisions a fundamental change in the traditional transportation system. It aims to provide intelligent and automated transport of passengers, goods, and services with minimal human interference. While ATS targets a broad spectrum of transportation (cars, trains, planes, etc.), the focus of this article is limited to the use of vehicles and road infrastructure to support healthcare and related services. This article offers an IoT-based ATS framework for emergency healthcare services using autonomous vehicles (AVs) and deep reinforcement learning (DRL). The DRL-enabled framework identifies emergency situations smartly and helps AVs make faster decisions on providing emergency health aid and transportation services to patients. Using ATS and DRL for healthcare mobility services will also contribute toward minimizing energy consumption and environmental pollution. This article also discusses current challenges and future works in using ATS for healthcare services.