Community Story: EasyCare Voice Agent Streamlines Healthcare Workflows

EasyCare Team proudly secured 1st place out of 100+ builders at the Vertical Specific AI Agents Hackathon, sponsored by aiXplain in collaboration with Creators Corner, Mistral AI, AutoGen, and others. Our solution, EasyCare, uses a voice agent that streamlines healthcare by pre-assessing patient needs, improving efficiency and reducing wait times.

Problem: Inefficient Pre-Appointment Assessments in Healthcare

Healthcare workflows often struggle with inefficiencies in pre-appointment assessments, leading to unnecessary delays, repeat visits, and increased burden on medical professionals. Many patients arrive at their appointments without undergoing essential preliminary tests, which prolongs diagnosis and treatment times.

Increased wait times: Patients often arrive at their doctor’s appointments without having undergone necessary tests, leading to delays in diagnosis and treatment.
Unnecessary repeat appointments: Doctors frequently need to request additional tests after initial consultations, causing patients to return for follow-ups that could have been avoided.
Higher costs for healthcare providers: Inefficiencies in pre-appointment assessments contribute to an increased workload for medical professionals and higher costs for healthcare providers.

Goal: Reducing Wait Times and Enhancing Patient Care

Meet EasyCare – a demo that shows how aiXplainʼs platform can improve healthcare workflows. This project uses voice AI agents to determine if additional tests, such as blood work or MRIs, are necessary before patients arrive at their doctor’s appointment. In this article, we will utilize the voice agent of Retell AI to capture patient details and symptoms and then pass the conversation to aiXplain in order to extract the relevant data and store the summary of it into a knowledge base, Senso. This tool brings value to both the patient and the doctor by simplifying doctor visits and significantly cutting down on wait times and unnecessary doctor appointments. This serves as an example of how aiXplain agentic solutions can be successfully utilized in a real-world scenario.

Key Features

Voice agent setup: Uses voice AI to capture patient details and symptoms.
Pipeline building with aiXplain: Analyzes conversation data with aiXplain’s pipeline builder tool, Design, to determine necessary tests (e.g., blood work, MRI).
Data storage with Senso: Stores extracted information in Senso’s knowledge base for easy retrieval and usage by healthcare providers.

Technology Overview

Tech stack

Languages: Python
Frameworks: aiXplain, Retell AI
Tools: Senso
LLMs: GPT-4o, Claude 3.5 Sonnet

Methodology

We built an AI pipeline using aiXplain to process patient conversations captured by Retell AI. The patient’s conversation is transcribed and sent to an AWS Lambda function, which serves as the core processing unit, triggering AI models to analyze the transcript. The results are formatted as structured JSON and stored in Senso’s database. Docker is used to package the Lambda function, ensuring compatibility across different environments.

Step 1: Setting up a voice agent with Retell AI

The system is designed to operate via a dedicated phone line, simulating a clinic’s reception process. As a first step we set up a voice agent with Retell AI. When patients call, the voice agent acts as a virtual receptionist, managing appointment scheduling and collecting relevant details. Prompt and behaviour should be:

## Identity
You are Kate, from the appointment department at EasyCare, and a parient calls you over the phone. The patient would would like to schedule an appointment at the clinic.  
You are a friendly, empathetic health clinic receptionist, deeply concerned about the patient's well-being. While you don’t provide medical advice, you use your knowledge to understand the patient's situation and assist the patient in booking the earliest possible appointment. Your tone is calm, warm, and reassuring, aiming to make the patient feel cared for and understood.

## Style Guardrails
Be Concise: Respond succinctly, addressing one topic at most.
Embrace Variety: Use diverse language and rephrasing to enhance clarity without repeating content.
Be Conversational: Use everyday language, making the chat feel like talking to a friend.
Be Proactive: Lead the conversation, often wrapping up with a question or next-step suggestion.
Avoid multiple questions in a single response.
Get clarity: If the user only partially answers a question, or if the answer is unclear, keep asking to get clarity.
Use a colloquial way of referring to the date (like Friday, February 16th, 2024 at 10am).

## Response Guideline
Adapt and Guess: Try to understand transcripts that may contain transcription errors. Avoid mentioning "transcription error" in the response.
Stay in Character: Keep conversations within your role's scope, guiding them back creatively without repeating.
Ensure Fluid Dialogue: Respond in a role-appropriate, direct manner to maintain a smooth conversation flow.

## Task
You will follow the steps below, do not skip steps, and only ask up to one question in response.
If at any time the user showed anger or wanted a human agent, call transfer_call to transfer to a human representative.
1. Say who you are and that we are from Easycare and ask questions about the caller
  - Ask the caller (e.g., Mich) about their name, date of birth, gender
2. Ask about what medical problems they have.
3. Ask about details of the pain. Ask where it is located more specifically.
4. Ask about when the pain started.
5. Ask about if something happened that may have caused the pain and ask them to try to share as much details that might have caused it.
6. Ask about the pain scale from 0-10. 0 is no pain at all, 10 is very bad pain.
7. Ask if she is feeling any other symptoms apart from stomach pains
8. Ask if there's anything that the doctor should know before proceeding (for allergies or family histories). Ask about any medication being taken to relieve the pain. Based on this and all other questions, you would need to understand based on this whether there is an emergency or not. 
9. Ask if there's anything she likes to add before we finalized the call.
10. Tell that you will get back shortly with her about next steps (example: if test is needed, or if she can directly meet the doctor and the time). Call function end_call to hang up
11.After getting answers to all the questions you can create a report about the diagnostic. In this case diagnostic is indigestion.

Step 2: Back-end lambda function

The Lambda function acts as the bridge between Retell AI and the aiXplain pipeline. When Retell AI transcribes a conversation, it triggers the Lambda function via a webhook, which sends the transcription data to aiXplain for further processing. To ensure compatibility with AWS Lambda’s runtime environment, we use Docker to package the required libraries specified in “requirements.txt” on a Linux environment (linux/amd64). A Python folder is created containing all dependencies, which must be zipped and uploaded as an AWS Lambda layer.

Once the layer is created, proceed to set up the Lambda function as seen in the screenshot.

After creating the Lambda function, add the layer by navigating to “Layers” → “Add a Layer.” Copy the function URL and paste it into the webhook settings section in Retell AI.

Step 3: Create the agent with aiXplain

In this stage, raw patient conversation data, captured by the voice agent and Lambda function, is transformed into actionable insights. We use aiXplain’s AI pipeline builder tool, Design, to create a custom agent that analyzes symptoms, identifies test requirements, and generates concise summaries.

Navigate to Design from your dashboard, start a blank canvas, add an input node, connect it to a router node, and configure connections to build your pipeline.

As shown in the image, we create the following text generation nodes: isBloodtestRequired, isMRIrequired, problemSummary, and nameAgePaindegree, and we define their behavior using GPT-4o (OpenAI) and Claude (Anthropic). Connect the Router node to each text generation node to ensure the conversation flows into each text generator. Each text generation node produces text outputs that should be labeled with descriptive names (e.g., isBloodtestRequiredOutput, isMRIrequiredOutput). Add processing nodes as needed for entity extraction, sentiment analysis, or database storage tasks. Here is a more concise version of our isBloodtestRequired node:

- isBloodtestRequired:
    - LLM: GPT-4o
    - Temperature: 0
    - Max tokens: 20
    - Prompt:
Analyze symptoms to determine if a blood test is needed based on predefined cases involving systemic issues, chronic conditions, infections, bruising, or other specific indicators. 

Cases Where a Blood Test is Typically Required:
[Insert different scenarios such as fatigue, weight changes, frequent urination, infection, etc.] 

Decision-Making Guidelines:
1. Match to the List:
    - If the user’s symptoms align with one or more cases in the list, mark as true.
2. No Match:
    - If the symptoms do not align with any case, mark as false.
3. Unclear or Non-Specific Symptoms:
    - If the symptoms are vague or insufficient to determine the need, mark as false.

Output Format:
The output is JSON and must be structured as follows:
{
    "isBloodtestRequired": boolean
}

Example 1:
Input:
    - User: “I’ve been feeling extremely tired and losing weight recently.”
Output:
{
    "isBloodtestRequired": true
}

Your Task is to:
    1. Focus on the user’s symptom descriptions.
    2. Use the predefined list of cases to assess the necessity of a blood test.
    3. Ensure that the output is strictly formatted as JSON with no additional commentary.
Process the provided symptoms and generate the JSON response accordingly.

Step 4: Set up Senso

We use Senso as a knowledge base for storing and managing patient data. You can interact with your database via RESTful requests using Senso’s API. For more details, refer to the official documentation.

We will showcase how it integrates aiXplain’s tools with Senso’s API for data storage with the help of lambda function. It processes Retell AI transcripts via a webhook, enabling efficient data extraction and preparation for doctor visits and completing the workflow.

Here are some essential helper functions and constants, along with additional utility functions that we used for interactions.

# Create agent
agent = AgentFactory.create(
    name="Doctor assistant",
    tools=[pipeline_tool],
 description="You are an agent that prepares patient information (name, age, pain degree, problem summary) and identifies necessary tests before the visit. Output the information in markdown format only.",
    llm_id=AGENT_LLM_ID  
)
# Parse event JSON
event_body = json.loads(event["body"])
# Extract the transcript
transcript = event_body["call"]["transcript"]
# Run the agent and capture the response
try:
    agent_response = agent.run(transcript)
    print("Agent response is: ", agent_response)
    # Set document content that should be stored in the Senso front-end
    document_content = agent_response.data.get("output", document_content)
    # Extract only the required fields
    response_data = {
        "status": getattr(agent_response, "status", None),
        "completed": getattr(agent_response, "completed", None),
        "output": agent_response.data.get("output") if
isinstance(agent_response.data, dict) else None
    }

    response = {
        'statusCode': 200,
        'body': json.dumps(response_data)
    }
except Exception as e:
    print(f"Error processing request: {e}")
    response = {
        'statusCode': 500,
        'body': json.dumps({"error": str(e)}),
        'message': 'Failed to run the Aixplain agent'
    }
# Add the insights from aiXplain to Senso, where the user can view it
try:     
    org_name = 'Aixplain Demo Org'
    collection_name = 'Patient Records'
    document_title = 'Patient Visit Notes'
    # Create an organization
    print(f"Creating organization: {org_name}...")
    org_response = create_organization(org_name)
    org_id = org_response['org_id']
    print(f"Organization created with ID: {org_id}")
    # Create a collection
    print(f"Creating collection: {collection_name}...")
    collection_response = create_collection(org_id, collection_name)
    collection_id = collection_response['collection_id']
    print(f"Collection created: {collection_id}")
    # Create a document
    print(f"Creating document: {document_title}...")
    document_response = create_document(org_id, document_title, document_content)
    document_id = document_response['document_id']
    print(f"Document created: {document_id}")

    # Add document to collection
    print("Adding document to collection...")
    add_response = add_document_to_collection(org_id, collection_id, document_id)
    print(f"Document added to collection: {add_response}")

Results

EasyCare is a great example of how aiXplain’s AI platform can enhance with your current systems, providing insights and improvements that help make workflows faster and boosts user experiences. Whether you’re focusing on automating early-stage diagnostics or need specific analysis for your industry, aiXplain lets you easily build, deploy, and improve complex AI solutions.

We hope this demonstration inspires you to explore new ways to improve efficiency, accuracy, and innovation with aiXplain. Keep an eye out for more practical examples that showcase what is possible with cutting-edge AI tackles everyday challenges.

Output example

Our agent is able to correctly classify outputs, which can then be used by downstream services like Senso to take action.

Conclusion

EasyCare integrates voice AI and aiXplain’s platform to automate pre-appointment assessments, reducing wait times and improving efficiency. The project highlighted the importance of secure data handling, seamless integration, and rigorous testing. Challenges included aligning multiple AI systems and ensuring accurate patient data extraction. Overcoming these hurdles resulted in a scalable, patient-focused solution.

Future Work

Future developments could focus on integrating multilingual support to cater to a broader patient demographic to further enhance EasyCare. Expanding AI capabilities to include predictive analytics could help identify high-risk patients before their visits. Incorporating real-time integration with electronic health records (EHR) would allow seamless data sharing between healthcare providers. Additionally, developing a mobile-friendly interface for patient self-assessments could improve accessibility and engagement. Lastly, refining AI-driven symptom analysis with more specialized medical datasets could enhance diagnostic accuracy and reduce false positives.