Search the OSCAR Documentation
< All Topics
Print

Lab Download Integration

UpdatedSeptember 5, 2024
By Peter Hutten-CzapskiKevin Lai

Preface

These are instructions on how to set up electronic data transfer (EDT) for downloading your labs in HL7 format from each individual major laboratory, and uploading them to your OSCAR EMR. This is the original method of getting electronic lab data to your EMR without the need for OntarioMD certification. The other newer methods are with OLIS (if you have an OntarioMD certified EMR), which the open source OSCAR Community Edition is not eligible. You vendor may have other newer methods of downloading lab data such as Excelleris (check with Open OSP Service Cooperative) for their solution.

Document Version History

  • v1.0 – Initial public release to oscarmanual.org – Jan 12, 2016
  • v1.1 – Initial OSCAR 15 release – Mar 5, 2016
  • v2.0 – Ported over to oscargalaxy.org – 11-Oct-2022 (KC Lai)
  • v2.1 – Added instructions on setting up lab download from lab vendors (KC Lai)
  • v2.2 – Updated and documented Linux and Excelleris approach – Aug 2, 2024

Prerequisites

  • You are familiar with the Command line, dealing with source code and Java build tools.
  • You have a dedicated server (or VM) that can stay on 24/7.
  • You have a static IP for your clinic system that runs the lab downloader. Some labs restrict lab download to known static IP’s of authorized clinics/physicians. You need to contact your internet provider and pay a little extra for a static IP. If you static IP changes (upgrade services or new provider), remember to update the labs of your new static IP.

Overview

As a physician, you may order labs and patients may go to one of several  laboratory vendors.  In Ontario there are three major labs: Lifelabs (CML/MDS), Dynacare (Gamma Dynacare Medical Laboratory), and AlphaLabs. There are other smaller labs, it will depend on your volume, the presence of a lab “channel” in OSCAR that can handle the labs Hl7 “dialect”, and the labs technical people if it is practical to set them up.

The general process is this: Each lab will have a SFTP or API service that you can download HL7 lab files (electronically coded lab data) to a folder on your dedicated lab download computer or virtual machine. An HL7 transport utility (Mule) will upload the HL7 lab files from the folders to your EMR via a secure channel with private-public keypair in OSCAR for each lab vendor. The EMR will then process the HL7 lab data and file to the corresponding patients, and will show up in your InBox for review.

These instructions are mainly for a Windows system, but you can try to adapt the instructions for Mac or Linux.

Get Electronic Data Transfer (EDT) accounts

Lifelabs Excelleris

  • Contact a LifeLabs Client Products Specialist by emailing ITHelpDesk@lifelabs.com and tell them that you are “a medical doctor that is self-hosting your own OSCAR EMR server” and that you need to be set up with the “Excelleris (ON)” and be given a “PCX file” and login credentials to download the labs.
  • Install the PCX file into your browser and you will be able to download manually for testing
  • Follow the ExcellerisDownload.sh instructions below for automation.

LifeLabs LRD (older approach)

  • Contact a LifeLabs Client Products Specialist by emailing ITHelpDesk@lifelabs.com and tell them that you are “a medical doctor that is self-hosting your own OSCAR EMR server” and that you need to be set up with the “LifeLabs Results Distribution System (LRDS)” and be given a “four digit route” and be given SFTP credentials (if possible) to download the labs.
  • You need to get the form “Electronic Results Request Form” to fill out and return to LifeLabs.
    • Vendor: specify your clinic name
    • Contact: specify yourself
    • PC/Server Operating System: Windows Pro  or Mac (unfortunately, there is no Linux version)
    • EMR Software Version: OSCAR McMaster / Community Edition v.19 (or whatever you are using)
    • Switch (Winblast to SFTP): leave this blank
    • New Install: checkmark this
    • Client information: fill this section out with your physicians that need lab download to your system (you can add multiple physicians now or later to your account)
  • LifeLabs will ask you for your OSCAR EMR vendor, and you will have to reply to them again and clarify that you are your own self-hosted vendor and need to be able to download the HL7 files yourself to upload to your EMR. Remind them that you are a MD medical doctor (not some random IT person), that you do not have OLIS, and that you need to be able to download labs to do your doctor work.
  • LifeLabs will not actually give you your SFTP credentials, but will pre-package a WinSCP-type program (“proprietary SFTP client”) for you to install that has your unique ID and password and some custom scripts to be able to access their secure SFTP server to download HL7 lab files from your account to your computer. This is a rather ancient and cumbersome method, but LifeLabs has no incentive to pay developers to update this original EDT system when they have OLIS or Excelleris.
  • You may also need to provide LifeLabs with your static IP address from where you are downloading the lab to.
  • LifeLabs will send you a one time download link via a secure email. Do not click this link yet!  Proceed with this next steps in the document to set up your dedicated lab download workstation  or virtual machine, before you continue to set up the LifeLabs downloader. The software download link is a one-time access, and it may be time limited, so you can consider setting up your lab download system first before requesting the link. If the link expires, you can contact your LifeLabs Client Products Specialist to send it to you again.

Dynacare

  • Contact Gamma Dynacare Medical Laboratories and tell them that you are a medical doctor, and that you are requesting to be set up with an “eResults SFTP account to download HL7 lab files for importing to your EMR“. Dynacare likes physicians to use their web-based Contact Us form (choose the option for Healthcare Providers and Hospitals). They don’t have a specific general email to contact. But once you get the email of one specialist, you can build a rapport with them and they can be quite helpful.
  • If you are asked which system you are interested in being set up, tell them that you need the “Version 2.0 Batch Service (EMR) Users“. You can also request and account for the web based Online Transaction Service User (v 3.x), but this is more for downloading individual lab results for one patient at a time (not what you need for EMR).
  • You will be asked to complete and sign a Dynacare eResults – Terms of Use Agreement.
    • Organization / Clinic: your clinic name
    • User Name: give yourself a unique download username
    • Email: your email
    • Results to be: choose “Downloaded into your EMR”
      • EMR Provider: self-hosted
      • Name of software: OSCAR EMR
      • Phone: your phone for tech contact
  • You will be given a SFTP username, password, and the website of their SFTP server. Please keep this credentials private and confidential!
  • You may also need to give them your fixed IP address from where you will be doing the batch lab download from.

AlphaLabs

  • Contact AlphaLabs and ask them for an SFTP lab download account.
  • You will be given an SFTP username, password, and the website of their SFTP server. Please keep this credentials private and confidential!

Create a Virtual Machine to download and process labs

Lifelabs “LRD” service requires a Windows PC or a Mac to be the lab downloader. For this reason most lab downloader instructions are based on Windows however the recent option of using the newer Excelleris “Rover” API for Lifelabs lets you run the service on Linux.

An even better solution, would be to set up virtual machine on your server. Follow the same instructions on setting up a virtual machine for OSCAR, but instead of installing OSCAR, install your own copy of Windows Pro.  We recommend Pro versions because it allows you to remote desktop to the Windows and manage the downloader files easily. This lab download method has been tested to work with Windows XP Pro. It has not been tested by us to work with newer Windows 10 / 11, check with Lifelabs technical people if their solution works with the Windows version you are installing. Do not allow anything else or anyone else to use this machine to limit user induced security vulnerabilities.

  1. Set up a Windows workstation or virtual machine to be the dedicated lab downloader.
  2. If you don’t know how to set up the downloaders to run as a Service, you could set up Windows to automatically log-in to an account on startup, and then immediately Lock Screen (to switch out but stay logged in)

Setup the Lab Downloaders

Lifelabs Excelleris

  • A script can be obtained free from Peter Hutten-Czapski to download on a linux machine the Excelleris files from their “Rover” API.
  • Place the supplied ExcellerisDownload.sh script into a DIRECTORY in your server and make it executable
  • Copy the Excelleris PFX file for Clinic to a subdirectory with structure similar to
    <SCRIPT DIR>/excelleris_download/cert/QA Peter Hutten Czapski MPC.pfx
  • Create a config_inc.txt file in the top <SCRIPT DIR> with the following contents: 
    CONTEXT="<Whatever you like to identify this set>"
USERNAME="<Excelleris supplied User ID>"
PASSWORD="<Excelleris supplied Password>"
PFX="<Excelleris supplied PFX file>"
CERT_PASS="<Excelleris supplied certificate passphrase>"
HL7URLPATH="<test or production URI for the given province as supplied by Excelleris>"
MULE_HOME="<path to mule-1.3.3 directory that you installed (see Mule Uploader below)>"
MULE_HL7PATH="<path to downloaded lifelabs reports from your Mule configuration>"
MULE_ERRORS="<path you configured/will configure for reports that fail to upload to OSCAR via Mule from your Mule configuration>"
MULE_DONE="<path you configured/will configure for uploaded files>"
MULE_LOG="<path to Mule log file>"
FILE_LOG="<path to Downloading script log file>"
EMAIL="<email address for error report>"
SLEEP=<no quotes just the typical number of seconds needed for OSCAR to process upload>
  •  Example settings 
    CONTEXT="OSCAR 19 Production"
USERNAME="hfhtclinic"
PASSWORD="password"
PFX="QA Peter Hutten Czapski MPC.pfx"
CERT_PASS="passphrase"
HL7URLPATH="https://api.on.excelleris.com/hl7pull.asp"
MULE_HOME="/home/peter/hl7_file_management/mule-1.3.3"
MULE_HL7PATH="/home/peter/HL7Data/incomingHL7Dir"
MULE_ERRORS="/home/peter/HL7Data/errorHL7dir"
MULE_DONE="/home/peter/HL7Data/completedHL7dir"
MULE_LOG="${MULE_HOME}/logs/mule.log"
FILE_LOG="excelleris.log"
EMAIL="peter@hfhtclinic.net"
SLEEP=300

LifeLabs LRD (older)

  • In your Windows or Mac dedicated lab download workstation or virtual machine, use the link in the LifeLabs email to download the LRDS software.
  • Install the LRDS software on the machine.
  • The installer will create the following folders
    • C:\lifelabs
    • C:\DATA\LIFELABS
  • The program to run to download labs is “C:\lifelabs\HL7Main.exe”
  • Create a “Scheduled Task” in Windows or Mac to regularly run this program “C:\lifelabs\HL7Main.exe” every hour (or however often you want to check for new labs). Remember to allow this Task to run as administrator.
  • The HL7 files will be dropped in to the folder “C:\DATA\LIFELABS”
  • The LifeLabs downloader will check for new labs, download the HL7 files, and then generate their “CURHST” file which they use to validate whether a lab was successfully downloaded before deleting it off their server.
  • A script can be obtained from Ian Pun to use Linux to download LRD.

Dynacare

  • Create a directory “C:\DATA\GDML” to store your log files and incoming HL7 files from Dynacare
  • Create a directory “C:\home\GammaDynacareScript” to hold the custom lab download scripts
  • Use the custom Windows batch file script (created by KC Lai) and the Python script (created by Ian Pun) to automatically download from Dynacare. Join OSCAR Ontario Facebook Group to contact Kevin or Ian for the custom scripts.
  • Edit the batch file and the Python script to match your instance (Dynacare SFTP username and password)
    • user=”username”
    • pw=”password”
  • Create a “Scheduled Task” in Windows or Mac to regularly run KC Lai’s batch file (for logging the execution of Ian Pun’s Python script) on an hourly schedule.

AlphaLabs

  • Create a directory “C:\DATA\ALPHA” for log files and incoming HL7 lab files from AlphaLabs
  • Create a directory “C:\home\AlphaLabScript” for the custom lab download script
  • Install WinSCP so you can run a custom download script from SFTP servers
  • Open WinSCP and make an FTP connection to AlphaLabs SFTP server with your username and password (given by AlphaLabs)
  • Copy down the SSH host key fingerprint to the authorized AlphaLab’s SFTP server. You will need this to put in to the custom script to automatically connect without manually confirming an authorized server each time the script runs.
  • Create a WinSCP script in “C:\home\AlphaLabScript\AlphaLabDownloadScript.txt” with the following text (be sure to edit the username, password, domain name of the SFTP server, and the hostkey fingerprint details to suit your instance). This text file must be kept secure (as it contains plaintext passwords) and should not be accessible by anyone except yourself or a trusted IT person.
# Connect to AlphaLabs and Download labs
# by Kevin Lai (2-July-2019)
# Accept any hostkey:
# open -hostkey=* sftp://username:password@alphalaboratories.ca
# Accept verified hostkey:
open -hostkey="ssh-rsa 1024 e3:4f:5h:63:g3:5h:6j:6g:2g:k8" sftp://username:password@alphalaboratories.ca
# Change directory
cd /outbox
# Download files and delete from source when downloaded
get -delete "*.*" "c:\DATA\Alpha\Alpha%TIMESTAMP#yyyymmddhhnnss%-*.*"
# Download files but leave on server
#get "*.*" "c:\DATA\Alpha\Alpha%TIMESTAMP#yyyymmddhhnnss%-*.*"
# Logoff
close
# Quit WinSCP
exit
  • Create a Windows batch file and save it to “C:\home\AlphaLabScript\AlphaLabDownloadBatch.bat” with the following text:
REM Batch file for downloading AlphaLabs HL7 labs
REM by Kevin Lai (2019)
REM Logs output with timestamp
echo off
REM Determining the datestamp for backup folder:
for /f "tokens=2 delims==" %%a in ('wmic OS Get localdatetime /value') do set "dt=%%a"
set "YY=%dt:~2,2%" & set "YYYY=%dt:~0,4%" & set "MM=%dt:~4,2%" & set "DD=%dt:~6,2%"
set "HH=%dt:~8,2%" & set "Min=%dt:~10,2%" & set "Sec=%dt:~12,2%"
set "datestamp=%YYYY%%MM%%DD%" & set "timestamp=%HH%:%Min%:%Sec%"
set "fullstamp=%YYYY%-%MM%-%DD%_%HH%%Min%%Sec%"
echo on
echo datestamp: "%datestamp%"
echo timestamp: "%timestamp%"
echo fullstamp: "%fullstamp%"
echo Running AlphaLabDownloadBatch.bat on... "%fullstamp%" >> c:\DATA\Alpha\Alpha-log.txt
"c:\Program Files (x86)\WinSCP\winscp.com" /ini=nul /script=AlphaLabDownloadScript.txt /log=winscp-log.txt /loglevel=0 >> c:\DATA\Alpha\Alpha-log.txt
  • Create a “Scheduled Task” in Windows or Mac to regularly run KC Lai’s script “C:\home\AlphaLabScript\AlphaLabDownloadBatch.bat” on an hourly schedule.

Installing Mule Uploader to EMR

Although you can now take the HL7 lab files downloaded with the above scripts and manually upload them in to your OSCAR EMR via “InBox -> HL7 Lab Upload” and select the correct Lab type depending on which lab vendor the file is from:

  • ALPHA is AlphaLabs
  • GDML is Dynacare
  • MDS/LifeLabs is LifeLabs (old format)
  • Excelleris(ON) is LifeLabs Excelleris (new)

We recommend you install Mule which automatically takes the HL7 files in a specific directory and uploads it to your EMR.  You may need to adapt the instructions for your situation (ie. version of Java or Maven, 32bit or 64bit installer etc)

Labwork can be pushed into OSCAR with the HL7 transport utility from a Windows workstation. Use similar instructions to install on a Linux machine.  Which system you install on really depends on what is the most convenient location for the downloaded files.

Installing the Infrastructure Packages

Java

The Java you have installed on your system is NOT the Java you will be using.  The version below has been tested, you can use a newer one, just ensure that it 32 bit.   DO NOT attempt building using the 64 bit version. The wrapper that starts this Mule needs a 32 bit Java.

For Linux this means that Ubuntu is NOT the OS for you as they have dropped all 32 bit support a while ago.  You will have to build on Debian or Mint if you want to use the .deb Java package and have the other 32 bit libraries installed for you.

The following example uses the file pattern for openlogic openjdk Java 8 update 422 from https://www.openlogic.com/openjdk-downloads.  Other sources of 32 bit openjdk Java may be appropriate but have not been tested.

openlogic-openjdk-8u422-b05-windows-x32.msi

openlogic-openjdk-8u422-b05-linux-x32-deb.deb

The typical windows installation process will place the executable files in c:\Program Files (x86)\Java\jdk1.8.0_422\jdk or similar

Maven

Maven is tool that you will use to compile a java JAR file configured for your system. Download the latest maven from apache. It will be in a bin tar which you need to extract using either a command line tool such as TarTool or a GUI extractor such as 7-zip. Extract to an appropriate location such as Program Files.

TarTool apache-maven-3.3.9-bin.tar.gz "C:\Program Files\maven-3.3.9"

The maven package is thus extracted into a subdirectory.

For linux you can 
sudo apt install maven

 

Git (or get the source as a zip)

Git is a software configuration management (SCM) tool that you can use to retrieve the source files for this program from the repository. If you prefer you can skip installing git and just download the source as a zip

For linux Ubuntu sudo apt install git. For windows you can download either the 32bit or 64bit installer from https://git-scm.com/download/win

Then use Git to clone the source files into an appropriate location.  For windows use C:\home

mkdir C:\home
cd C:\home
git clone git://git.code.sf.net/p/oscarmcmaster/hl7_file_management oscarmcmaster-hl7_file_management
// OR if you have bitbucket credentials
git clone git@bitbucket.org:oscaremr/hl7_file_management.git

OSCAR Keys

Log into the OSCAR server and click the admin tab at the top right. Scroll to the bottom of the Administrative Page and click on the
Key Pair Generator link to open the Key Manager

Click on Create New Key.  If you don’t have a “OSCAR public key” visible (as above) then input ‘oscar’ without the quotes as the service name, press the ‘Create Key Pair’ button. If you received the error: “Failed: The oscar key pair has already been created”. It is ok to ignore it.

If you have already an OSCAR public key then you need to generate a private key for each lab type. Click on the Create New Key and then provide the service name and Lab type for each of the lab types you need to import.

The Oscar key is one shared key which authenticates who the server is to other people, that’s why there’s only one Oscar key (Oscar server is always the legitimate server to everyone). The “private keys” are service keys, which are keys specific to each service. So if you need 5 different lab services you have 5 different service keys so you can control access to each service individually (ex. disable a single connection).

If you need to delete any key (this should not be done lightly), they are stored in the publicKeys table.  The OSCAR Key is stored in oscarKeys

 

Remember the location where you saved it. (ie.  “C:\home\oscar-keypairs\keypair_gdml.key”)

Note: The keypair file saved contains line feeds code (hidden CR/LF) which must be removed for Mule to read the keys correctly. In other words, you need to delete the hidden return/line feed that separates each line of the key so that the key is on one single line with no word wrap. Use an editor such as Notepad++ to edit the keyfile and can save in Linux format and also show you the hidden CR/LF line feeds. The default Windows Notepad does not save well in Linux format and so the keyfile may fail.

Configure Properties File

Edit the file ‘LabProperties.properties’ in the ‘~/oscarmcmaster-hl7_file_management/src/main/resources’ directory to use your own directories. Note the UNIX style forward slash usage “c:/” for directory paths even for Windows.  The directories you specify will be created at runtime if they don’t exist. Also make sure to include the address of the OSCAR server and the location of ‘keyPair.key’ that you saved in the previous step. Instructions are included at the top of the file, example settings are listed below.  The smpt settings are optional but put something there in case you want to implement it later.

Example settings for a Linux system downloading from one Lab (Excelleris)

incomingHL7dir1 = home/peter/HL7Data/incomingHL7Dir
errorHL7dir = home/peter/HL7Data/errorHL7dir
completedHL7dir = home/peter/HL7Data/completedHL7dir

keyLocation1 = home/peter/HL7Data/excelleris_lifelabs.key

oscarURL = https://your.oscaremr.com:8443/oscar
# smtpServer for notifying errors (optional)
smtpServer = localhost:25
senderEmailAddress = labupload@yourdomain.com
recipientEmailAddress = username@email.com

For Windows use Linux style paths prefixed with the drive eg for two labs

incomingHL7dir1 = c:/DATA/LIFELABS
incomingHL7dir2 = c:/DATA/GDML

Configure Config File

Depending on how many different message formats you wish to use you will have to use a different config file. If you are downloading just one lab then you can leave mule-config.xml as is. Otherwise use the template for the number of message formats (eg. mule-config_two-formats.xml for when you are downloading from Lifelabs and Dynacare), and save to ‘src/main/resources/mule-config.xml’. You will not usually need to edit the source template.

WINDOWS (linux follows)

Environmental Variables WINDOWS

You should set some system variables so that Windows knows where to find the various components

  1. Open the ‘System’ control in the ‘Control Panel’
  2. Under the ‘Advanced’ tab click the ‘Environment Variables’ button
  3. In the ‘System variables’ window click the ‘New…’ button
    1. Variable Name: MULE_HOME
    2. Variable Value: C:\home\oscarmcmaster-hl7_file_management\mule-1.3.3
    3. Click ‘OK’
  4. In the ‘System variables’ window click the ‘New…’ button
    1. Variable Name: MAVEN_HOME
    2. Variable Value: C:\Program Files\maven-3.3.9
    3. Click ‘OK’
  5. In the ‘System variables’ window highlight the ‘Path’ variable and click the ‘Edit…’ button
    1. add ‘;%MULE_HOME%\bin;%MAVEN_HOME%\bin’ to the end of the ‘Variable Value’
    2. Click ‘OK’
  6. Click ‘OK’

Build the Package Windows

For windows open the command line by selecting ‘Run…’ from the start menu and typing in ‘cmd’ and clicking ‘OK’

In any system navigate to the ‘oscarmcmaster-hl7_file_management’ directory within the command line.

For Windows set JAVA_HOME to point to the JDK using the following command ( be sure the directory is correct ):

set JAVA_HOME=c:\Program Files (x86)\Java\jdk1.8.0_422\jdk


Use maven to build the source:


mvn package


Wait until you get “BUILD SUCCESS” and copy the resultant jar to the mule library (the existing jar in source needs to be overwritten):


copy target/IncomingHL7Management-1.0-SNAPSHOT.jar mule-1.3.3/lib/user/IncomingHL7Management-1.0-SNAPSHOT.jar


Leave the command line open for the next step.


Install and Start Mule Windows


Type the following into the command line:


mule install


Mule is now installed as a Windows service, it will start when windows boots. It can be started or stopped using the following commands:


mule start
mule stop


Mule can be removed from being a service with the following command:


mule remove


If you want to add a third lab or change the config file, remember to stop Mule, remove Mule as a service; delete the target jar and then go back to editing the properties file, generating the Java package, and reinstalling the updated Mule.


mule stop
mule remove


Linux Instructions


Build the Package Linux


Navigate to the ‘oscarmcmaster-hl7_file_management’  (or similar) directory within the command line.


For Linux set JAVA_HOME to point to the JDK using the following command ( be sure the directory is correct for your system ):


sudo export JAVA_HOME=/usr/lib/jvm/openlogic-openjdk-8-hotspot-i386
sudo export MULE_HOME=/home/peter/hl7_file_management/mule-1.3.3


Type the following into the command line:


cd /home/peter/hl7_file_management/utils
sudo ./install.sh


Mule is now installed as a Linux service, it will start when Linux boots. It can be started or stopped using the following commands:


sudo /etc/init.d/muled start
sudo /etc/init.d/muled stop


If you want to add a third lab or change the config file, remember to stop Mule, remove Mule as a service; and then go back to editing the properties file, generating the Java package, and reinstalling the updated Mule.


sudo ./uninstall.sh


Other Notes


Original instruction for Importing HL7 Labs by Peter Hutten-Czapski initially posted to oscarmanual.org

Supplementary instructions from Marc Dumontier that might be helpful for Linux as well.


Credits


    

  • Dr. Peter Hutten-Czapski
    • 

  • Marc Dumontier
    • 

  • Dr. Kevin Lai
    • 

  • Dr. Ian Pun
    • 




		
            				
 				
  				

			   				
  
		
        

		        
     

			
Table of Contents