Listener Volume

It's important to remember exactly WHAT the UltraPort Listener does at the most basic level. It receives an HL7 message, it writes that HL7 message to a file in a data folder, it creates some log entries (if configured), and then (if required) it will send an HL7 Acknowledgement back to the sender. That is ALL that it does. The HL7 message is left in the folder for some other process to consume and dispose of.

How fast is TOO FAST? This means how quickly can YOUR process consume and dispose of HL7 message data files in a folder? If the UltraPort Listener can write 100 message files per second into a data folder, and your application can only process/consume and then delete 50 message files per second does that pose a problem, yes or no? If the 100 messages per second is sustained traffic then it certainly poses a problem, a fatal one. Remember that we are using the same MS Windows operating system and file system that you are, so you can do your own math to determine how long it will take before there are too many files in the file system and your server crashes or becomes too unwieldy to function properly.

It is extremely important that you define what your reasonable expectations are.

Below is a real (paraphrased for clarity) example of a conversation between our HL7 support professionals and customer(s).

HermeTech: What is your anticipated daily message volume?

Customer: 100,000 messages, so I need a listener which can handle 100,000 messages per day.

HermeTech: 100,000 per day, so if we round up to the nearest thousand that's 5000 messages per hour over a 24 hour period. Is that correct?

Customer: Well, no, 90% of the traffic will be received in normal business hours 8AM to 5PM. So that's 90,000 messages over 9 hours that's 10,000 messages per hour. I need a listener that can handle 10,000 messages per hour.

HermeTech: Ok, so you need the ability to handle 10,000 messages per hour of sustained traffic, that's 240,000 messages per day. Is that correct?

Customer: Yes.

HermeTech: Now what happens if your receiving server is down for 4 days, or something you have no control over, their sending server is down for 4 days (a long weekend). When connections are reestablished there will be 400,000 messages backlogged/queued from the 4 down days PLUS the 100,000 new messages expected for the current day. Your total capacity is 240,000 messages per day so it will take you a minimum of 2-3 days to handle the backlog. Is that acceptable?

Customer: Goodness, no! My clients would kill me. I need to be able to handle a backlog or data dump like this in a couple of hours at most.

So, this customer example went from needing a HL7 listener capable of handling 100,000 messages per day, to needing one capable of handling 200,000 messages per hour what's the real answer?

One thing to keep in mind is that HL7 message interfacing is ALWAYS a negotiation between the two parties (trading partners). It is also completely reasonable for either trading partner to establish reasonable expectations.

Here's a real world scenario that we have encountered the actual details are an amalgamation of client scenarios consolidated for clarity and to make a point. This conversation is quite blunt and to the point.

Customer: A customer is running a 500 port Virtual Machine license for the UltraPort HL7 Listener and they find that they are being overwhelmed by traffic from their trading partner (yes, of course it's possible). On discussing it with them we discover that 5 of their customers are using specialized (and highly expensive) mirth devices. Hardware specifically designed to handle high volume HL7 message traffic.

HermeTech: So, let's make the situation absolutely clear and drill the problem down to just one of your trading partners. They have spent over US$30,000.00 on special hardware capable of delivering 150-200 messages per second to you. This hardware is state-of-the-art and has a massive amount of memory, all solid state HDDs, the very best network connectivity, etc. These are the steps that THEY took to send YOU HL7 messages. Now, what steps did YOU take to receive them. You spent US$4.00 on HL7 Listener software (if they paid the full price for a 500 port UltraPort Listener License the "per-port" cost works out to uS$4.00). Rather than dedicate a physical server, you chose to use a virtual server (anyone who thinks that virtual computers somehow run better or faster than their underlying hardware honestly needs to go back to school) rather than dedicate hardware, you are using IDE drives rather than SSD drives, etc. Why on earth would you possibly expect your solution to be adequate without predefined expectations and boundaries agreed to by both trading partners?

Building on section 3, if you are exchanging high volumes of HL7 messages it is vitally important that you establish boundaries and manage the expectations of your Trading Partner and yourself. It is perfectly reasonable for you to ask that your trading partner manage or "throttle" the speed at which they deliver HL7 messages to you. On your side with the UltraPort Listener you have the option of setting up a Message Throttle to regulate the speed with which HL7 transactions are processed. Also make sure that you review section 5 on how to optimize the environment for your HL7 Listener.

To review a point made in section 1. It's important to remember exactly WHAT the UltraPort Listener does at the most basic level. It receives an HL7 message, it writes that HL7 message to a file in a data folder, it creates some log entries (if configured), and then (if required) it will send an HL7 Acknowledgement back to the sender. That is ALL that it does. The HL7 message is left in the folder for some other process to consume and dispose of.

Think carefully about what this means and remember that the UltraPort Listener is using the same MS Windows Operating System that you are. When you are doing very high volume then being able to shave just a few milliseconds off of any operation can dramatically speed things up.

•Consume and DELETE your HL7 message data files ASAP. If you are receiving 1 million HL7 messages per day and whatever process of yours that consumes the HL7 messages is DOWN then you should probably STOP the UltraPort Listener until your process is back up. Why? It's simple, isn't it better to have your Trading Partner not be able to send you messages than to have 1 million individual files build up in a folder on YOUR file system? Try a simulation and actually create 1 million files in a folder then try to view that folder in MS Windows Explorer, try sorting it by date, or file name. That result speaks for itself. The UltraPort Listener has to look at that folder, it has to create unique file names in that folder and verify that they don't already exist. As the folder size grows, these operations become slower and slower. Left unchecked, your entire system runs slower and slower and will eventually fail entirely. MS Windows absolutely HATES (and always has) folders with hundreds of thousands or millions of files in them.

•Always make sure that you and your Trading Partner are using the full HL7 transaction (IE always make sure that you are using HL7 Acknowledgements). If you aren't using Acknowledgements then there is no guarantee that your HL7 messages will be successfully processed.

•Always use LOCAL HDDs. As opposed to network shares or even NAS (network attached storage). It doesn't matter how fast your network connection is, it's still slower than a local HDD.

•Use Solid State Drives. This can make a HUGE difference. With the UltraPort Listener, the biggest bottleneck (after network/VPN speed) is always the file system. Solid state drives will give you an order of magnitude performance capability boost, and their price/capacity has now become reasonable enough that there really is no excuse for not using them.

•Disable the Anti-Virus for HL7. HL7 messages are ASCII text streams. It is impossible to transmit or pass a virus in a raw HL7 message. You can improve performance by setting up exceptions in your Anti-Virus so that it's not monitoring all disk/writes and file creation in your HL7 data folders.

•Virtual or Physical.  Do you need to dedicate hardware to your Listener solution? Or is a Virtual Machine sufficient? This question is really up to you, unless you think that Virtual Machines somehow run better or faster than the underlying physical hardware that hosts them, in which case, you should seek assistance in making this decision.