The eflow blog

I made a second crucial memory leak discovery on Friday. Apparently, the default JSP compiler for Tomcat, Jasper, has a “bug” that doesn’t free memory with objects used in a ForEach or Set taglib (found via the always helpful webapp memory leak page).

Both of these tags are used judiciously on the patient dashboard and other pages throughout OpenMRS. As the day progressed, the 40+ users we have in Eldoret would gradually visit more and more pages with different data displayed. The foreach tags that looped over the encounters, obs, patients, etc, wouldn’t release those objects. Eventually, enough of our database would be kept in Tomcat’s memory to cause an Out of Memory (OOM) error — usually just manifesting as a confusing PermGen error in the log files.

The fix outlined on that tomcat bug report page is actually quite simple. We simply have to tell tomcat not to “pool” taglibs and magically the memory leak is gone. In <TOMCAT HOME>/conf/web.xml file, in the jsp servlet definition add the following element:

<init-param>
<param-name>enablePooling</param-name>
<param-value>false</param-value>
</init-param>

After changing this, OpenMRS has successfully been up for a full day of data entry for the first time in a few months!

If your installation of OpenMRS seems unstable or slows down after a few hours of use, I would suggest trying this fix. The memory/cpu trade-off that occurs from not having taglib pooling is definitely in your favor.

I’m beginning to feel like a paid blogger for yourkit, but I really must thank YourKit again for the their license donations. Their newest 7.0 release helped me track down the source of this latest bug with relative ease.

This is in the openmrs category tagged as memory, tomcat

3 comments »

I just finished poring over OpenMRS with a java memory profiler in an attempt to find where we’re leaking memory. It appears as though OpenMRS takes up more and more of the server’s memory as time progresses. However, my final conclusion is that we aren’t leaking much. I found and fixed what appeared to be memory bugs, but by and large, I think we just demand a lot of Java objects to be loaded at any given time — and Tomcat holds onto the total memory its been allocated just in case it needs it again.

A very huge find that came out of this was that we were “leaking” Tomcat sessions. In Eldoret we have about 30 data assistants right now that are filling in Infopath forms for about 8 hours a day. The average form takes about 5 minutes to fill out, so they are opening and closing Infopath fairly often. When opened, the taskpane on the side of Infopath is a mini IE browser. The first action taken by that taskpane browser is a stealing of the user’s session from their other web browser. The original session is left hanging around in Tomcat though. After 30 minutes of inactivity that session is dropped. Apparently, in our setup at least, either Tomcat or Apache has somewhere around a 250 limit on the number of sessions. With the DAs entering forms at a rapid pace, it would only take a few hours before the limit was reached — at which point Apache/Tomcat halted all traffic until some sessions were freed by timing out or Tomcat was restarted.

While trying to find the real cause of the problem, my first solution was to shorten the length of the session timeout. In <Tomcat Home>/conf/web.xml:

<session-config>
<session-timeout>15</session-timeout>
</session-config>

this obviously only lengthened the time between restarts.

The true solution came in the form of a beautiful one line fix in the formentry module: when initially loading the taskpane (before the session stealing happens) set the timeout of the current session to 10 seconds. Voila! Success! Our Eldoret server is now much more “dependable”!

Now, as far as the memory leak goes, I’d love to have some input from people who might have a better idea of what they’re doing. At times I felt as though I were just clicking randomly through the profiler looking for anything that had ‘org.openmrs…’ in the package name. If you have any pointers to give me or you have found a few memory leaks yourself, let me know!

This is in the openmrs category tagged as java, memory, tomcat

Add a comment »