The Comprehensive Guide to Product Lifecycle Management

‘Premature optimization is the root of all evil’. This is what Donald Knuth, author of the famous book series The Art Of Computer Programming, wrote in 1974. That does not mean however that you should not optimize for performance at all or wait until the performance problems show up in production.

In this blog article I will suggest when and how to do performance optimization in the application life cycle of Mendix applications.

WHEN TO DO PERFORMANCE OPTIMIZATION?

Let us first treat the ‘when’. There is one simple answer. Always keep performance in mind, but try to design and build the simplest solution of your application first. If the simplest version performs well you are done. Otherwise your application might serve as a prototype for analyzing performance or serve as a test to compare it against a more complex optimized application.

Trivial performance optimizations like obvious indexes and optimal query-where-clauses should always be used. For example when you know an attribute is searched against and the amount of records in the database is large you create an index. And you always build smart query statements (xPath, OQL) that do not use ‘NOT’ in where clauses unless you really have to.

If you develop a new function for your application it is advised to use the performance tools when the functionality works for the first time. Do not wait until all the other stuff like UX, security, maintenance and such are done. It is best to look at performance in parallel with those activities and base your design decisions upon the measured performance facts.

Since we all develop in an agile way, you should test agile as well. So you build up your test scripts and you run them very frequently. If you have automated your test scripts you can run them daily as a regression test and give developer very fast feedback when something is broken. Also when you measure the response times in the script you build up a history and you can do trend analysis on functions even when still in development.

During the test and acceptance sprints you let tooling run along to measure usage by test users and response times of the application. During the test you fill your database with (if needed fake) data to match the (expected) production size or you use a production database dump in later iterations.

During production you measure usage statistics. You also add monitoring scripts for selected functionality to measure behavior in a consistent way. When you run into performance issues in production you need tooling to analyze it quickly. You review the issue and hope it does not lead to a complete redesign of your application. In future blog article I will go into more detail how to approach such issues.

HOW TO DO PERFORMANCE OPTIMIZATION?

Next, let us get into the ‘how’. To analyze performance you need to know what is happening in the application internally (under the hood). You need to be able to link business model items from the Mendix modeler to execution steps in the java runtime, so you know which steps consume a certain amount of time.

Off course performance issues are not limited to the Mendix application or database. It is important to look at the application infrastructure as a whole. This includes the browser, the network, the web server, the Mendix business server, the database and the supporting operating systems and java virtual machines.

The following picture gives a comprehensive overview of the ‘how’:

This overview assumes an issue is to be investigated, but you can read ‘new function’ where is says ‘issue’. You first measure the basic responses and see where you need to look further. I will introduce the tools I use in a moment.

After the first step you need to decide where to look further. There is always a part that takes the most time, but if that amount of time is acceptable you can (and often should) decide to stop analyzing and certainly not fall into the trap of pre-mature optimization.

The final steps are the optimizations steps. Once you know where to look for the issue that contributes to a degraded performance of your application, you can zoom into that component. The how to fix performance issues is a whole other topic of a future blog.

Let me now share with you some of the tools I use. The first tool is the developer tool of your browsers (built in or plugged in). These tools allow you to measure and debug just about everything on the client-side. The following screenshot shows the network tab of the Google Chrome Developer tools when logging into a Mendix application:

As seen above the network timelines are an easy indication of where to look further. Waiting usually means the server is busy. Content download time can be changed by using compression on a web server or by reducing the amount of content (by selecting less rows or columns in a grid for example).

To look further into the time spent on a Mendix Business Server and see what is happening I use CLEVR’ Performance Tool from the APM Tools Suite. With this tool you can drill down into microflows, actions, loops and even see the SQL statements that Mendix generates. In the screenshots below you will see a sequence of steps:

• We first show you the microflow as called when pressing a run report button in our application
• Next we drill down into:
  • The loop action where you can select an iteration
  • The ‘loop-microflow’, which is the inside of a loop in the Mendix Modeler
• Finally we drill down some levels deeper until we reach an action that generates a SQL statement

Screenshot of the microflow as called when pressing a run report button:

Then we drill down to the loop-action:

Subsequently we drill down to the loop-microflow, i.e. the inside of the loop in the Modeler:

We drill down further until you reach something you can optimize, for example a database retrieve statement:

The performance tool of the APM suite of tools perfectly helps you to pinpoint where the issue is in your application code.

The performance tool is capable of running in production. If needed with all kinds of safety measures to ensure that you don’t claim too much of the server’s resources. This contributes to the just-in-time part of the title. Sometimes performance issues only appear in production and production is the only place where you can gather information. The performance tool is designed to help you with this.

Yet, another tool I use, is the Statistics tool in the APM Tools Suite. At CLEVR we let this tool run at every customer in production to collect usage information all the time. This tool can be used to get a first impression of which microflows take the most time. Also this tool can point at frequent running microflows that all together consume a lot of time and where a small improvement can thus have a large effect.

The data collected by the statistics tool is also used to make trend analysis reports and see if microflows are slowing down over time because the dataset is growing, or if microflows are slowing down suddenly, maybe due to a new deployment.

The bottom line is that you need good tools to do your job. I’ve created some tools for myself to help me to do my job and CLEVR has bundled them in the APM Tools suite to benefit you. If you want to know read more about the APM Tools suite please go to this page.

When a customer calls the helpdesk they want help and they want it fast, right? But do they know what happened that caused their problem? And is the right information logged? Or does your Mendix application need a Flight data recorder?

The helpdesk needs to understand a problem to the level that either the solution can be given to the customer or the helpdesk can proceed looking into the problem without the customer on the phone. To get to that point the helpdesk wants to collect information (what steps did the customer take, log files), to reproduce the problem and to have access to an environment where they can play around and eventually to test the solution.

When looking at the CLEVR helpdesk for Mendix Apps I noticed two patterns. The first pattern I call: ‘show me the money’ and handles the ‘can you turn on the detailed logging’ question. The second pattern I call: ‘thinking inside the box’ and is about the ‘can you send me a database dump’ or ‘can I get access to the system’ questions the helpdesk often has to ask.

SHOW ME THE MONEY

When a customer approaches the helpdesk the first step of the helpdesk agent is focused on discovering the actual problem. Furthermore, the helpdesk agent need to know what actions the end user performed so the helpdesk can reproduce the problem. The first trouble-shooting question is always about explaining the problem to enable reproduction of the problem with a focus to actually catch the problem.

If the problem can be reproduced the helpdesk can start working on fixing it.

However, reproducing the problem is often very time consuming, or the problem cannot be reproduced at all. In addition to reproduce the problem, the next usual response to a customer problem is to look at the log files. These log files sometimes provide more technical error messages but often they do not tell you what happened, only what broke.

So the helpdesk, in the friendliest way, asks the customer to turn on more detailed logging, lets the customer wait for the problem to happen again, and then lets the customer feed the helpdesk with more information before they can fix the problem and provide a permanent solution.

This process should be more effective and less time consuming. Would it not be nice to have something similar to a Flight data recorder on an airplane that records all vital information just before the crash?

THINKING INSIDE THE BOX

A helpdesk person needs to be very good at predicting what the customer saw or which actions the customer performed. Over the phone (or mail or chat or twitter or whatever communication channel is used) the helpdesk cannot see the customer’s desk or the inside of the servers. If the software and the computers are not complex enough on their own, the helpdesk also lacks the proximity and permissions to see and touch the stuff themselves. See it as a kind of a black box around a black box. The helpdesk person needs to think inside the box he or she can’t see.

The first attempt is often to imagine what happened and then to try and see if this will happen on a local (lab) system at the helpdesk as well. If this works the helpdesk can start fixing and does not need further help from the customer for this. More often however extra information from the customer is needed.

When the helpdesk can’t imagine what happened a series of actions often happen:

• The helpdesk asks the customer for extra logging and often the customer has to ask their IT guys to deliver this
• The helpdesk asks the customer to try things and describe what happens
• The helpdesks asks for a database dump to build a situation in the lab that resembles the customer situation as much as possible
• The helpdesks visits the customer to see for themselves or asks the customer for remote access

This involves a lot of activities where the helpdesk asks the customer to deliver information. And often the customer has to ask additional IT guys to deliver the information to the helpdesk. So I started thinking what can be done to speed up things and lessen the amount of actions to be taken by the helpdesk as well as the customer. There are several options to help the helpdesk:

• Prepare helpdesk support facilities up front. Expect the logging and database dump questions and use a fast procedure to answer those questions.
• Be even more pro-active and give the helpdesk access to a database dump weekly or monthly and give the helpdesk access to the logging or to the server remotely. This is similar to the access the CLEVR helpdesk has managing a Mendix cloud node. For on premise nodes similar access needs to be arranged as well.
• Give the helpdesk extra tooling to pinpoint the problem and prevent a lot of questions.

This last option needs some more explanation. If the helpdesk does not know what the problem is they will perform an analysis of different components: The database, the server or the network and so on, and the helpdesk will ask information or logging from all those components. So I would like to give the helpdesk direct access to the information of these different components. If they can see the database log, check if the network is up and see the server stats the helpdesk can decide for themselves where to look first and whether or not they need the extra information of other logs at all.

THE SOLUTION

The options to help the helpdesk can be supported with available monitoring and logging technologies. For example remote access tools or generic APM solutions for log reading and system stats. However, often due to security restrictions the helpdesk does not get access to the components they need for analysis.

And remember, we all wanted that Flight data recorder that records all relevant information just before the crash. I have developed some Mendix specific (Flight data recorder) tools that assist the helpdesk to do a better job in diagnosing problems:

• The Trap tool that captures all information right before an error like your Flight data recorder
• The Log tool to log long term data
• The Log rerouting (tool) to capture extra detailed information

The JVM Browser (tool) to look at the java and system component from inside the application.

The next part of this blog describes how the trap tool, log tool, log rerouting and JVM browser help the helpdesk to perform a better job.

THE TRAP TOOL (FLIGHT DATA RECORDER)

The Mendix business server, additional libraries and custom java code generate logging. This logging can be of several levels being critical, error, warning, info, debug or trace. The last two levels, debug and trace, are especially interesting when the helpdesk is troubleshooting. However the amount of messages with level debug and trace is very high. The risk is that logging all levels all the time would overflow the system, or would let the system run out of disk space. It could make the system very slow, because resources would be used to write debug and trace messages to the disk or database.

The dilemma: the helpdesk wants the debug and trace information without hampering the system too much. The answer to this is the Trap tool. This tool is listening to all levels of logging and keeping them in memory so the performance of the system is not negatively impacted by the desired logging level. Now to prevent running out of memory, the trap tool only keeps the lasts seconds in memory. So when an error message comes along the Trap tool has the last seconds before the error in memory and saves the log messages of those final seconds to the database. Does this sound like your airplane’s black box Flight data recorder or not?

THE LOG TOOL

You might ask why the helpdesk needs the log tool if they already have a Trap tool. The answer to that question is simple. There are situations where no error appears in the log and still your application doesn’t function properly.

Another answer might be that the helpdesk wants a specific log node to log for a longer period of time. The Mendix business server consists of many parts that write logging tagged as different log nodes. A log node is an indication of the source of the message. So it might be acceptable for the systems resources to select certain log nodes to log at debug or trace level for a limited amount of time.

Yet another very good reason is that the Log tool is available to the helpdesk. The helpdesk or a technical application manager can change log levels themselves. By using this tool relevant log information is available a lot faster.

LOG REROUTING

There is logging generated by non-Mendix libraries that do not end up in the Mendix logging. Java mail is an example.

The log rerouting tool listens to other logging systems and sends the log messages of those systems to the Mendix logging. This makes those messages available in the log tool and trap tool.

JVM BROWSER

Java has built-in management extensions called JMX. Those extensions can be used to get information from the JVM. This information includes startup parameters, version information, memory and CPU usage and a lot more.

This information is normally available to java specific tools. By making this information available to the helpdesk you give them the right tool to see in which direction they should look further.

SUMMARY

In this blog I have tried to explain some come helpdesk patterns that focus on reproducibility and information collection. These are the first stages of solving issues and they were in the past time consuming activities.

By means of giving access to information to the helpdesk and by means of providing the right tools (something similar to an airplane’s black box flight data recorder that catches the last seconds of logging) you can enable your helpdesk to do their job better and faster and hopefully solve much more problems in a shorter time.

A helpdesk that uses the APM suite of Tools, is a helpdesk that can help themselves: the expected outcome is that the helpdesk can help the customer faster and better, and that what it was all about in the first place.

Do you want to learn more? Check out the first blog from Bart on application performance management.

For optimal user exercise the responsiveness of your application is vital. When you click on a button you want an immediate response… Mendix’s solution “the progress bar” is way too much in your face when you need to add it to many button.

It is time to find something more subtile for this.

SO WHAT’S THE ALTERNATIVE?

WELL… VERY NICE WIDGET! NO, IT’S NOT!

With a bit of styling and standard Mendix features you can do this.

• Create an action button
• Set a nice icon, like the Glyphicon refresh
• Set the new option “Disabled during action”
• Add the class “has-spinner”

That is the Mendix side, Next we need to set the css styling.

• Show the refresh icon when the button is disabled
• Animate the rotation of the refresh icon
• Smooth transition for showing the spinner and increasing width of the button.
• Show the progress pointer when hovering over the button.

Is it perfect?
No; it works on the disabled attribute of the button. Unfortunately Mendix does not add a special class to a button while it is performing an action. So, don’t used it with button you like to disable based on an attribute.

THE CSS CODE:

.has-spinner .glyphicon {
/* initial, do not display*/
display: inline-block;
opacity: 0;
width: 0;
-webkit-transition: opacity 0.25s, width 0.25s;
-moz-transition: opacity 0.25s, width 0.25s;
-o-transition: opacity 0.25s, width 0.25s;
transition: opacity 0.25s, width 0.25s;
/* delay animation, for fast responses*/
-webkit-transition-delay: 0.8s; /* Safari */
transition-delay: 0.8s;
}

.has-spinner:disabled {
/* show the progress pointer */
pointer-events: auto;
cursor: progress;
}
.has-spinner:disabled .glyphicon {
/* show the spinner */
opacity: 1;
width: auto;
/* This doesn't work, just fix for unknown width elements */
/* make the spinner spin*/
-moz-animation: spin 2s infinite linear;
-o-animation: spin 2s infinite linear;
-webkit-animation: spin 2s infinite linear;
animation: spin 2s infinite linear;
}
/* set width for different btn sizes */
.has-spinner.btn-mini:disabled .glyphicon-refresh {
width: 10px;
}
.has-spinner.btn-small:disabled .glyphicon-refresh {
width: 13px;
}
.has-spinner.btn:disabled .glyphicon-refresh {
width: 16px;
}
.has-spinner.btn-large:disabled .glyphicon-refresh {
width: 19px;
}
/* create the animation */
@-moz-keyframes spin {
0% { -moz-transform: rotate(0deg)}
100% {-moz-transform: rotate(359deg)}
}
@-webkit-keyframes spin {
0% {-webkit-transform: rotate(0deg)}
100% {-webkit-transform: rotate(359deg)}
}
@-o-keyframes spin {
0% {-o-transform: rotate(0deg)}
100% {-o-transform: rotate(359deg)}
}
@-ms-keyframes spin {
0% {-ms-transform: rotate(0deg) }
100% { -ms-transform: rotate(359deg)}
}
@keyframes spin {
0% { transform: rotate(0deg)}
100% { transform: rotate(359deg)}
}

Cheers and have fun! Andries Smit.