Operating Systems and Internet Messenger in the USA

The Hopper Cookbook (Step-by-step instructions to improve application stability)

The intent of this document is to introduce Hopper to the application developer who is targeting applications for the Windows Mobile platform. The purpose of running Hopper against your application is to find exceptions, hangs and deadlocks that may be lurking in your application that may go unnoticed during testing and may cause device crashes during customer acceptance or field trials. This introduction will include all the necessary materials to begin Stability testing of your application using Hopper as well as pointers for advanced techniques.

 

Goals

The goal of this document is to educate ISV development community on Hopper so they can improve application quality and stability.  Developing with Hopper, ISV’s will reduce customer acceptance time as well as find more application bugs that will contribute to the overall quality of their application. The expected steps from the ISV community are:

 

1. Consistently run Hopper during the development cycle and sort the bugs into three categories:
   1. Category #1: ISV application dependent bugs (to be fixed by the ISV)
   2. Category #2: OEM device dependent bugs (to be reported to the respective OEM for fixes, if applicable)
   3. Category #3: WM dependent bugs (to be reported to Microsoft for evaluation and possible fixes)
2. Coordinate with both the respective OEM and Microsoft (if applicable) to secure bug resolutions and proper application stability
3. Run Hopper during WM device development by coordinating with their respective OEMs (if applicable)

 

The ISV development community following the above process will enjoy improved application quality and stability. Hopper takes the guesswork out of quality measurement, thus reducing application time to market.

 

Prerequisites for this document

This document builds upon existing materials published for the OEM and the ISV development community. It is highly recommended that the reader review the following topics before reading this document:

 

1. Hopper for ISV’s – part I (http://blogs.msdn.com/hopperx/archive/2006/06/05/618018.aspx)
2. The Cat Parade (http://blogs.msdn.com/hopperx/archive/2005/11/30/498113.aspx)
3. Improving the Cat Parade (http://blogs.msdn.com/hopperx/archive/2006/11/21/improving-the-cat-parade-part-1.aspx)

 

The Hopper Doctor blog, which contains the above materials is updated frequently and often contains ISV-specific information, so it is recommended that ISVs check the main blog page (http://blogs.msdn.com/hoppeRx/) frequently.  This document also assumes the reader has Visual Studio 2005 and the Windows Mobile 6 SDK installed.

  

Review – and overview of the Hopper system

As mentioned in the Hopper for ISV’s blog, Hopper is a stability test tool that randomly sends keystrokes and screen taps to the device. Hopper is designed to stress the entire device and not remain in a single application.  However, application developers are not interested in stability of the entire device– they want to locate and fix bugs only in THEIR application. To achieve this, ISVs need a way to focus Hopper on their application and to minimize the time Hopper spends away from their application.

 

 

To successfully perform stability testing with hopper, three elements are needed:

 

 

The Hopper component is available under your SDK tools directory. A sample focusApp.exe is provided in the same Hopper tools directory and will need to be customized for each application you are testing. To begin testing, you need to perform the following steps:

 

1. Using Visual Studio, deploy your application to a device or the device emulator and attach the debugger. This will enable you to debug any exceptions or hangs that your application encounters. Step-by-step instructions on how to do this are located below in the section labeled Demo.

 

2. Customize the focusApp.cpp source to keep your application in the foreground. Refer to “The Cat Parade” blog entry for more information as well as the end of this document for more ideas. Once customized, this tool needs to be deployed to device and your application

 

3. Deploy and run Hopper while your application is running. The focusApp will force Hopper to remain “focused” on your application and avoid the rest of the system.

  

By combining these three elements, we are able to remove platform stability from the equation and allow Hopper’s effort to be focused on your application and away from other potential problems. For example, if you are running this test on a device that has known stability issues, it is unlikely that Hopper will encounter those issue because your application is processing all the key strokes. This is very convenient since it is the application bugs that you are interested in, not the bugs elsewhere in the system.

 

Demo - Watch Hopper in action and find real bugs!

Below represents a step-by-step procedure for running Hopper on a sample application that is included in the SDK. Ultimately you will want to proceed through the following steps using the application you are developing and not CECamera.

 

Step 1: Launch Visual Studio. Launch the SDK tools folder from the start menu and navigate into the Hopper Folder and then again into focusApp folder. Double click on this solution to bring up VS.

 

 

 

 

Step 2: Customize focusApp. We need to customize focusApp to point to the application under test. For this sample, open the file FocusApp.cpp and replace the “\\Windows\\WMPlayer.exe” with “\\Program Files\\CECamera\\CECamera.exe”. Rebuild the solution. When testing your own application, you will replace the CECamera  path and filename with the path and filename of your application

 

 

Once this application has been rebuilt – you can close this solution.

 

 

Step 3: Launch the CECamera solution provided under the Samples folder – this is the application we will torture under hopper.

 

 

Navigate to the CECamera directory under the following directory: Samples\Common\CPP\Win32\CECamera – double click on this CECamera solution to re-launch Visual Studio with this solution active.

 

 

Step 4: Add Hopper dependencies to your deployment.

Change the Project properties to automatically deploy your focsuApp.exe and Hopper.exe. Under the menus Project->Properties, select Configuration Properties -> Deployment and add the following lines to the existing:

 

focusApp.exe|C:\Program Files\Windows Mobile 6 SDK\tools\hopper\focusApp\FocusApp\$(OutDir)||\|0

hopper.exe|C:\Program Files\Windows Mobile 6 SDK\tools\hopper||\|0

 

 

Please note that your Path locations may be different from the example, and care must be taken to copy & paste the correct paths. When this is complete, your dialog should look similar to:

 

 

Test your deployment. Using the menu action Build -> Deploy solution, check the results of your deployment, you should see:

 

 

 

1>------ Deploy started: Project: CECamera, Configuration: Debug Windows Mobile 6 Professional SDK (ARMV4I) ------

1>C:\Program Files\Microsoft Visual Studio 8\VC\ce\dll\ARMV4I\msvcr80.dll

1>C:\Program Files\Microsoft Visual Studio 8\VC\ce\dll\ARMV4I\msvcr80d.dll

1>C:\Program Files\Windows Mobile 6 SDK\tools\hopper\focusApp\FocusApp\Windows Mobile 6 Professional SDK (ARMV4I)\Debug\focusApp.exe

1>C:\Program Files\Windows Mobile 6 SDK\tools\hopper\hopper.exe

1>c:\Program Files\Windows Mobile 6 SDK\Samples\Common\CPP\Win32\CECamera\Windows Mobile 6 Professional SDK (ARMV4I)\Debug\CECamera.exe

========== Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

========== Deploy: 1 succeeded, 0 failed, 0 skipped ==========

 

 

 

You will note that the Device Emulator will launch in the background and fully boot before copies can begin. If there are any errors during deployment, please check your pathnames above to make sure you are pointing to the correct locations. You must be able to copy & paste the full paths and reference the binaries. Please correct any mistakes before continuing.

 

 

Step 5: Close off exit points.

If your application responds to WM_CLOSE to exit, we need to ignore this command so Hopper does not close the application. CECamera sample contains both a menu item and an “X” box that will sent a WM_CLOSE to the app – so we need to comment out this code. Locate the CECamera.cpp file and search for WM_CLOSE like below:

 

 

 

…

                case IDCANCEL:

                    // Fall through

 

                case IDM_EXIT:

                    // PostMessage(hwndDlg, WM_CLOSE, 0, 0);   ** Comment out this line **

                    break;

 …

 

 

 

Step 6: Start debugging the CECamera.exe application.

 

Before we begin debugging we need to verify the debugger will detect and break when an exception is hit. Menu Debug -> Exceptions will bring up the following dialog – make sure all thrown check boxes are checked and your settings saved.

 

Next, Select the menu Debug -> Start Debugging to deploy and debug CECamera. This will launch the application under test and if you switch back to Device Emulator, you will be presented with the sample application.

 

 

Step 7: Launch the Hopper and focusApp.       In this step, we need to access the File Explorer so we can launch focusApp and then Hopper.   Click on the Start Menu, select Programs and then File Explorer.
Navigate UP to the root directory and select focusApp.exe – this will launch focusApp.exe which will quickly bring CECamera back to the foreground.
Bring the File Explorer back by clicking the Start Menu -> File Explorer. Use this opportunity to launch Hopper.exe.   If you wait too long, focusApp will bring the CECamera application back to the foreground and obscure File Explorer. Simply Bring the File Explorer back by clicking the Start Menu -> File Explorer and click on Hopper.exe. This tight-timing is expected and part of the design.   In a moment, Hopper will take over your system and begin pressing keys and tapping on the screen. Once Hopper encounters an exception in your application, Visual Studio will break and you will be notified.

 

How to debug breaks

Having Hopper focus on your application is only useful if it finds exceptions, hangs or deadlocks. Hopper’s job is to identify these bugs and it is left to the developer to actually find and fix the root cause for any break The Hopper Doctor Blog (http://blogs.msdn.com/hoppeRx/) is a great resource for how to debugging exceptions, hangs and deadlocks found by hopper.

 

Since Visual Studio is an application debugger you will notice that Hopper will continue stressing the device after your application has broken into the debugger. You will also note that focusApp.exe continues to run and will continue trying to bring your application to the foreground.

 

This is normal and should not interfere with debugging the break that stopped Visual Studio.

Published Wednesday, June 06, 2007 5:53 PM by shende

Filed under: ISV focusApp

Comments

While at MEDC I had the chance to present a new addition to the Windows Mobile APIs: WISP Lite

WISP stands for Windows Ink Service Platform and it is the technology for ink recoginition used in other platforms like Tablet PC. In the Windows Mobile 6 SDK Help, it is referred to as Windows Mobile Ink. Even though WISP Lit is just a subset of WISP, it aligns Windows Mobile with the rest of Microsoft. WISP Lite will eventually replace our current rick ink control.

I played with it a little bit, and found out that adding simple ink recognition was not as hard as I expected.

These are the steps that allow you to recognize ink on a given window.

1.       Create a Window where input will be recognized as ink. Something like the lines below would do the job. Nothing special about the window, except it is big enough for someone to scribble something.

   // Create the signature window

    hWndInk = CreateWindow( TEXT("static"),

                            NULL,

                            WS_VISIBLE | WS_CHILD | WS_BORDER,

                            left,

                            top + 20,

                            215,

                            150,

                            hWnd,

                            NULL,

                            hInstance,

                            NULL);

2.  Attach an InkOverlay object to that window and enable it.

    hr = ::CoCreateInstance(CLSID_InkOverlay,

                            NULL,

                            CLSCTX_INPROC_SERVER,

                            IID_IInkOverlay,

                            (void **)&g_pInkOverlay);

    ASSERT(SUCCEEDED(hr));

    hr = g_pInkOverlay->put_hWnd((long)hWndInk);

    ASSERT(SUCCEEDED(hr));

    hr = g_pInkOverlay->put_Enabled(VARIANT_TRUE);

    ASSERT(SUCCEEDED(hr));

3.       Let the user write something on your window.

4.       When you are ready to recognize ink: Get the strokes (or dots recognized) and convert the strokes to a string.

        g_pInkOverlay->get_Ink( &pInk );

        // Get all the strokes in the ink object

        pInk->get_Strokes(&pStrokes);

        // Get the recognition result for these strokes

        pStrokes->ToString(&resultString);

5.       And voila! resultString now contains the recognized result.

A few samples that use WISP are shipped with the Windows Mobile 6 SDK.  On my machine they installed to \Program Files\Windows Mobile 6 SDK\Samples\PocketPC\CPP\win32\WISPLite

Luis Cabrera

An old blog entry continues to get requests for us to allow users to move their scroll bars to the left side of the screen.  I’ve forwarded the suggestions on to the appropriate people, but, since I’m not on that team, I don’t actually know what will happen with the feature.  That said, I figured this would be a good example to use in describing a huge part of software development: Feature Prioritization.

Fun vs Reality
To most engineers, the fun part is making stuff.  The reality of our situation, though, is that we’re more constrained by what we can’t do than by what we can.  Outside of software engineering, this is inherently obvious.  No, these civil engineers aren’t going to make a building that’s a million feet tall.  No, these automotive engineers aren’t going to make a car that gets a thousand miles per gallon.  Boeing’s customers don’t go to their blog and say, “Why don’t you make a passenger plane that can fly across the Atlantic in ten minutes?”

I’m sure that most engineers don’t spend too much time thinking about this.  When the answer to the question is, “It’s impossible with current technology,” there usually isn’t much room for debate.  And, even when new technology is invented, people have been trained to expect it to take time to become available.  You see some new technology in a concept car at an auto show and you know that it’ll take five to ten years before you’re able to buy a car with that technology in it.  You also rarely expect updates in the things that most engineers build.  “This house I bought was made of wood.  But now they’ve got better composite materials.  The builder should upgrade my house for me.”

On its surface, software engineering seems like it’s different.  It’s rare that we software engineers get to use the, “I canna change the laws of physics, Captain!” excuse that our peers in other engineering fields have.  Is there anything that makes it impossible to put the scroll bar on the left side?  Nope.  And, we’ve trained consumers to expect changes to take months, not years.  Releasing seven major versions of Windows Mobile in the last seven years is kind of a double edged sword for us.  On one hand, we can say, “Look at how much we’ve done.”  On the other, you can say, “So why haven’t you done what I want yet?”  And we software guys have trained you to expect upgrades.  “XP SP2 was free.  Why isn’t this?”

Software Constraints
Despite not being encumbered by many of the laws of physics, software engineering does have constraints.  The trouble is, most of these constraints aren’t very obvious to most people.  You all know that it’s hard to change a road, because you’ve witnessed it happening.  You saw it take a lot of time.  You saw a lot of people working on it.  You probably swore under your breath at the traffic mess the work was causing.  But even that congestion contributed to your understanding that it’s a big effort.

Making software that will be used by millions of people also takes a big effort.  It’s just that few of you have seen this effort firsthand.  And it’s not even apparent when seen after the fact.  For example, in using one of our devices, it’s probably not obvious that any text you see needed to be translated into 24 languages.  But that effort is part of the fun stuff.  Making things is what we do.  The real challenge isn’t in doing the features.  The real challenge is in deciding which features to do.

Ironically, our largest constraint is a direct result of our being able to do just about anything.  We can do anything, but we can’t do everything.  Though the laws of space usually don’t apply to us, the laws of time do.  And with only so much time, we can only do so much.  Worse, unlike the automotive engineer who can just say, “Well, it’s impossible to make a car that can fly to the moon,” there’s little that naturally narrows down our list of possibilities.

Feature Prioritization
With an infinite set of features possible and finite time to do them in, we have to spend a lot of effort deciding which features to do and which not to do.  Lacking natural constraints, we’ve had to come up with some other way to make these decisions.  In a nutshell, our solution is to come up with a priority for each potential feature and then go through the list in priority order.  Nothing earth shattering there.  But, literally, every feature we do is a tradeoff against hundreds of others we won’t have time for.  So prioritizing them is the only possible way to ever ship anything.

Prioritization isn’t an exact science, and a lot of pieces go into it.  For instance, the more people who will benefit from the feature, the higher its priority is.  On the other side, the longer it will take to implement the feature, the lower its priority will be.  Sometimes it’s better to do five small features than one large one.  We also look at the ability for our partners to implement the feature, giving higher prioritization to things only we can do than to things an ISV or OEM could do.  We also look at the existence of other ways to accomplish the same task, giving higher priority to things that can’t be done any other way. 

On The Left Hand
Again, I’m not part of the team that would work on the “let the user move the scroll bar to the left side” feature.  I’m not part of the decision on that feature (other than to make sure that they know it’s something people have asked for).  But here are the kinds of things that will probably go into the feature prioritization for it.

First, the feature description.  With the scroll bar on the right side of the screen, it’s very hard for left handed users to use a stylus to scroll.  While doing so, their hand covers the screen, and they can’t see what they’re trying to scroll.  The feature would provide a control panel that enables the user to say that the scroll bar should be on the left side instead of the right.  The feature would also make most or all of our built in apps move the scroll bar when the user changed the setting.  Ideally, it would also work with third party apps.

Things that increase the priority:
Now, I believe that left handed people are roughly 10% of the world’s population. (Don’t send hate mail if I’ve got the number wrong.  The people who decide these things for real will make sure to get the correct number.)  We’ve got over 10 million users, so that’s potentially a million of them.  A feature that benefits a million people is pretty significant.

Aside from something really esoteric, there’s really no way for an OEM or ISV to implement this feature.  If it’s going to get done, it needs to be done by us. 

Things that decrease the priority:
Now, this feature only matters on a device with a touch screen.  I won’t give actual numbers on how many of those we sell, but for the sake of discussion let’s pretend they’re half of our devices.  That potentially cuts the number of users who would benefit from this in half. 

You’ll have to take my word for this, but the task is fairly large given the current system design.  This could only be considered in a major revision.  But, if we were to do a major reworking of the system design anyway, the cost of adding this in as well would be lessened.

From a functionality standpoint, every one of our devices has at least a Dpad, and many have a scroll wheel.  So, even if a user can’t use a stylus to scroll, it will still be possible to interact with the device. 

All together, this adds up to left hand scroll bars being a fairly compelling feature.  So it would get added to the list with more compelling ones above it and less compelling ones below it.  At that point, whether or not the feature gets implemented would depend on how much time we have for the appropriate release, and how far down the list that time allows us to go. 

The List is Unending
Most of the controversy I’ve read here has been a result of people disagreeing with our prioritization of various features.  This is the general method we use for figuring out which features to do.  Although not perfect, it’s the best scheme we’ve been able to come up with.  If you’ve got a better one, I’d love to hear about it.  
 
I don’t know if you’ll get your left handed scroll bars or not.  And, if you don’t, I don’t expect this explanation to make you any happier about it.  But I hope I’ve at least given you a better understanding of how we go about making our decisions.

Mike Calligaro