The Evolution of Testing at StudioNorth

The art of testing at StudioNorth has come a long way since I first joined the team back in 2001. At that time, most of our site testing was primitive at best, conducted at the browser level using Netscape and Internet Explorer for PCs and Safari for Macs. Testing was primarily to ensure all of the links were leading to the right pages and that design worked consistently on every platform. For the more complex projects, we detailed the business rules and made sure the application worked according to the rules laid out.

But since 2001, the Studio has evolved, undergoing many technical changes that naturally led the testing environment to change as well.

Our first endeavor was creating a content management system that gave our clients the ability to add, edit and delete content on their own web pages. This step had extensive implications for back-end administration as it required resources to manage security, user accounts, pages, and some file pooling for sharing between the document management system (DMS) and the CMS.

As the CMS application expanded and changed, the test plan evolved as well. To accommodate, we created a generic test plan that could be applied to all clients that used the CMS. This helped with regression testing and ensured that an update in one area did not impact the application in another area.

The next Studio enhancement was the added emphasis on Flash programming. Initially, we included a .swf file in the HTML that would be isolated on a page and usually involved some slick animation that caught the user’s eyes. In order to sustain this, the testing department needed to become familiar with all aspects of Flash, including its different versions and the effects of uninstalling it in browsers that did not already have the application. Sometimes, corporations keep tight control over what version of Flash resides on the desktop. At SN, we’re prepared with contingencies because, as always, our goal is to maintain a seamless experience for all types of users, from the savvy guy running the application on Vista/IE7 to the one running Windows 98 on an IE 5.5 browser.

Over the course of seven years, perhaps the greatest challenges in our evolution have been learning from our mistakes and being able to anticipate the unexpected. For example, there will be times when a user leaves a field blank and hits the enter key. In order to prevent a system error from popping up on the screen because the programmer and tester did not anticipate that a null value would be entered, we’ve learned to test what is inputted as well as what is omitted. We’ve also learned that some users like to “test” the integrity of the system by inputting as many characters as the field allows. Thanks to this caveat, all of SN’s newbies know that “maxlengths” are a programmer’s best friend.

Java came with its own set of challenges — beginning with apostrophes that translated into question marks and limitless integer fields. Now, all input fields are routinely tested for “weird characters” and extremely large, zero and negative values to ensure that if the user does enter these into the system, the system will be able to handle it gracefully. For added security, we’ve adapted our thinking to match the naive user and the hard-core hacker. Whether a user enters the system via the login screen or tries to enter by hard-coding the security link into the URL, our job is to guarantee that the system will be safeguarded against innocent and malicious alike.

As we learn to master the defects and accept the enhancements to our department, we also like to celebrate our accomplishments here at the Studio. One of our biggest pats on the back came when we developed our proprietary Defect Tracking System. This SQL back-end web application offers the QA department and developers a chance to collaborate on projects and track the issues on a per-project basis. Because projects are not deemed ready to go live until all issues are resolved, this systems helps ensure that problems are not forgotten or lost in the shuffle.

In the growth process, we not only learn from our mistakes, but we take recommendations from our most important critics — our clients. We enjoyed a great leap in quality when we took the recommendation of a client to put our system under a “stress test” to ensure that it would operate smoothly with 100 users hitting it at the same time. Since we don’t even have 100 computers here at the Studio, we had to look into a stress tool that could simulate 100 users accessing the system. After researching other tools on the market, we opted for an open source application to run our test against. This tool gives us the unique ability to go into the code and make the necessary adjustments in order to simulate 100 different users logging into the system via a login screen. Most stress tools do not allow for such adjustments; rather, all 100 simulated users would be using the same login — which of course would not be a true test of the system’s capabilities.

We were able to run the tests after business hours and report on the number of errors encountered, the server thresholds, and the number of requests processed in a second. Once our client saw the reports, they were more than satisfied that their application would run without incident. We’re thrilled to report that it did, in fact, run flawlessly when the application opened to audiences a week later.

Going forward, QA is looking for ways to improve the productivity and accuracy of all of the applications we develop. With Andy at the helm pushing our developers to be on the cutting edge of technology, it’s never boring or stagnant in our area. We realize that when collaboration happens between departments, the needs of the client are better met and managed — that’s what makes StudioNorth an integrated brand communications agency. Currently, we are reviewing a new online defect tracking/project management tool that will allow us to align projects with any issues associated with the program. Because it’s accessible at the project management level, the application will be open to viewing and editing by both our Client Services and Interactive departments.

We’re also looking into automated testing tools to aid us with regression testing. Regression testing involves testing the application to make sure that what was working before is still working after the updates are applied. Regression testing tools save time and money, and ensure that the application behaves as expected.

To ensure that our evolution continues, we make it a priority to be up to date on the latest technologies, hot trends and current forecasts. We receive QA feeds in our email accounts on a daily basis, order books and videos, and attend training classes to make us the best testers that we can possibly be.

Laura Zavala

Laura leads the quality assurance team in the testing of applications at StudioNorth. Before any program is released, she subjects all applications to rigorous testing in such areas as browser compatibility, operating systems, business rules and requirements, stress testing (performance), link checking, and overall usability. Laura has been with the Studio since 2001. With more than 15 years of experience in programming and project management, she is well versed in testing methodologies as they relate to Web, Flash, desktop, and extranet applications.