Journyx Case Study: Improved Software QA Testing with TESTCo

Journyx Quickly Tests New Software and Improves Software QA Testing on Multiple Platforms with TESTCo

When Journyx originally released their marquis software solution, Journyx Timesheet, features were added week-in and week-out without any serious testing considerations. “It was just a great big mess,” said Meredith Zachritz, product manager at Journyx. “At that point, there was no testing aside from the customers in the field. Our developers would add features and do some minor software QA testing on their own perhaps, or perhaps not, and they might not even document what they had done.”

“Everything was really in the wild at that point,” continued Zachritz. “We fixed things as we found them.”

The company tried to build a product management process, but things got even more complex as the number of supported platforms increased and software regression testing demands grew. Eventually they couldn’t keep up with product roadmap deadlines.

Challenge

At first, Journyx tried to stem the flow of bad feature additions to their products. They had acquired the habit of adding ‘customer-request’ features without understanding how those features would impact the broad user base. At the same time they were falling behind on their testing.

Software regression testing was taking longer than new feature development, and this would slow new version launches and patch distributions. Customers were also finding high severity defects caused by major, last minute changes in the regression test cycle.

Early on, the developers handled testing. The support department would help out, too, but all software QA testing was ad-hoc and could not be reproduced. Then the company attempted to document their software QA testing procedures. An expert was hired for the express purpose of developing some manual test scripts for them.

“We approached it in a piecemeal fashion,” said Zachritz. “The project manager was responsible for a bunch of other things, so testing wasn’t given the attention it deserved. At one point we brought in some of our children and paid them to run through the scripts and bang on the keyboards. Those scripts were pretty basic.”

Journyx hired a dedicated staff member to handle software QA testing, but, since their process wasn’t very thorough or well documented, they continued to have difficulty getting the results they wanted.

“We had difficulty supporting the budget for it, as well,” said Zachritz. “When times tightened up, that person was one of the first people to get laid off.” Eventually, the company decided to go with outsourced software testing and the project was sent to a group in India.

“We ran into more problems with outsourced software testing,” said Zachritz. “There wasn't any local management, and there wasn't even a formal management process. We were communicating directly with the people testing the software. They didn’t bring any sort of software QA testing expertise to the table. They were kind of a glorified version of the children who ran the test scripts.”

The software testing company in India helped Journyx expand the number of test scripts they were using. “We had more coverage with the new scripts, but we didn't get any smarter,” said Zachritz. “They didn’t help us look forward and plan process improvements.”

To top it all off, there were communication issues, time zone challenges and some tricky language barriers to overcome. “The whole adventure didn't turn us off to outsourced software testing, but we did indentify our outsourcing challenges in detail. I was confident that we could find a software testing company that would overcome those challenges and offer something more.”

Solution

Fortunately, when it became obvious that outsourcing to the group in India was not working, Journyx found out about TESTCo. “TESTCo came in and immediately overwhelmed us with all the ideas they had for improving our test coverage,” said Zachritz. “They made suggestions about how to improve our software QA testing process in ways that we would never have envisioned on our own given time and resource constraints. That was the biggest overall benefit. Somebody was looking out for us, watching what we were doing, then recommending new approaches, new processes and improved software QA testing practices.”

“TESTCo has managers in the U.S.,” continued Zachritz. “That’s great, because we can now make plans and talk strategy during working hours. At the same time, there’s very tight communication between the U.S. managers and the overseas managers.”

TESTCo expanded and standardized Journyx’s software QA testing coverage. Anyone with minimal training can now run a series of highly organized tests. Testing for fault tolerance was also added to the mix. Journyx now includes TESTCo in the feature design process, as well. This allows the TESTCo test engineers to define success factors before a feature is fully designed. “That’s really helped speed up our development process,” said Zachritz.

TESTCo automated a large portion of Journyx’s test coverage. That’s important because Journyx is constantly testing software updates on a wide variety of platforms. “In terms of OS’s, we support Windows versions, several Linux versions, AIX and Solaris,” said Zachritz. “On the database side, we support Oracle, DB2, PostgreSQL, and SQL Server. There are a lot of platforms that customers could be running our software on.”

The automated TESTCo testing coverage allows Journyx to roll out new features and updates to the various platforms much more quickly. Test cases that do not require human intervention – approximately 65% of their cases – were automated. These cases are put through a continuous execution of regression tests and defect reporting against daily development builds.

“Now coverage is more complete,” continued Zachritz, “but we also get the results much earlier in the process. For example, we used to get to our AIX DB2 platform at the very end of a test cycle, and then we’d find out that something was horribly broken. That's just not the case anymore because we can hit all those different platforms very early in the process.”

Results

The new software QA testing regime delivered several quantifiable benefits. Journyx was able to cut testing cycles from 96 business days to 32 business days. And, 65% of their test cases can now be executed against all platforms in 4 business days. This provides valuable feedback for future feature development. New platforms can be added and tested (to 65% coverage) with just 12 hours added to the test cycle. The new processes allow them to test current code drops and find out if anything’s broken within 24 hours or less. In addition, with software regression testing, Journyx has found zero critical defects when they add new features.

In terms of budget, Journyx feels like they’ve dodged a lot of hidden costs by using a software testing company. “TESTCo’s services are a little more expensive than the outsourced software testing crew we used previously,” explained Zachritz. “But if you look at the value, TESTCo is a far better deal. We’ve compared the services to a full time manager and a testing team, and TESTCo’s pricing stacks up favorably.

“With TESTCo, we don't have to train new testers. We can ramp up and ramp back down, which is just huge,” continued Zachritz. There are periods, for example, when Journyx wants to add a part or full time tester for three months at a time. “TESTCo can do that at the drop of a hat.”

“It's TESTCo's problem to make sure that person's trained. We only pay for that person for the three months the tester is on the project. There's just no equivalent if you hired everybody internally. And what happens when someone you hire decides to jump ship? You’re out of luck, and you loose all the time spent on training. With TESTCo that doesn’t happen.”

The entire TESTCo experience has been highly productive for Journyx. “It's been very nearly perfect,” said Zachritz. “It's been nice and easy, just a really painless process to add them to our team.”

Print A Copy Here (Adobe PDF)

Get Adobe Reader

About TESTCo

TESTCo customers eliminate software testing problems, slash support costs by up to 50% and achieve their release date commitments ... all without the hassle of hunting down the best outsourced software testing vendor, the pain and frustration of learning how to properly manage them and the unpredictable results so common with most outsourced software testing vendors.

About Journyx

Austin, Texas based Journyx is the first company to provide Web-based time-tracking, project accounting and resource management solutions that guide customers to per-person, per-project profitability. The company was started in 1996 by a software developer who built his own time-tracking system for billing hours. The application he created eventually became Journyx Timesheet™, which was launched as a 100% Web-based application in 1997. The business has grown organically ever since. Journyx ProjectXecute™, a robust project portfolio management and resource management system, was added to the Journyx suite of products in 2009.

TESTCo, a software testing company, provides Jourynx with Software QA Testing, Software Regression Testing and Outsourced Software Testing.