New York Media Case Study: On Demand Testing Provides Help When It's Needed

New York Media Only Pays for Testing When They Need It

Situation

When New York Media acquired MenuPages in the summer of 2008, they needed to integrate the MenuPages.com assets into their technology stack, porting Visual Basic and ASP/.NET Web applications to the Linux-hosted Python and MySQL platform at NYMag.com.  It was an extensive, ground-up rewrite effort.  Part of that effort included comprehensive software QA testing for new MenuPages applications.

Problem

With a limited number of staff available for software QA testing at New York Media, the company's resources were typically taxed when larger software testing requirements arose.  "To fit additional testing for such a large project into our existing product roadmap meant deferring testing for another product," said Albert Lee, Executive Director of IT.  "It was important that we hit our goals without taking shortcuts or adopting grassroots testing efforts."

As with most high-traffic Web applications, this project had a very low tolerance for defects.  "MenuPages hadn't been redesigned for six years," said Lee.  "It was very important that the New York Media brand was able to take a company [MenuPages] with good quality data and support it with a solid user experience."

As far as website QA testing goes, New York Media needed guidance and strategy.  "We had software requirements and use-cases but not a lot of written documentation for a formal test plan," said Lee.  "Time constraints were a huge problem.  Cost was also a concern."

Decision

New York Media would typically hire a contractor or freelancer for a website QA testing project like this.  But that approach presented challenges, as well.  "We have to validate whether the person has the right skills," explained Lee.  "Are they going to work remotely or on site?  It's a complicated challenge just to get people in house, and it doesn't always happen on our schedule.  There are recruiter placement fees and contracts.  We were anxious to avoid these things and the ramp-up time involved."

As luck would have it, one of New York Media's staff engineers had prior experience with a SQA services company, TESTCo.  The company engaged TESTCo based on his recommendation.

Solution

"With one phone call, we got TESTCo working immediately," said Lee.  "They did most of the heavy lifting, and found the right candidates to work on the project."

Lee appreciated the fact that New York Media wasn't required to jump through a lot of hoops and develop a formal scope of work for the project.  "It was nice to know that we didn't have to burn a lot of internal resource hours to educate TESTCo on our environment and requirements.  They were ready for us from the get go."

The actual testing process validated site functionality and user experience.  Things that worked on the previous site had to be reproduced in their redesigned environment.  A lot of A/B testing was used to validate functions.  "TESTCo does a lot of comparisons and user experience testing," explained Lee.  "They performed complicated cross-browser website QA testing for Firefox, Safari, Chrome and Internet Explorer."  TESTCo used New York Media's bug tracking database to manage the process.  They established a VPN connection to access the MenuPages test environment and spent three to four weeks testing the site.

"We were very happy with the results," said Lee.  "We could have gone out and hired two or threee QA testers and paid all the expenses associated with that.  But testing follows the ebbs and flows of a software development lifecycle.  There are periods of time when everyone on staff needs to be "heads down" for a couple of weeks and isn't able to integrate new people and projects into the workflow.  That's why TESTCo is so valuable - we can just take advantage of them when we need them.  It's a good way to maximize our website QA testing investment."

Benefits

The new MenuPages site launch was very successful, according to Lee.  Users quickly embraced the new features and functionality of the site, and continued to enjoy the features they had been enjoying for years.  "It was a smooth cut over," said Lee.  "We saved at least 20% to 30% in internal resource costs.  The savings were substantial - we were able to scale up a strong testing team on short notice without the typical overhead of recruiting and training.  We could rely on TESTCo for their core strengths and turn our attention to the things that we do best."

"TESTCo made On Demand Software Testing extremely simple.  We engaged them as if they were part of our staff, and they didn't need a lot of direction and definition to get started.  That's not typical."

New York Media is currently launching new On Demand Testing projects with TESTCo and exploring new ways to integrate the outsourced process into their various product development lifecycles.

Print A Copy Here (Adobe PDF)

Get Adobe Reader

About TESTCo

TESTCo customers eliminate software testing problems, slash support costs by up to 50% and achieve their release date commitments ... all without the risk of hunting down the best offshore vendor, the pain and frustration of learning how to properly manage them and the unpredictable results so common with most offshore testing vendors.

About MenuPages

MenuPages, which is owned by New York Media LLC (the parent company of New York magazine and the web site Nymag.com), is an online restaurant guide that serves restaurants and consumers.  The Web application delivers thousands of detailed restaurant listing and reviews from eight U.S. cities, including New York, San Francisco, Los Angeles, Philadelphia, Boston, Chicago, Washington, D.C., and South Florida.  The MenuPages.com site curently attracts approximately 1.5 million unique users and 15 million page views per month.

TESTCo offers On Demand Testing, On Demand Software Testing, SQA Services and Website QA.