At first, a complete, single solution for all your Application Lifecycle Management seems tempting and worth spending millions for. The ROI seems high because such a solution will help QA teams collaborate more efficiently, and provide a bird’s eye view of the application lifecycle to QA managers to help them make better decisions.

However, what many QA managers fail to realize is that such an ALM tool might force either their team or the development team to change their existing processes to adapt to the new tool. This is likely to introduce newer challenges that managers would then have to face. Traditionally, such all-in-one solutions were highly sought after by decision makers, because they seemed to solve all problems at once. Bundling products has been one of the successful strategies that companies have employed over the years. For instance, Microsoft successfully packaged IE with windows for many years before users realized they are being denied the freedom of choosing the browser they like. Consumers today want flexibility, and software professionals are no different. They want to be able to pick and choose the best solutions for managing different stages of application lifecycle, based on their goals and processes.

Today, a large number of application lifecycle management (ALM) tools for different stages of application lifecycle have come up in the last five years. There is a flood of requirements management, test management, test automation and defect tracking tools in the software development market. This market has so far been a green pasture for most of these tool companies to feed upon and co-exist in harmony. The key to success of so many tools is that they have been able to find their own niche, be it a particular industry, a development process, customer size or geography. Most ALM tool companies have realized that if they want to survive in this market, they need to differentiate not only on price and features; they also need to design their solutions to flexible enough to seamlessly integrate with other ALM tools.

This gives software teams a lot of options to choose from, but it also limits their visibility into the software development process. Users should be able to choose JIRA or Bugzilla for defect tracking, QMetry for test management and different tools for test automation and so on, depending upon their priorities and business goals. The condition however is that, once integrated, these best of the breed tools should be able to provide a seamless experience to the end users. ?Consolidating reports, collaborating with distributed teams and centralizing assets becomes a great challenge for teams using different tools for different aspects of ALM. But, with open web APIs and technology, most of the tools today are able to provide a deep integration with one another. ?All-in-one ALM tools could prove useful for teams that follow very standard processes and do not change them for long time frames.

However, some companies that don’t want to change in today’s world don’t survive for too long. Software teams need to quickly respond to changing technology ecosystem by adapting their processes. They need to be able to change one part of the process without affecting the other. All-in-one ALM tools do not allow this flexibility. If you want to change one part of your process, say for example, the way you manage defects, you will need to get rid of the tool itself. On the other hand, if you were using an integrated solution, you could easily plug in a different defect management tool without having to change your test management workflow.

To summarize, QA managers can greatly reduce their costs and bring efficiencies by investing in best of the breed ALM solutions for different aspects of software testing lifecycle. All they need is to carefully evaluate the extent to which these solutions can integrate with other ALM tools to provide a seamless experience.

Recent economic downturn and several reports of irregularities in financial world have put financial institutions like banks and credit unions under the public scanner. These institutions now not only need to fix various loopholes in the system, but also need to provide a complete transparency into their processes.

The compliance review and auditing of different aspects of the IT processes are driven by different internal governance bodies like Risk Assessment and Control, Operation Control, Business Disaster Recovery Team, as well as external bodies like SOX Compliance.

To restore customer trust, financial transactions should comply with industry requirements and financial IT systems should adhere to highest quality standards. A compromise on quality is now unacceptable in this industry. For this reason, companies are heavily investing into testing their IT applications, which form the backbone of their business. They want to ensure that they cover all aspects of IT systems to minimize their risk if something were to go wrong. However, it is not always easy to prove that all necessary checks and tests were performed because companies do not yet understand the importance of retaining test evidence.

To confirm and authenticate the actual testing effort, it is important that the test documentation is retained as a proof. Test documentation includes the test plan, test results and test evidence. All test results are to be recorded in a central repository system along with supporting test plans and evidence.

Quality assurance is no longer just about meeting certain quality criteria and ensuring that the testing team finds a minimum number of bugs. It has now become one of the mainstream activities for businesses as they realize the impact of quality on business. With increasing emphasis on testing, globalization, advancements in technology and complexity of software ecosystem, a systematic approach to manage software testing process has become crucial. At any point in time, software quality assurance (SQA) teams need to have answers to following questions:

• Where am I at?
• Where am I going?
• What risks am I taking?
• Am I doing enough or am I doing too much?

Real time visibility into software testing process is necessary for evaluating test progress and deciding on next steps. Most organizations use various application lifecycle management (ALM) products like defect trackers, requirements management and test automation tools. This means that generating consolidated reports and metrics could be a great challenge, which could hinder a QA manager’s ability to get a bird’s eye view.

Demand for reduced time to market has put increased pressure on QA teams, which means not everything can be tested. QA managers need to prioritize and delegate.They should be fully aware of the risks they are taking, so that impact on quality can be correctly evaluated and communicated to other stakeholders.

To systematically tackle the process of software testing, a well documented test plan becomes an important element of test management process. This gives managers and testers a visibility into the direction their software testing process is headed to and they can plan for resources and activities accordingly. This can help optimize the overall software testing process.

In our next blog, we’ll be discussing about the essential features and selection criteria for test management tools. Stay tuned.

QMetry team announces its participation at the 10th Annual International Software Testing Conference ? STC 2010 on November 21st and 22nd.

November 21st to November 22nd
The Chancery Pavilion #135, Residency Road, Bangalore

QMetry team is leveraging this conglomeration to meet delegates from the Testing fraternity worldwide and establish business relationships. You can reach out to the QMetry team for more information on this widely used Total Test Management Tool, request a briefing session, free evaluation and also book their time to invite them for an Onsite Demonstration for the QA team back at your organization.

If you would like to learn more about QMetry or reach out to the team before the conference, please contact Rupal Der at 408-833-6187 or email me on rupal.der@qmetry.com

About STC:

QAI’s Annual International Software Testing Conference has been a highly popular offering that brings together practitioners and thought leaders from the Software Industry, Academia and Government for sharing and exchange of experiences, ideas and learning. This year, the theme of the Conference is Testing 3.0 ? From Transaction to Transformation.

Source: www.Infostretch.com

With the shrinking development cycles, pressure of reducing the time to go-to-market and repetitive processes, automation is no longer optional. Also, there is no relaxation on the quality of the final products; companies just can’t afford to launch products that are not ready for the prime time. While test management tools like QMetry, Testopia, Quality center etc. provide a great way to manage and improve the overall test process, various test automation tools help further optimize it. Therefore, it is not surprising that the demand for test automation tools has increased drastically and more so for less expensive ones, given the current economic situation.
This has given rise to a number of open source test automation tools (See the list below). These open source tools offer an inexpensive way to automate the testing process. However, we still need to evaluate them against the commercially available tools that seem to offer greater predictability and support. Tools like QTP, Rational Robot have served the test automation needs of QA teams for a long time, but there have been many challenges and constraints that users have been facing with these tools. Some issues with proprietary tools that need to be addressed are:

- These tools are highly expensive, often times outside the budget of smaller QA teams
- As the application undergoes revision over time, maintenance costs of test scripts rises sharply
- Restrictions on licensing
- Lack of support for scripting language
- Dependency on add-ons

As far as the open source testing tool market is concerned, with so many players jumping onto the bandwagon, it is a herculean task for new entrants to differentiate and find a place for themselves. The number of open source automation tools makes it hard for users to identify the best for their needs. At Infostretch, we evaluated almost all the tools out there on different parameters and discovered various strengths and weaknesses associated with them. (I’ll be publishing those findings in my next blog, stay tuned). We have been helping our clients in selecting the best tools and in developing various frameworks to address their specific needs. We spend most of our time in understanding the nature of application, business objectives, and the features that are required of the automation tools. This information is critical for selecting the right tool and developing robust frameworks.

In the later blogs on automation, I will be talking about our research findings, frameworks, testing on the cloud and open source test automation tools in general. In the meanwhile, if you have any questions or suggestions regarding any of these topics, please feel free to write to me at Durgesh.kaushik@infostretch.com.

List of Open source testing tools for functional testing (Source: http://www.opensourcetesting.org/)

Abbot Java GUI Test Framework
Anteater
Apodora
Arbiter
Autonet
AutoTestFlash
Avignon
Blerby Test Runner
Canoo WebTest
Capedit
Concordion
Crosscheck
csvdiff
CubicTest
Cucumber
DBFeeder
DbFit
DejaGnu
Dogtail
Doit: Simple Web Application Testing
Eclipse TPTP
EMOS Framework
Enterprise Web Test
Expect
expect-lite
Frankenstein
GITAK
GNU/Linux Desktop Testing Project
Harness
httest
IdMUnit
Imprimatur
ItIN – Infopath testing in .Net
ITP
ItsNat, Natural AJAX
ivalidator
Jacobie
Jameleon
jDiffChaser
Jemmy
JFunc: JUnit Functional Testing Extension
JSystem
JUnitum
jWebUnit
Latka
Linux Test Project
LogiTest
LReport
MActor
Marathon
MaxQ
miTester for SIP
Mockito
MozUnit
OLVER – Open Linux VERification
Om
Om
org.tigris.mbt
PAMIE
Pounder
pywinauto
QAliber
QAT (Quality Assurance Tests)
QMTest
Rasta
Robot Framework
Sahi
Samie
Sauce Labs: cloud-based Selenium service
Scalable Test Platform
Selenium
SharpRobo
soapui
Software Automation Framework Support
Software Testing Automation Framework (STAF)
Solex
SWAT (Simple Web Automation Toolkit)
SWTBot
SymbioTeam Lite
Systin
T-Plan Robot
tclwebtest
TestGen4Web
TestPlan
TextTest
Tomato
Toster – The Object-oriented Sofware Testing Environment
Watij
WatiN
Watir
Web Form Flooder
WebDriver
WebInject
WebTst
WET
Win32::IEAutomation
XML Test Suite

Other Resources:

http://www.slideshare.net/sebastian_bergmann/testing-with-phpunit-and-selenium

http://www.slideshare.net/hsplmkting/fitnesse-watir-and-ruby-based-test-automation-system

Requirements – the foundation of any (every) software application/ product. For years software organizations are constantly suffering from various issues related to requirements and hence trying to find better solutions and techniques to manage requirements. But, there is nothing concrete and reliable solutions or techniques/ methodologies yet that can be used as a cent percent solution for managing and maintaining requirements in better way. Ideally, it seems very little is done to mitigate this long-living problem and the scope is still limited but highly demanding. Well defined and clear to understand requirements is always one key player that can lead to fewer mistakes and other uncertain or certain errors in the later stages and ensure successful project completion.

What are requirement(s)?

Basically, requirement is a condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document. The IEEE Standard 1233-1998 focuses on 3 important parts (points) that a requirement must contain:

i) Capability

ii) Condition(s)

iii) Constraint(s)

Where, capability means ability of doing something e.g. Computer has data processing capabilities. Condition, state of something (e.g. weather conditions) or in other context existence/ appearance of any entity for something to happen (e.g. Account access can be availed on certain conditions.) and Constraint means either limitation or restrictions e.g. dropping testing activities due to time constraints.

Not just the above three points are equally vital to define and design any requirements for a target application/ product but I would like to add 2 more points and these are Clarity – means free from obscurity and easy to understand and the second one is Complete, which means perfect and up to the mark in every respect. The more clear and complete the requirements, lesser are the chances for errors and mistakes and then the derived hassles.

In other context requirements are the base which finally results in the form of application or a product. Requirements are the documented needs of a project that are gathered to identify the specific constraints (scope) of each project component and act as the foundation for everything else that occurs in a project.

Requirements – The Foundation

“Requirements” is the foundation on which any application or product is built upon and also are the pillars on which the product stands on and holds to. Hence, stronger, clearer and constant the requirements better and reliable the product is built and higher the quality achieved.

In any field if observed closely, if the requirements/ objectives are not clear and properly (precisely) defined the results are always a worry and headache in all the later stages and finally a complete failure; lets take a simple example “One cup of sugar free tea” now here, the tea to be prepared for a diabetic patient (or sugar free) or a normal person is not clearly specified (because too many meanings can be derived like i) tea using Sugar free Sugar for diabetic patient or normal person, ii) tea for a diabetic patient with normal sugar, iii) tea for normal person with normal but less sugar, and so on.); thus, it is natural that the prepared tea will contain sugar (either sugar free or normal as it is no t clear enough) which means that the final product is not as expected and this may lead to completely scrap the final product i.e., “Failure” and the reason is poor, unclear or incomplete requirements.

The above requirement may be easy and much clear to understand in the re-phrased structure as “One cup sugarless tea for diabetic patient”.

Similar simple scenario in different context, tea for 5 people, 2 cups for diabetic patients. The requirement seems too simple and may be understandable enough but still lacks clarity and completeness thus, can be re-phrased as “5 cups of tea, 2 cups sugar free for diabetic patient and 3 cups for non-diabetic people”.

Even though elaborated, above examples still may be incomplete and confusing if they do not have other relevant data specified like the type of tea i.e., Black tea/ dark (strong) tea/ Light tea or if the ingredients-wise classification is not clear like Ginger tea/ Lemon tea, etc.

The above example may not seem much technical but it explains the context either way. As stated earlier there is little done and exists that can be a vital workout for these classic chaos, the woes are also due unavailability of standard and universal tools/ systems that can be used to mitigate and overcome the requirements management concerns. There are few traditional ways that industry knows and a few select tools that can be quite helpful to the software companies but depending on the availability and the requirements of the requirements management.

What’s the current story?

Presently, till I know, excel and or word doc based requirements management is one of the widely used (accepted) and a classic way of organizing, managing and maintaining the requirements. It still takes the major share in the industry because of its simplicity and easy availability. But this is really a cumbersome job when it comes to the part of gathering these requirements to define the specifications and the higher level designing.

Now, as the software development process is gradually changing and the development techniques have been faster and more time-bound it is crucial that the requirements needs to be properly formatted and well organized and conveyed thus, managed and maintained. This may not be true enough in case of traditional excel/ word doc based methods.

Any Tool supports?

This is a critical and more or less complex question to answer with justification that is there any tool that helps in managing the requirements management but, up to greater extent the answer is yes and there are few valuable and useful tools that would really help managing these tasks, some of the software’s that are available are JAMA’s requirements managements tools and Rational’s DOORS, RaQuest and few others which are very less known or either used.

Is it wise investing in tool?

This is very crucial question that needs to be very well justified that whether the investment on a tool for just a single task is enough and worth? Will it give the intended ROI? And what if the expectations fail? Because investing in any tool is a big cost and software companies usually do not make up themselves for investing on tools. Ideally, it is a wise decision that investing in a tool will definitely be helpful and guarantee better process approach and organizing of requirements managements activities. But, the decision is of management; as cost factor that matters!

How about Test management tools?

Thinking in different way is considering the Test Management as an alternative and this is really value driven approach in order to get more from the investment made as many of them provide or possess the requirements management capabilities as well. Today, there are many test management tools available in market and many of them have their own advantages and disadvantages like QC, ClearQuest, SilkCenter, QMERTY, etc all with their unique features and functions. While, QMERTY is one of the tools that is used usually for Test management activities in many organizations with one of its extreme valuable feature of “Requirements management”.

QMETRY (http://www.qmetry.com) provides extensive capability to manage requirements and link them to testcases as well as defects for traceability analysis. By defining Requirements, you can plan and manage tests that are more focused on your specific business needs. Not only this but as requirements are also associated to tests and defects to provide complete traceability and to aid the decision-making process. QMETRY with its advanced search feature allows you to locate any requirement within seconds and see all versions. In addition, QMetry has a highly demanding and well formatted reporting mechanism which provides basic reports and allows you to generate advanced reports based off your requirements. Just select your requirements and plug away, simple yet efficient. Just as if you invest and then earn the interest with periodic bonus because QMETRY is just not a requirements management tool but a complete test management tool so the ROI is assured and highly appreciated!

Hence, managing requirements is not just easier task but needs sufficient time and proper organizing capabilities in order to manage and coordinate them. As if the foundation is laid it must be strong enough and always one grade ahead than the target so as to withstand the structure strongly and unmoved, thus building the foundation strong!

References:

· Effective Requirements Definitions and Management, White Paper. Source – Borland.

· Introduction to Requirements – The Details That Make or Break a Project. Source –Global Knowledge.

· 7 Essential Tips to Ensure Success with Requirements Management. Source – Jama Software.

· The Secret to Designing Products Customers Love – Manage Requirements Effectively. Source – Aberdeen Group.

· Requirements planning: overlooked and undervalued. Source – IBM.

· www.wikipedia.com

· www.qmetry.com

At the heart of Test Management lie organization, planning, authoring, execution, and reporting. In the contemporary times where teams are geographically distributed, operating in different time zones and functioning collaboratively – working on multiple projects & different modules, testing in tandem with various other tools, it is totally inhuman to ask the testing team to sit and draft the statistically representable and comprehendible status summary by the end of the day, for their higher ups for their perusal.

Reporting is a very vital feature of a test management tool which enables the testing team to focus on what they are best at – Testing! The gratuitous efforts spend on summarizing the status report and making them presentable can be eliminated by an efficient reporting facility provided in a test management tool.

Reporting comprises mainly of graphically summarized information as well as detailed reports which are either replicas of are Excel/word/pdf.

A good reporting system should provide:

The Dashboards: graphical summary of the statistics

• Information in an easy to use/comprehend format
• Should exhibit live/dynamically updated data
• Should show data/stats on mouse over
• Graphs/chart in relevant structure (for e.g):
  • Pie chart for distribution
  • Bar charts for comparison
  • Line graphs for timeline comparison

• Ability to view the information/metrics in the desired format – graphical/tabular
• Complete drill down to fetch the information/data lying behind the graph on a click of a button on the graph
• Ability to remember the charts most frequently visited
• Metrics that can be downloaded and distributed to the team members/managers
• Base to pull out multiple charts at one time for comparison of information across parameters i.e. timelines/components/team members working on it etc.

Reports: detailed information viewed/downloaded for reference:

• Should be viewable/downloadable in desired format (xls, rtf, pdf)
• Should provide an interface to modify/ write commands to pull out any relevant/required information from the tool for reporting purposes
• Should be easy to use and maintain

So, the bottom line is that, the reporting should be so easy that even our Managers should be able to get it and comprehend it.

At QMetry we have kept the reporting very SIMPLE to use, Easy to comprehend and Flexible to customize.

As testing is a science in itself and the activities involved within testing are always never-ending (recursive), outsourcing these activities not only reduces the burden of the organizations developing software but also guarantees a much reliable and quality software in affordable way; this reliability is achieved due to the disciplined approach and well defined/ designed test activity plans done by professional independent testing companies.

According to the Gartner Research Report, ‘Market Trends: Application Development, Worldwide, 2008-2013’, the global software testing market equals $13 billion and the global market size of outsourced QA & Testing services is around $6.1 billion. Analysts from Gartner believe that the appeal of QA & Testing outsourcing will be increased in the near future due to the companies ‘understanding that offshore services are a valuable tool in leveraging testers’ expertise, tools and processes and reducing the cost of testing, while improving the overall application quality.

Why outsourcing?

Current Issues: 

1.       Insufficient time: This is a major issue that every organization faces during entire development process of any software application. Tight deadlines in delivering the product (which need to be strictly followed), many times become a crucial player in dropping testing activity or winding it up in a shorter time.

2.       Unskilled Manpower: Another major impacting factor is the unavailability of proper and skilled resources, which usually results when testing is not planned properly and no proper testing cycle is implemented or not at all considered.

3.       Limited budget: Budget is also one of the major factors that sometimes affect the software development cycle which limits the testing cycles to one or two. Insufficient budget margins not only limits the testing activities but also limits the testing types/ phases needed based on the software application, which if bypassed may result in huge impact on the overall quality.

4.       Lack of focus consideration: Organizations which develop complete software (right from initiation till closure) are involved in many activities; various phases are to be considered during this entire process in strict time frames, thus the focus is diverse with major focus on development and completion of the tasks (which are back to back) as earlier as possible which many times deviates the focus of testing leaving it unattained. 

Benefits: 

1.       Focus on core competency: Most of the outsourcing firms are involved in to the same kind of activities i.e., testing, thus, developing a good and competent resource is not a challenge and is many times readily available.

2.       Better output due to activity outsourced to the firm specializing in the same domain: As most of the outsourcing organizations are specialized in specific domains and technologies and focus on particular fields, choosing a right outsourcing partner based on the core competencies always result in huge gains.

3.       Marginally less investment cost: As outsourcing do not involve much investment like, resource management cost, cost for planning, infrastructure, etc. it is much cheaper and cost effective and can be achieved in much less investment as compared to the investment required in deploying an in-house testing activity. Also, there is no maintenance cost required if the task is outsourced.

4.       Optimum time, task and resource management with higher productivity: The outsourcing firms with their expertise in specific domains and on particular technologies and testing activities can manage tasks and resources in much better manner by optimum time utilization as the major focus and goal is to deliver quality product with reliable and stable service to their clients.

5.       Stronger risk assessment and management: With testing becoming as major and prime task for outsourcing organizations it gives deep focus on the major impacts, potential risks and other quality hazards due to regular work and the previous experience, this helps the outsourcing firms in much better way to manage risks and plan the activities based on the risks involved and their priority. This also leads to well planned risk mitigation management and avoiding potential risks. 

Outsourcing is just not a business partnership but also collaboration for transformation towards mutual growth which, can also lead a dying/ failing business to a leading business (leader) because of its numerous hidden advantages (value-adds). Thus, in today’s competitive market moving to outsourcing is call of the hour, a wise decision!

The rapid progression in technology just highlights the fact further that test

management is no longer optional. Test Management tools are necessary as they

enable objectivity and efficiency. A proper test management tool can help address

several areas of concern:

• What to test and how does it help to address business objectives?
• Where we are in the testing cycle and what is the coverage?
• To address the change in business lifecycle, what should QA team be doing next?
• What are the risks and how are they being mitigated?
• Are we doing enough to meet customer expectations?

What is Test Automation?

 Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.

 Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection, defect logging, etc.

 Test Automation requires appropriate planning i.e. which tests we have to automate and which we have to perform manually. Also, test automation is implemented where the functionality is stable in the application over volatile ones.

 Benefits of Test Automation: 

1. Fast: Time duration of the testing process will reduce drastically since testing is a repetitive activity throughout the development cycle.
2. Reliable: Accuracy of the testing will be high in automation testing than manual testing.
3. Repeatable: can test how the software reacts under repeated execution of the same operations.
4. Comprehensive: Effectiveness of the application will be improved and can rerun the scripts across multiple platforms simultaneously.
5. Reusable: creates scripts for reusable tests that can be run each time the site or application changes
6. Cost Effective: Total cost of performing manual testing is exorbitant in comparison to automated testing. By executing automation scripts the rate of error will be less than manual testing and it can execute the tests much faster.
7. Improves Quality: By repeatedly executing the automated scripts, more defects can be found and further improvement of the quality of end product can be done .With Automation we can run more tests in less time with fewer resources. 

Type of Test Automation Approach:

There are two general approaches to test automation:

• Graphical user interface testing: A testing framework generates user interface events and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.
  Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected. The advantage of this approach is that it requires little or no software development..

• Code-driven testing: The public (usually) interface to classes, modules, or libraries are tested with a variety of input arguments to validate that the results that are returned are correct. Code-driven testing allows the code to conduct unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected.

 Integration of Test Automation Tool and Test Management Tool:

 In the test management tool, we have an entry of automated Testcases so we can integrate it with the test automation tool. Automated Testcases will be automatically executed and their result will be compared with the expected result in the test management tool.

Also, the actual result and status will get updated automatically in the test management tool. By using Test Automation tool, we can send the acknowledgment mail with the automated Testcases status to the QA persons.

 References:

http://www.wikipedia.org/