Test Requirements Gathering stage involves gathering key information like:

       Definition of workflow or usage scenario

       Estimation of volume for each scenario

       Response time requirements for the scenario as a whole and for critical intermediate points

Test Planning and Strategizing

In the Test Planning and Strategizing stage, a comprehensive Performance Test Plan is prepared based on analysis
Of the requirements collected. A decision is taken on the test environment and the tools to be used for monitoring the
Parameters on each tier (Web Server, Application Layer, and Database Layer). The agenda are detailed out for Scalability analysis, Load, Stress and Endurance testing. A strategy is developed for the number of virtual users and the method of delivery.

Test Script Development

During this phase, the test environment is verified by executing the smoke test. Test scripts are recorded for the test, correlations are identified and the scripts are updated accordingly. For scripts which involve different data for different

 Users, parameterization is done.

Test Execution

The test is carried out over the Internet at Client location or in the Application development environment or is simulated in EPTS LAB based on the chosen delivery mechanism. The sequence of events are captured and observed by setting up Logs. The load is generated as per the strategy decided. Test Results are computed and the required Graphs are generated.

Test Analysis and Recommendation

The test results computed in the Test Execution stage are reviewed and analyzed by a team of test engineers having experience in Hardware platforms, Operating Systems, Databases and Software design. Performance requirements and the results obtained are analyzed by the team. Performance gaps are evaluated. Solution approaches are generated. Cost effective best solution is agreed upon. Tuning suggestions are implemented.

What is the primary goal of TDD?

 One view is the goal of TDD is specification and not validation. In other words, it’s one way to think through our design before we write our functional code. Another view is that TDD is a programming technique. As Ron Jeffries likes to say, the goal of TDD is to write clean code that works. I think that there is merit in both arguments, although I lean towards the specification view, but I leave it for you to decide.

1. What is TDD?

The steps of test first design (TFD) are overviewed in the UML activity diagram of Figure 1. The first step is to quickly add a test, basically just enough code to fail. Next we run our tests, often the complete test suite although for sake of speed we may decide to run only a subset, to ensure that the new test does in fact fail. We then update our functional code to make it pass the new tests. The fourth step is to run our tests again. If they fail we need to update our functional code and retest. Once the tests pass the next step is to start over (we may first need to refactor any duplication out of our design as needed, turning TFD into TDD).

Figure1 – The Steps of test-first design (TFD).

 I like to describe TDD with this simple formula:

 TDD = TFD + refactoring.

TDD completely turns traditional development around.  Instead of writing functional code first and then our testing code as an afterthought, if we write it at all, we instead write our test code before our functional code.  Furthermore, we do so in very small steps – one test and a small bit of corresponding functional code at a time.  A programmer taking a TDD approach refuses to write a new function until there is first a test that fails because that function isn’t present.  In fact, they refuse to add even a single line of code until a test exists for it.  Once the test is in place they then do the work required to ensure that the test suite now passes (Our new code may break several existing tests as well as the new one).  Once our code works, we then refactor it to ensure that it’s remains of high quality.  This sounds simple in principle, but when we are first learning to take a TDD approach it proves require great discipline because it is easy to “slip” and write functional code without first writing a new test.  One of the advantages of pair programming is that our pair helps us to stay on track.

An underlying assumption of TDD is that we have a unit-testing framework available for us.  Agile software developers often use the xUnit family of open source tools, such as JUnit or VBUnit, although commercial tools are also viable options.  Without such tools TDD is virtually impossible.  Figure 2 presents a UML state chart diagram for how people typically work with the xUnit tools.

Figure2 – Testing via the xUnit Framework

Kent Beck, who popularized TDD in extreme Programming (XP), defines two simple rules for TDD.

First, we should write new business code only when an automated test has failed. 

Second, we should eliminate any duplication that you find.  Beck explains how these two simple rules generate complex individual and group behavior:

We design organically, with the running code providing feedback between decisions.

We write our own tests because you can’t wait 20 times per day for someone else to write them for us. 

Our development environment must provide rapid response to small changes (e.g. We need a fast compiler and regression test suite).

Our designs must consist of highly cohesive, loosely coupled components (e.g. our design is highly normalized) to make testing easier (this also makes evolution and maintenance of our system easier too).

For developers, the implication is that they need to learn how to write effective unit tests. 

• Experience is that good unit tests:
• Run fast (they have short setups, run times, and break downs).
• Run in isolation (We should be able to reorder them).
• Use data that makes them easy to read and to understand.
• Use real data (e.g. copies of production data) when they need to.)
• Represent one step towards your overall goal.

2. TDD and Traditional Testing

TDD is primarily a design technique with a side effect of ensuring that our source code is thoroughly unit tested.  However, there is more to testing than this. We’ll still need to consider other testing techniques such as agile acceptance testing and investigative testing.  Much of this testing can also be done early in our project if we choose to do so (and we should).  In fact, in XP the acceptance tests for a user story are specified by the project stakeholder(s) either before or in parallel to the code being written, giving stakeholders the confidence that the system does in fact meet their requirements. 

With traditional testing a successful test finds one or more defects.  It is the same with TDD; when a test fails we have made progress because we now know that we need to resolve the problem.  More importantly, we have a clear measure of success when the test no longer fails. TDD increases our confidence that our system actually meets the requirements defined for it, that our system actually works and therefore we can proceed with confidence.

As with traditional testing, the greater the risk profile of the system the more thorough our tests need to be. With both traditional testing and TDD we aren’t striving for perfection, instead we are testing to the importance of the system.  To paraphrase Agile Modeling (AM), we should “test with a purpose” and know why we are testing something and to what level it needs to be tested.  An interesting side effect of TDD is that we achieve 100% coverage test – every single line of code is tested – something that traditional testing doesn’t guarantee (although it does recommend it).  In general I think it’s fairly safe to say that although TDD is a specification technique, a valuable side effect is that it results in significantly better code testing than do traditional techniques.   

3. TDD and Documentation

Like it or not most programmers don’t read the written documentation for a system, instead they prefer to work with the code.  And there’s nothing wrong with this.  When trying to understand a class or operation most programmers will first look for sample code that already invokes it.  Well-written unit tests do exactly this – they provide a working specification of our functional code – and as a result unit tests effectively become a significant portion of our technical documentation. The implication is that the expectations of the pro-documentation crowd need to reflect this reality. 

Similarly, acceptance tests can form an important part of our requirements documentation.  This makes a lot of sense when you stop and think about it.  Our acceptance tests define exactly what our stakeholders expect of our system; therefore they specify your critical requirements. 

Are tests sufficient documentation?  Very likely not, but they do form an important part of it.  For example, we are likely to find that we still need user, system overview, operations, and support documentation.  We may even find that our require summary documentation overviewing the business process that our system supports.  When we approach documentation with an open mind, I suspect that we will find that these two types of tests cover the majority of your documentation needs for developers and business stakeholders.  Furthermore, they are a wonderful example of AM’s Single Source Information practice and an important part of your overall efforts to remain as agile as possible regarding documentation.

4. Test-Driven Database Development

At the time of this writing an important question being asked within the agile community is “can TDD work for data-oriented development?”  When we look at the process depicted in Figure 1 it is important to note that none of the steps specify object programming languages, such as Java or C#, even though those are the environments TDD is typically used in.  Why couldn’t we write a test before making a change to our database schema?  Why couldn’t we make the change, run the tests, and refactor your schema as required?  It seems to me that we only need to choose to work this way.

My guess is that in the near term database TDD, or perhaps Test Driven Database Design (TDDD), won’t work as smoothly as application TDD.  The first challenge is tool support.  Although unit-testing tools, such as DBUnit, are now available they are still an emerging technology at the time of this writing. Some DBAs are improving the quality of the testing they doing, but I haven’t yet seen anyone take a TDD approach to database development.  One challenge is that unit testing tools are still not well accepted within the data community, although that is changing, so my expectation is that over the next few years database TDD will grow.  Second, the concept of evolutionary development is new to many data professionals and as a result the motivation to take a TDD approach has yet to take hold.  This issue affects the nature of the tools available to data professionals – because a serial mindset still dominates within the traditional data community most tools do not support evolutionary development.  My hope is that tool vendors will catch on to this shift in paradigm, but my expectation is that we’ll need to develop open source tools instead.  Third, my experience is that most people who do data-oriented work seem to prefer a model-driven, and not a test-driven approach.  One cause of this is likely because a test-driven approach hasn’t been widely considered until now, another reason might be that many data professionals are likely visual thinkers and therefore prefer a modeling-driven approach.

5. Scaling TDD via Agile Model-Driven Development (AMDD)

TDD is very good at detailed specification and validation, but not so good at thinking through bigger issues such as the overall design, how people will use the system, or the UI design (for example).  Modeling, or more to the point agile model-driven development (AMDD) (the lifecycle for which is captured in Figure 3) is better suited for this.  AMDD addresses the scaling issues which TDD does not.

Figure 3. The Agile Model Driven Development (AMDD) lifecycle.

Comparing TDD and AMDD:

• TDD shortens the programming feedback loop whereas AMDD shortens the modeling feedback loop.
• TDD provides detailed specification (tests) whereas AMDD is better for thinking through bigger issues.
• TDD promotes the development of high-quality code whereas AMDD promotes high-quality communication with your stakeholders and other developers.
• TDD provides concrete evidence that your software works whereas AMDD supports your team, including stakeholders, in working toward a common understanding.
• TDD “speaks” to programmers whereas AMDD speaks to business analysts, stakeholders, and data professionals.
• TDD is provides very finely grained concrete feedback on the order of minutes whereas AMDD enables verbal feedback on the order minutes (concrete feedback requires developers to follow the practice Prove It With Code and thus becomes dependent on non-AM techniques).
• TDD helps to ensure that your design is clean by focusing on creation of operations that are callable and testable whereas AMDD provides an opportunity to think through larger design/architectural issues before you code.
• TDD is non-visually oriented whereas AMDD is visually oriented.

Both techniques are new to traditional developers and therefore may be threatening to them.

Both techniques support evolutionary development.

Which approach should you take?  The answer depends on you, and your teammates, cognitive preferences.  Some people are primarily “visual thinkers”, also called spatial thinkers, and they may prefer to think things through via drawing.  Other people are primarily text oriented, non-visual or non-spatial thinkers, who don’t work well with drawings and therefore they may prefer a TDD approach.  Of course most people land somewhere in the middle of these two extremes and as a result they prefer to use each technique when it makes the most sense.    In short, the answer is to use the two techniques together so as to gain the advantages of both.

How do you combine the two approaches?  AMDD should be used to create models with your project stakeholders to help explore their requirements and then to explore those requirements sufficiently in architectural and design models (often simple sketches).  TDD should be used as a critical part of your build efforts to ensure that you develop clean, working code.  The end result is that you will have a high-quality, working system that meets the actual needs of your project stakeholders.

6. Why TDD?

A significant advantage of TDD is that it enables you to take small steps when writing software.  This is a practice that we have promoted for years because it is far more productive than attempting to code in large steps.  For example, assume we add some new functional code, compile, and test it.  Chances are pretty good that your tests will be broken by defects that exist in the new code.  It is much easier to find, and then fix, those defects if you’ve written two new lines of code than two thousand. The implication is that the faster your compiler and regression test suite, the more attractive it is to proceed in smaller and smaller steps.  We generally prefer to add a few new lines of functional code, typically less than ten, before I recompile and rerun my tests.

It is rightly said “The act of writing a unit test is more an act of design than of verification.  It is also more an act of documentation than of verification.  The act of writing a unit test closes a remarkable number of feedback loops, the least of which is the one pertaining to verification of function”.

The first reaction that many people have to agile techniques is that they’re ok for small projects, perhaps involving a handful of people for several months, but that they wouldn’t work for “real” projects that are much larger.   That’s simply not true.  Beck (2003) reports working on a Smalltalk system taking a completely test-driven approach which took 4 years and 40 person years of effort, resulting in 250,000 lines of functional code and 250,000 lines of test code.  There are 4000 tests running in under 20 minutes, with the full suite being run several times a day.  Although there are larger systems out there, we’ve personally worked on systems where several hundred person years of effort were involved, it is clear that TDD works for good-sized systems.

7. Myths and Misconceptions

There are several common myths and misconceptions which people have regarding TDD which we would like to clear up if possible. Table 1 lists these myths and describes the reality.

Table 1. Addressing the myths and misconceptions surrounding TDD

 Figure 4:  Agile testing.

 8. Summary

Test-driven design (TDD) is a development technique where you must first write a test that fails before you write new functional code.  TDD is being quickly adopted by agile software developers for development of application source code and is even being adopted by Agile DBAs for database development.  TDD should be seen as complementary to Agile Model Driven Development (AMDD) approaches and the two can and should be used together.  TDD does not replace traditional testing, instead it defines a proven way to ensure effective unit testing.  A side effect of TDD is that the resulting tests are working examples for invoking the code, thereby providing a working specification for the code. My experience is that TDD works incredibly well in practice and it is something that all software developers should consider adopting.

• Continuous Integration
• Automated Build Generation

When we say continuous integration, it means:

• Monitor changes in the repository
• Publish the results via web and/or email

When we say build automation, it means:

• Get source code from the repository.
• Build the complete application package.
• Executing tests cases.
• Coding standards validation using FxCop or some other tool.

Build Automation using CruiseControl.Net

CC-Tray

CruiseControl.Net (Continuous Integration)

This tool as we mentioned is used for Continuous Integration. Once the repositories are in place, we can make use of this tool so that on every change in the repository OR in a timely manner, this process could be started off.  It is purely xml based; we just need to configure its configuration file named ccnet.config. In the following, we will find a sample script which will contain tags for the repository to monitor, calling NAnt & publish results on web and email.

<cruisecontrol>
<project name=”ProjectName”>

<sourcecontrol type=”vss” autoGetSource=”true”>
<ssdir>\\YourServer\vss\</ssdir>
<executable>\\YourServer\vss\win32\SS.EXE</executable>
<project>$/ProjectName</project>
<username>YourUsername</username>
<password></password>
</sourcecontrol>

<!– Changed to correct url –>
<webURL>http://localhost/ccnet</webURL>
<schedule type=”schedule” sleepSeconds=”60″/>
<build type=”nant”>
<executable>DrivePath\nant\bin\NAnt.exe</executable>
<baseDirectory>DrivePath:\projects\ProjectDir\</baseDirectory>
<buildFile>MyBuild.build</buildFile>
<targetList>
<target>build</target>
</targetList>
<buildTimeoutSeconds>300</buildTimeoutSeconds>
</build>
<modificationDelaySeconds>10</modificationDelaySeconds>
</project>
</cruisecontrol>

The above is a sample script file that contains source control tag for VSS. Repository mentioned in this tag will be monitored for any changes. Then in the build type tag, NAnt has been invoked using build tag. NAnt will automate build generation. When cruise control is being invoked, this script gets executed.

NAnt (Build Automation)

The above script will invoke NAnt whenever there is a change in the repository. NAnt is also open source, which is an equivalent for .Net like Ant for Java. Using NAnt we will generate the complete build and related. NAnt’s configuration file is named like FileName.build. It’s an xml file as well, so no need to worry about any specific structure. Just need to get familiar with its tags. Sample script is as follows:

<!– ********** –>
<!– Properties –>

<!– Solution.Filename represents the filename of the solution to build –>
<property name=”Solution.Filename” value=”DrivePath:\Project.sln”/>
<property name=”Solution.Configuration” value=”RELEASE”/>
<property name=”Build.OutputFolder” value=”Drive_Path\BuildLocation\”/>
<!– End Properties –>

<!– build will trigger main build –>
<target name=”build” description=”compiles the source code”>
<call target=”runGetLatest”/>
<solution solutionfile=”${Solution.Filename}” outputdir=”${Build.OutputFolder}${sys.version}\” configuration=”${Solution.Configuration}”/>
<call target=”runUnitTests”/>
<call target=”runFxCop”/>
<call target=”runCreateDocumentation”/>
<call target=”runCopyResults”/>
</target>
<!– End Targets –>

<target name=”runGetLatest”>

<vssget

user=”username”
password=”pwd”

localpath=”PathToGetCode”

recursive=”true”

replace=”true”

writable=”true”

removedeleted=”false”

dbpath=”\\Path\VSS\srcsafe.ini”

path=”$/DirNameInVSS”/>

</target>

Owner: <<Feature Test Owner Example: Selvamohan>>

Feature Team Members: <<Dev/PGMT/Test Owner>>

Feature Rating: (This section to be filled by Test team) <<2.5/3/3.5/4/4.5/5>>

 Please refer to the feature ratings defects:

 For Qualitative rating, look at the following: 

• 2.5 – feature is totally unstable
• 3.0 – still finding significant issues (S1+S2) and/or a lot of small problems
• 3.5 – finding occasional serious issues (S1+S2) and quite a few small problems (S3+S4)
• 4.0 – finding small problems or issues are very very deep and under vague scenarios
• 4.5 – finding very very minor problems – we can ship
• 5.0 – can’t find any problems at all

 Scenarios/Tests covered:

 <<Example: ->Search based on Genre/category->Simple Search>>

 Known Issues /Open Issues: 

• Known Issue: Search does not display the no. of records with <Bug Id>
• Open Issue: Yet to decide on search based on wild characters

 Comments (if any): <<>>

Some places where we can use AOM

This is a small list of places (but not limited to) where we can use AOM. Thumb Rule – Use it at any place where you find yourself doing repetitive tasks while using QTP.

• AOM can come handy when you have a large no of scripts to be uploaded to QC. A simple script can save you hours of manual work!
• Use AOM to initialize QTP options and settings like add-ins etc.
• You can use AOM to call QTP from other application: For ex: You can write a macro for calling QTP from excel.

Caution: AOM should be used outside of QTP and not within the script (during playback). Though there is no harm using it inside but some of the AOM statements might fail.

 How to write AOM scripts?

You need to understand that the very root of QT AOM is Application Object. Every automation script begins with the creation of the QuickTest “Application” object. Creating this object does not start QuickTest. It simply provides an object from which you can access all other objects, methods and properties of the QuickTest automation object model. You can create only one instance of the Application object. You do not need to recreate the QuickTest Application object even if you start and exit QuickTest several times during your script. Once you have defined this object you can then successfully work and perform operations on other objects given in Quick Test Pro > Documentation > QuickTest Automation Reference.

For example: Let us connect to TD QC using AOM and open a script “qtp_demo”

Dim qt_obj ‘Define a Quick Test object
qt_obj = CreateObject(“Quick Test.Application”) ‘ Instantiate a QT Object. It does not start QTP.

qt_obj.launch ‘ Launch QT
qt_obj.visible ‘ Make QT visible

qt_obj.TDConnection.Connect “http://tdserver/tdbin”, _ ‘Referencing TDConnection Object
“TEST_DOMAIN”, “TEST_Project”, “Ankur”, “Testing”, False ‘ Connect to Quality Center

If qt_obj.TDConnection.IsConnected Then ‘ If connection is successful
    qt_obj.Open “[QualityCenter] Subject\tests\qtp_demo”, False ‘ Open the test
Else
    MsgBox “Cannot connect to Quality Center” ‘ If connection is not successful, display an error message.
End If

To quickly generate an AOM script with the current QTP settings. Use the Properties tab of the Test Settings dialog box (File > Settings) OR the General tab of the Options dialog box (Tools > Options) OR the Object Identification dialog box (Tools > Object Identification). Each contains a “Generate Script” button. Clicking this button generates a automation script file (.vbs) containing the current settings from the corresponding dialog box.

You can run the generated script as is to open QuickTest with the exact configuration of the QuickTest application that generated the script, or you can copy and paste selected lines from the generated files into your own automation script.

Smoke

Sanity

Ratio of the number of defects to program length (KLOC)

The Measure Defect-Density (DD) is besides the Measure LOC one of the most widely used measures in industry. The Measure DD is defined as:

Where Defects are the number defects some time after release (normally six months). We will not discuss here the problem of the application of this measure in practice. The next figure illustrates the effects which can occur when using the Measure DD.

Assume, we have two program Versions 1 and 2, and each of them consisting of five modules. For each Version we consider the Measure DD. We see that every module of Version 1 has a lower Defect-Density than every module of Version 2. Summing up the Defects and LOC of each version then we get the result in the line Sum. The result is surprising. Although the value of DD for each of the modules of Version 1 is less than the Values of DD for Version 2 (Relation), the value of DD of all modules of Version 1 is greater than DD of Version 2.

The problem with the measure Defect-Density (DD) occurs if the following relationship between Defects and (K) LOC is violated:

If you do not have this relationship in your collection of programs, then the use of the Measure DD is questionable. You cannot predict the value of DD for the whole system by the DD values of the single programs.

Model:  Defect density is a measure of the total known defects divided by the size of the software entity being measured. 

Number of Known Defects

Size

The Number of Known Defects is the count of total defects identified against a particular software entity, during a particular time period.  Examples include:

    Defect to date since the creation of module

    Defects found in a program during an inspection

    Defects to date since the shipment of a release to the customer

Size is a normalizer that allows comparisons between difference software entities (i.e., modules, releases, products).  Size is typically counted either in Lines of Code or Function Points.

Uses:  Defect Density is used to compare the relative number of defects in various software components.   This helps identify candidates for additional inspection or testing or for possible re-engineering or replacement.  Identifying defect prone components allows the concentration of limited resources into areas with the highest potential return on the investment.  Figure 1 illustrates a typical reporting format for Defect Density when it is being utilized in this manner.

Another use for Defect Density is to compare subsequent releases of a product to track the impact of defect reduction and quality improvement activities.  Normalizing by size allows releases of varying size to be compared.  Differences between products or product lines can also be compared in this manner.  Figure 2 illustrates a typical reporting format for Defect Density when it is being utilized in this manner.