(no title)

LoadVerify performs tests for the 2011 Rugby World Cup

September 5, 2011

This September, New Zealand is hosting the 2011 Rugby World Cup. According to Wikipedia it will be the largest sporting event ever held in New Zealand, eclipsing the 1987 Rugby World Cup, 1990 Commonwealth Games, 1992 Cricket World Cup, 2003 America’s Cup and 2005 British and Irish Lions tour to New Zealand. The organizers expect 95,000 visitors from overseas to travel to New Zealand for the event. To support this event, the teams and its visitors, an army of volunteers and staff would be needed to provide the friendly, knowledgeable and efficient face of New Zealand.

LoadVerify was asked to perform load and performance testing on the website and applications that trained the volunteers and staff. This was great vision by the event organizers and its IT support partners recognizing that the training systems and infrastructure in place needed to sustain the large number of users in a short time frame.

The websites needed to sustain up 1000 users an hour with adequate response times which didn’t distract from the training aims of the training tool. It was logical that this load was expected as there was a set number at staff and volunteers that needed online orientation and online job specific training. Using the same methodology as a go-live date where traffic was expected to be heavy, we were able to determine the large number of users that would be emailed and sent subscriptions to the training portals and the likely user rates that the portals needed to sustain in-order to achieve training objectives.

Identifying the use-cases were easy, as everyone needed to follow a certain path and answer short exam questions after each module. We established a ratio of user-cases (e.g browsing users VS users actively using the training modules VS users choosing to download preparation material VS users actually sitting the on-line tests) By establishing a ratio and having all use-cases running at the same time, we were able to simulate as close as possible to the realistic load expected.

We tested initially in a test environment before re-testing and comparing results in a full production environment (best practice but not always possible). Scripting for the websites themselves were reasonably straight forward due to the portals being well designed and surprisingly lightweight for a website running flash (yes in most cases we can performance test flash technology), but we did identify bottlenecks that we were able to provide feedback on several key elements that most customers asked for.

Number of concurrent users sustained
Average response times for each request or action
Transactions completed within a minute and over an hour
Number of users sustained over an hour with acceptable response times
Error rates vs concurrency rates
Areas of concern

We ran a total of 5 tests, iteratively working through issues that we discovered. The website had dizzying speeds on the final run, leaving us in no doubt that the training portals and infrastructure would support the expected user numbers giving our customer peace of mind and insurance.

Comments Off | Uncategorized | Tagged: go-live testing, loadverify, rugby world cup 2011, testing | Permalink
Posted by Terence Frani

Want to automate your regression testing?

February 18, 2011

Traditionally and even as late as 2008, intending to automate your regression testing or certain parts was a hard task. There were several arguments against doing this.

Automation technology did not cut the mustard. Either too expensive and cumbersome to use or it didn’t have enough features to be useful.

Development methodology, documentation standards and supporting software was still maturing, placing a further burden of cost, time and resources on managing the automation suites for each development life-cycle.

The time it took to implement the tool or make changes to the automation tools to cater for the new versions requiring testing

You needed someone dedicated to run the tools. The investment in this was resource, either by hiring them or training them for the automation tool was debatable given the high turnover of staff in the testing and development arena

In most cases back then, the automating of regression testing became a whole new project unto itself, where the automation testing cost rivaled in time and cost, actually developing the product, app or website that required regression testing

Feature rich automation tools that dealt well with error handling was still in its infancy

And lastly the usual traditional scenario of Test methodology, its benefits and its budget requirements were not given the focus, priority or budget to invest in ‘nice to haves’ – Testing was not understood or valued as much as it is these days.

However fast-forward to 2010/2011 and testing has become a hotbed or activity, there’s tool’s and methodologies galore, testers becoming further skilled and respected as a important addition to complement, work alongside and within development and infrastructure teams.

And as always business units look to cut cost, speed up delivery, and assure quality standards, cue the latest generation of automation test tools; providing everything from load and performance testing, to monitoring services, to providing the real possibility of regression automation and unit test automation.

So why now? All the reasons against automation before don’t predominantly exist any more, there are still some challenges around automation but with careful understanding and management; automating your regression testing should reap you rewards in time, effort and money saved.

Achieving 100% automation would be still unusual these days, however automating 60-70% of your regression test-set should be a good goal to have for your application or website if it has matured enough, as opposed to the 1-30% that we might have managed in the early 2000′s.

Contact Loadverify.com if you wish a consult about automating your regression testing, and keep an eye out for our new regression testing service page and regression testing library options coming out for you in early 2011.

Leave a Comment » | Uncategorized | Tagged: automate, automate regression testing, automate your regression testing, automation, loadverify, regression, regression testing, specialist load testing service, specialist testing, test suite, test-set | Permalink
Posted by Terence Frani

Are you seen as a bearer of bad news when testing?

August 9, 2010

This was a question recently raised on the forums.

Generally it depends on the environment within the project and how successful the project is going.

The tester could always be the bearer of bad news.. or the tester could always be the bearer of great news on items that have been discovered to ensure quality and success. Its the old the glass is half empty vs full mindset possibly coupled with a lack or wealth of SDLC experience.

I always see finding bugs and issues as a positive, because 1. we are able to prioritise and address them if needed, and 2. it means less issues further down the SDLC resulting in a ‘cheaper’ and ‘easier path’ to ensuring quality and early risk assessment.

Teams may see issue/bug discovery as a negative if they are inexperienced or suffering from a bad project experience/support/burnout etc. Generally speaking they should see it as a positive when finding and addressing issues in project mode, rather than after the project has been completed when it will cost exponentially more in time and money to address those issues. (if at all possible).

What should a tester do to avoid being the bearer of bad news? Explain that its a good thing that you are able to address those issues early, be positive about the process and be clear that issues will always be found, its neither good or bad but a fact of software/system development, however its infact a bonus having to address them now than later on. Note finding a bug may not necessarily mean that they get fixed right away, but your job as a tester is to get those issues assessed right away for priority and severity.

Happy testing and be positive with your discoveries.

Leave a Comment » | Uncategorized | Tagged: sdlc, testing 101, testing attitude, testing communication, testing positively, testing sdlc | Permalink
Posted by Terence Frani

How do you estimate testing effort?

July 6, 2010

Generally when pressed I always estimate testing effort including re-testing to be the same duration as development.

So if a piece of code/functionality takes 2 hours to complete, I usually say it will take 2 hours to test. This ensures I have enough fat built in. This also depends on how much unit testing has been performed, the complexity of the code and the experience/skill of the developer including the testing cycle iteration that you are in, the further along you are on the testing cycle the less time it should take to test a piece as most of the issues should get ironed out as the project progresses. There will come a time when the amount of testing required will remain constant which in my experience has been about 50% of original development time to be assured of risk assessment.

e.g For the 2 hours development, I would initially estimate 2 hours testing, but towards the end of the project or test cycle (after the first couple of passes), if any regression/testing reiterations have to be conducted, you may find that the minimum time on this piece would be 1 hour no matter how big the change is to this 2 hour dev component. 50% has always been the minimum rule I apply.

Itemising pieces of work will aid in estimation. I have found that in most when combining time to create the testing framework and conducting the time usually equates to 50%-100% of initial dev time.

This includes

Use case/checklist creation, investigation and documentation
Use case checklist testing and review
Test execution and retesting bug fixes

This rough rule has always worked for me for over 30 or so projects.

Note – If you have historical data of previous testing efforts that you can base your estimates on the more accurate your estimates can be.

Note 2 – Test estimations should not include release management and the time it takes to release into environments including smoke-testing.

Leave a Comment » | Uncategorized | Tagged: loadverify, loadverify.com, project estimation, test estimates, testing 101, testing estimations | Permalink
Posted by Terence Frani

How many checklists do you need in the software testing life cycle?

June 27, 2010

You can have too much documentation.

Check-list numbers should be scalable, due to size and complexity of the project and keeping to KISS principle also always works with lists.
The number of business and technical requirements should give you a rough feel of how many check items you will have, however your first list should be based on risk assessment/heuristics guided by accessibility, audit-ability and completeness\thoroughness of your coverage . This is more important than the number of check-lists. (google: Rapid testiing by James Bach)
My consultancy approach and on mostwww.loadverify.com projects we initially use the Rapid testing methodology on every project, this will dictate the initial list you test from therefore may cover 50-60% of whatever check-lists you come up with in the long run hopefully you also apply the use of the 80/20 rule.

For neatness and simplicity – I typically split the testing checklists into several categories so end up with 4-5 on average :

1. Risk based heuristics checklist (usually covering core functionality, financial impact, complex processes with complex code\integration, high business priority, high risk areas depending on platform or changes being integrated etc etc)
2. Functional and business requirements checklist
3. Usability checklist
4 Load and Performance checklist
5. Go-live checklist

^ etc as a rough idea to formulate my areas of coverage.

As a personal guide, the 80/20 rule should always be applied to all check-lists and should be check items should also be prioritised as such after risk-based heuristics (especially in time critical projects) if not already covered in the original heuristic list.

Leave a Comment » | Uncategorized | Tagged: checklists, James Bach, loadverify, Rapid testing, testing, testing 101 | Permalink
Posted by Terence Frani

Will cloud computing affect the way we test our applications… Especially in terms of load/performance testing?

June 7, 2010

Testing methodology should not change and generically should remain the same, as the apps are moving off a physical network and into the cloud should represent no real change to their functionality either for SaaS or PaaS or both.

When moving to the cloud, there may be some additional test requirements and scrutiny placed on security, privacy concerns and data integrity due to the emerging cloud technology. The length of time, focus and risk based testing on these certain areas and the length of test phases may increase. Specifically the area that must pick up increased testing focus is cloud integration with existing physical infrastructure\apps raising the profile/importance/risk of integration testing.

In general Cloud computing may imply better access for the tester and test tools, it also means that Non-functional testing will most likely become more practical or more ideal from the cloud due to the cloud resource scalability.

What may change is the types of test tools and services you may be able to access that also exist in the cloud. (most vendors have moved toolsets into the cloud)

Part of our challenge (@ www.Loadverify.com) is actually getting into test environments on physical networks from the cloud, for us our testing methodology does not change but the way we access test environments will become ‘easier’ if everything migrates to the cloud.

Before the next technology revolution in the cloud takes off however, most of the testing occurring in-conjunction with the cloud will involve testing (predominantly benchmarking) to justify the benefits and ROI of the cloud vs the costs and constraints of investing\maintaining and frequently upgrading the physical infrastructure and physical environment.
This is initially where I think load and performance testing will be affected (as a justification tool to move to the cloud – showing off the clouds ease or scalability)

IMO The main challenge for load and performance testing is keeping up with emerging web technology’s as everything becomes ‘webified’ such as silverlight, ajax, flash and java etc. One change that the cloud may bring to testing is better reporting and more accurate testing ability due to everything operating in the cloud without any of the usual physical network and infrastructure bottlenecks.

This need has already seen an influx of new and old load/performance tools and services using the cloud.. cue www.Loadverify.com

Leave a Comment » | Uncategorized | Tagged: cloud services, cloud testing, load test, load-testing service, loadverify, specialist load testing service, testing 101, testing in the cloud | Permalink
Posted by Terence Frani

How do you decide what to load and performance test?

March 6, 2010

In our experience, deciding what to load test is no different to how and what you would test functionally to assess system readiness for go-live or for daily operation.

As a LoadVerify a safe rule you need to:

1. Load and performance test your core user journeys. These are the paths most commonly used which would imply the larger expected target load at any given time on your hardware and software.

2. Load and performance test anything that might impact you financially for example the billing user journey, the collecting customer query or order journey to get a handle on any possible bottle necks that might stop you getting ‘paid’.

3. Load and performance test the most complex parts of the system or application. These will be parts of the system where complex tasks are asked of software and hardware, these tend to push for resourcing of either. Your BA or developer should be able to point out these dicey areas.

4. Load and performance test areas that may have been highlighted as risky or problematic areas. For example: A particular function may have been hard to implement, or has other dependencies or third-party services and tools that may cause performance issues. These then should be highlighted as risky areas that need load and performance coverage (These may already have been identified in #3)

Note: Before you commence load or performance testing you need to set expectations in your concurrency, throughput, server/page response times and user journey duration and delays, (You may already have benchmarks from your old system/setup) by having a set of measures to ‘gate’ your decision on risk vs readiness of your new webtool, website, infrastructure or web application – You are able to determine what you should test and whether you are ‘ready’.

If you do not have prior benchmarking and you are moving to a new environment, server or platform, LoadVerify can perform your benchmarking for you.

Leave a Comment » | Uncategorized | Tagged: decide what to load test, load testing 101, Loadverify rule, performance test, specialist load testing service, testing 101, what do you load test? | Permalink
Posted by Terence Frani

What are the benifits of testing with Load Verify’s real browser users?

March 2, 2010

In the past when you think about performance testing via a web application, you think of a tool that sends direct and simple get and post HTTP requests, typically a tool like that also defaults to using 1 connection and multiple threads within that one connection. This utilized in many applications and environments, here at Load Verify it still has a place. Those basic requests for us can still test concurrency and provide some load and ”noise’ for your tests.

However the real pièce de résistance in Load Verify’s toolbox, is the ability to tell you what the real user experience is like when browsing through your site or performing actions on your web application/infrastructure. We do this by opening real individual browsers for each connection or user you require.

This is handy if you want to know what one single user experiences when there are 99 other real users using your application or website and how that experience changes if you add another 1-99 users or ‘noise’ to the mix.

By using real browsers we don’t just simulate making HTTP requests, but actually step through a user action to validate each scenario requirement that has been identified. This can provide a 100% realistic load with threads, sessions and real user wait times and actions required to complete a use-case.

Using real browser users also mean that you can keep up with new technology developments and call different elements within a JAVA or AJAX environment. Scripting for your scenarios is easier and provides realistic load on your platforms simulating real load when the AJAX traffic or requests are being submitted, this when most other approaches may miss similar transactions therefore not providing realistic application and infrastructure load.

We can also set parameters and conditions to a real user action meaning that there is some artificial intelligence or conditioning making the script perform as a user would if a page or element is slow in displaying or being available. All of this adds up to a fantastic advantage and a tangible ROI on your performance testing costs when trying to tell the board or your boss how many real users your site can support and what their specific experiences will be like when submitting or waiting for a request.

In the process of coding our real browser scripts we have previously for our clients uncovered development errors with elements not correctly being called by valid methods, therefore providing by proxy some functional testing along the way. That is how close the real browser users simulate a real user.

Click here to see what sort of reporting we provide on real browser users

Stay tuned for performance tool comparisons in the next couple of blogs.

Leave a Comment » | Uncategorized | Tagged: load testing with real users, loadverify, performance test, performance testing with real users, real browser users, real browsers, specialist load testing service | Permalink
Posted by Terence Frani

Free Web Performance Comparison Offer

March 2, 2010

Free Performance Comparison

We’re looking for people who need to quantify the performance improvement on a web-based application from a technical change. For example:

Would your site be faster if you upgraded to .NET 4? If so how much?
What about adding a CDN?
Or an upgrade to MySQL 5.5?

Putting these changes into production will probably mean making sure everything works and possibly updating code that isn’t compatible. That’s a big commitment if you don’t know how much it will help performance.

So here’s what we’d like to do to help.

We’ll run up to 10,000 page requests per hour against two configurations and give you quantifiable results.

In the world of cloud computing it’s pretty easy to replicate your set-up and make the change you want for a few days for peanuts. So why not get some accurate performance improvement stats?

If you agree to us posting a write-up on our blog and we think it will make an interesting post we’ll happily help you out for free.

Click here to send us your idea.

Leave a Comment » | Uncategorized | Tagged: cloud testing, Free performance testing, Free testing, loadverify, outsource free testing, performance test, specialist load testing service, testing 101 | Permalink
Posted by Neil Webster