This is the forth post in a series, you may want to read the previous posts first: part 1, part 2 and part 3.

Now that we learned all about split testing you probably already want to test two designs of your web site to see if you can improve your sales (and if you don’t you should) but there’s one more small thing you should know before starting – what to test first.

The first thing you need to test is your A/B testing software and your site, you do this by running your current design against itself – an A/B test where both option A and option B are identical.

Why are we doing this? first we want to see how a test runs – if you don’t have a huge amount of traffic you will probably see one of the options gaining a sizable advantage over the next for a short while, if everything is working over time this advantage should get smaller until both options get the the same conversion rate.

The time it takes both options to converge is the minimum time you should run future tests.

You should also remember the behavior of this test in future tests – if one options performs much better then the other it doesn’t mean a thing until the result is statistically significant.

And last but not least - this is a good test of your tools, if the two options do not converge or your tool report statistically significant results you know your tool is not reliable and you should switch tools.

In the next posts of the series I’ll talk about my tools and share some results.

A lot as been said about the benefits of releasing your product as early as possible – you get feedback earlier, if the product is doomed you find out earlier and you get some cash-flow earlier.

But there is one even grater advantage, especially for tiny companies – it gets the product released.

Back before I released yaTimer I had 4 mostly finished product I was going to sell, I wrote a web page for yaTimer with a form that let people request access to the beta, put it on-line and ran a small AdWords campaign.

Before I had the web pages for the other 3 products ready I started getting beta requests – so I had no choice but to finish yaTimer and start selling it.

By the way, as far as I know, non of those who requested beta access even downloaded the software and I suspect that those requests were mostly click fraud (when a site owner clicks – or hire others to click – on ads on his/hers own site just to get money out of the advertiser).

And it’s the same with Squirrel Clock, the beta is up (you can create your beta account on this page), I have some boring and tedious reporting code I have to write – but I have no choice, the program is already out there and I have to complete the reports feature, I can’t work on some other almost complete product.

Releasing a product in an early incomplete stage will force you to complete the product – if you wait until the product is truly ready there’s a good chance you will never ship it.

I just uploaded yaTimer version 2.5.1 to the server, this is a bug fix only release with no new features, this is a free upgrade for all customers.

If you have yaTimer 2.2 or later you can upgrade from within yaTimer by clicking the “check for updates” button in the about window or the options window - or you can wait for the automatic updates check (by default yaTimer checks for updates once a week).

If you have an earlier version of yaTimer or you don’t want to use the automatic updater you can also upgrade from the upgrade page on the web site.

If there’s any problem with the automatic updater please tell me (you can use the contact form, the Feedback option on yaTimer’s main menu or the e-mail address in the bottom of this page).

This is the third post in the A/B testing series, you may want to read part 1 and part 2 first.

So you started your test and after 10 visitors arrived to your site (5 in each group) 2 visitors from group B converted compared to just one from group A – the change improved conversion rate by 100% – time to party.

Well, maybe it’s too early to party, just 10 visitors isn’t much and it’s possible those two customers arrived at the same group by accident, how do we know the next 3 visitors to group A won’t buy and turn the results around?

What we need is some way to reliably predict the actions of a large number of people in the future based on the actions of a small number of people in the past – luckily we have such a tool – statistics.

Even more luckily our software will do all the math for us – but we do have to know how to read the results.

First there’s the confidence level, remember we are trying to predict the future – we can’t be absolutely certain of anything but we can know exactly how much we are uncertain – the confidence level is the chance that the result we are seeing is the true result.

The minimum confidence level we can accept is 90% with most using 95% – this is because lower levels practically guarantee we’ll have errors – if you run just 5 tests with 80% confidence level one of them will show the wrong results, with 95% confidence one in twenty tests will be wrong – on the other hand we can’t always require a high confidence level because for higher confidence levels we need more visitors – so the test will take longer and we can run a smaller number of tests and get the results later.

Most tools will show us something like:

Original    2.5% +/- 1.5%
Alternative 3.9% +/- 1.5%    56% improvement

So what does this tell us? that the alternative is 56% better than the original? not exactly.

This tells us that there is a 95% chance that the original conversion rate is somewhere in the range 1.5%-4% and the alternative conversion rate is 2.4%-5.4% – or in other word its possible the original conversion rate can be 4% and the alternative can be 2.4%

So, to summarize:

• Anything with a confidence level below 90% doesn’t mean a thing and you should aim for at least 95% in most tests.
• The result isn’t meaningful until the highest number is the losing option’s range is lower than the lowest number in the winning option’s range

In the next post I’ll tell you about what your first test should be

Like anything in science there are some thing you need to know before running A/B tests, there are some basic things you have to get right or the whole test will be meaningless.

As I said in the previous post A/B testing is done by dividing your visitors into two groups, we’ll call them A and B, and showing a different page to each group and than measure what group performed better, there are quite a few split testing tools out there and they should handle the technicalities I describe here automatically – but it’s important to know this in order to understand what your tool is doing.

First, to get any result at all you need a measurable result, usually we simply count how many time visitors from each group performed some action (typical actions are: buying something, downloading something or filling a form) – those actions are usually called “conversions” (the person converted from a visitor to a customer or a lead).

After we decided what our conversion action is we get to the second point, have to split our visitors between those groups, it’s extremely important to randomly divide visitors between groups because otherwise we are likely to get people how are more likely to buy to begin with in one group – and this will make this group perform better regardless of the actual design we show them.

As a (overly simplified) example, if we show one design to people visiting on workdays and a different design to people visiting on the weekend and we sell business software the workdays design will win simply because people who want to buy business software usually don’t look for products on their day off.

It’s also important to know that all sort of tiny details effects visitors behavior (page load times is a notable example) and that if the visitor knows he/she is participating in a test the test is just about completely ruined.

In the next post I’ll teach you how to read the results of a split test (it’s not difficult but can be somewhat counter-intuitive).

A lot was already written about the speed advantage of small businesses, in the time it takes the big established competitor to organize a pre-meeting to set the schedule for setting the agenda for a meeting to brainstorm if farther meetings are required to decide if a new product is worth exploring the small business can get the product to market.

This speed advantage is especially important because nobody really know the right thing to do, there is no way to find out how profitable a product will be or what the effect of a change to the web site will be without actually trying, so trying a lot of things quickly – and failing at most of them (as long as you learn from the results of those trials) – is much more valuable than any process that is designed to minimize the risk of doing the wrong things.

While a small business can be free from the bureaucracy and politics of large companies it is also free from all those employees sitting there all day long working, this means people is a small business have to much to do and there isn’t any free time for things like getting a new product to market or updating the web site.

I have a very long list of things I’d like to do, from minor web site updates to major development efforts – and I discovered that there’s nothing speedy about how much time it takes me to do those things.

The answer is to automate everything, in a big company it can be find to spend 30 minutes to manually deploy a new version of an application – in a small business you don’t have that luxury, deployment must be a one-click operation.

Also, you have to compromise, my content management system requires me to manually code new pages in HTML – because I couldn’t find a good enough editor – coding HTML by hand is tedious and error-prone so I find more cost-effective things to do and don’t write those pages.

The truth is that any crappy editor-generated HTML on the site is better than perfect hand-crafted HTML that’s trapped inside my head.

But I do learn from my mistakes, I spent a few hours adding Markdown support to my CMS (Markdown is an easier way to write HTML), I made sure deploying web sites updates to the site will be fast and effortless and I’m back on track to once again be a fast business.

Let’s imagine for a moment you run a web site, and that you make money from this web site somehow (for example selling time tracking software) – how can you improve the web site to increase your profits?

One very common method is to blindly make changes to the site and hope for the best, sometimes it even work, but there got to be a better way.

And that better way is A/B testing or split testing – in this method you show half the site visitors your old design while the other half get the new design – then you compare the actions of visitors in the first group with the actions of visitors in the second group and see which design is better.

This is similar to how new medication and medical procedures are tested before they get approved, it gives you scientific, evidence based and clear results.

I’ve installed split testing software on this site and I’m going to document the tests I run and what I’m learning from them here.

In the next post I’m going to talk a little about some important concepts and considerations in A/B testing.

You can also split test with more than two options or run multivariate tests where you test multiple parts of the page instead of two complete designs – but I’m not going to talk about those here any time soon.

yaTimer version 2.5.1 is a bug fix release with no new features, it is scheduled for release later this month and is almost ready.

If you encountered any bugs in yaTimer now is a good chance to tell me about it so I can solve them for this version.

You can report bugs using the contact form, clicking “Send Feedback” on yaTimer’s main menu, via e-mail or in a comment on this blog.

yaTimer 2.5.1 will be a free upgrade for all existing customers.

I’ve been so busy lately I didn’t have time to blog so I’m writing this post to summarize everything I’m working on and my plans for the future.

The most important thing for 2010 is the decision to retire StartMenuSearch and focus on time tracking and time management related products, this will let me expand the Nbd-Tech product line in a way that makes sense from a marketing perspective.

I’ve released the second time tracking product - Squirrel Clock – and I have 4 (yes four) other time management products in various stages of planning and development.

I want all my product to be the best in their category, this means every product have to be finished, polished and marketed – so only one of those four is currently planned to be released this year.

Below is a detailed breakdown of my development plans for 2010, all those plans are just plans and can change at any time without notice – don’t say I didn’t warn you.

yaTimer – The easiest way to track your time

yaTimer 2.5.1 will be released this month, this will be a service release and will only contain bug fixes, no new features.

The next version is yaTimer 2.6, this version is planned for the second quarter of 2010, this version will contain new features, especially in the area of project management.

The version after that will be yaTimer 3.0, as the name suggests this will be a major upgrade, this version is planned for December.

Squirrel Clock – The easiest way to track your employees time

Squirrel Clock is still in private beta, you can register to participate in the beta on the Squirrel Clock home page, everyone who registered for the beta will get a 50% discount for the first year after the beta is over, so register now.

The private beta, together with the 50% discount, will end this month (or the beginning of march at the latest) when all the functionality planned for the first version will be ready.

As a web application Squirrel Clock will have small frequent updates instead of a large update every few months, the schedule and content of updates after coming out of beta isn’t set yet and will depend on customer feedback.

Productivity Tips

There hasn’t been any new productivity tip on the blog for some time now, this is because the productivity tip series is expanding, it will leave the blog, get a brand new format and it’s own section on the web site.

The new and improved productivity tips will make their first public appearance later this month or early march.

This Blog

The blog will continue as it is today (just without the productivity tips category) with a mix of in depth technical content, business topics and product related announcements.

Undisclosed Future Product

The third time management product is planned to be released in the third quarter of 2010, I will publish more details closer to the release date.

I am very happy to announce my second time tracking product: Squirrel Clock.

yaTimer is a wonderful time tracking tool, it is designed for people who need to track their own time - freelancers, consultant and anyone who wants to improve personal productivity – it does not fit very well with the need of employers who have to know how much time employees spend in the office.

For tracking how much time employees spend at work I’ve built Squirrel Clock – an on-line employee attendance tracking time clock.

Squirrel Clock is now entering a very short beta period, anyone who registers for the beta will receive a 50% discount for the first year after the beta is over – so if you, or anyone you know is looking for an employee attendance tracking solution register now.