The growing pains of NGEDIT

A couple of weeks ago, Google updated it’s public page rank – that is, the page rank that the Google toolbar shows you on a given page. Back in March 29, I released the graphical cheat sheet and tutorial, which was and has been pretty popular, with hundreds of links around the web and many tens of thousands of visitors (and still the most important traffic driver to my site). The graphical cheat sheet page and tutorial had been showing PR0 since then, meaning not-yet-assigned. The main www.viemu.com page itself had only been showing a meager PR2, as I had linked to it some time before. I was pretty eager to see Google show its love, not only by showing the page among the top 5 results for “vim tutorial” and several other searches, but also with its PR.

Finally, a couple weeks ago, I was glad to find out that Google had assigned it a page rank of 6, which is a pretty respectable number. It seems the rank is kind-of logarithmic, so a single page rank point may reflect a 10x variation of popularity.

My previous highlights were PR5 for this blog and for some ngedit.com pages, both of which have dozens of links around the web. It seems you need hundreds of links, possibly with at least several of them from reputable pages, to get into PR6. I don’t know whether del.icio.us user/link pages are spidered and accounted for by Google, that would get it in the thousands.

Anyway, the update in the page rank as reported by the toolbar hasn’t had any effect in the traffic Google directs to my page, or the results ranking. This is expected, as it seems the page rank reported by the toolbar is just a more-or-less quarterly snapshot of the internal pagerank that Google actually uses.

As a side effect, there is another SEO trick I think I’ve found. Google has assigned a very modest PR of 3 to the main www.viemu.com page. There are some links around the web to this page, but nowhere near the amount and significance of the links to the cheat sheet itself. But I’ve been pretty surprised to find that the vi tips page in the site has gotten a wonderful PR5! What is my interpretation? The vi tips page is the only one that links to the cheat sheet & tutorial page directly. Nobody has linked to the tips page directly, so I’m pretty sure all of those points are assigned by Google thanks to the fact it is the only parent of the PR6-popular page. Good to know?

I will try to set some time aside to restructure the site so that the home page itself, which is the best landing page for potential customers, links directly to the graphical cheat sheet & tutorial page. You might be interested in applying this knowledge to future design decisions about your site, as well.

On other issues, sales in July have been slower than usual, but still good – given I’ve just ended the special introductory price, and that I had cannibalized most natural-July-sales by announcing the price increase prominently during all of June. I’m expecting ViEmu will continue to sell well at the new price after the slow period of the year, and I’m looking towards some increase with the release of 2.0 later during summer. Hopefully, even higher afterwards, thanks to some extra tricks I’m preparing. I wanted to finish 2.0 by early August, but this, of course, has turned out to be a very optimistic timeframe – late August or early September is much more likely. I’m also feeling I need some time away from hands-on development to “recharge” my motivational batteries, so I will be taking a quiet, calm August, and advance slowly towards the next steps. All is fine and I’m looking forward to a very exciting second half of 2006.

Enjoy the summer everyone!

Today is the 1st anniversary of the conception of ViEmu. That is, this very day last year, I came up with the idea of developing a vi/vim emulator for Visual Studio. I had been working for months in the kodumi text editor (back then it was just ngedit), and the last stretch had involved developing a scripting language compiler and VM, and implementing a vi/vim emulation module in this language.

It would only take me about one month and a half to actually release version 1.0. It was a really hectic month, though. Actually, the short time-to-release was largely thanks to the fact that I already had the basic vi/vim emulation code – even if I had to port it from ngedit’s scripting language into C++.

ViEmu is nowadays a very solid product, having gone far beyond what I expected both in functionality and in sales performance. I’m now concentrating in preparing ViEmu 2.0, which will finally integrate the codebase back with kodumi, and provide some pretty advanced features to existing customers. I will also be ending the introductory pricing at the end of this month. I initially planned to introduce the new price at the same time as ViEmu 2.0, even if 2.0 is a free upgrade to existing customers, but the new version will be taking a bit more than that, and I really think ViEmu is a very good value for its full price. Actually, it seems a bit absurd that ViEmu 1.0, which was a much, much more basic product, cost the same as today’s ViEmu.

Working on two projects is a challenging dynamic for me. I am a “depth-over-breadth” type of guy, and I have trouble switching focus. I’ve worked both in kodumi and in ViEmu for the past few months, and I expect to keep doing so for a long time to come. It’s even more challenging because of the different nature and status of both products: one is for a very niche audience, with no competition, while the other is for a large public, with plenty of competition. One is already a selling product, while the other is still in pure development towards 1.0. One has a limited potential, while for the other one I see the sky as the only limit. One needs development work, while the other needs marketing work. One of them already earns me both a long user request list and a large amount of flattering user feedback, while the other is still something that only I have used. One already helps pays the bills, while the other one only helps reduce my social life. I always have some trouble in setting the priorities, but I think I’m striking some kind balance in both improving ViEmu and advancing towards kodumi 1.0.

Fortunately, most of the codebase of both products will shortly be shared, and that will help with at least the part that is common. Also fortunately, the current customers of ViEmu are potentially also interested in kodumi, so I see the effort in improving and supporting ViEmu as an investment in establishing a good relationship with customers that can result in a business benefit.

As a summary of the ViEmu marketing week I last posted about, which of course ended lasting about 10 days, I must say I’m happy that ViEmu sales are breaking new records during June. I cant be sure whether this is due to the announcement of the new pricing policy, to the redesigned web page, to the latest maintenance release, to the richer trial period user experience (no nags, just better notices and a welcome screen that provides the most relevant information), or to a certain maturity of the product. But I’m sure all of them help. I’m looking forward to seeing how sales figures evolve in July, just after the effective pricing changes. I’ll let you know during the next few months what the general trend is, both after the pricing change and after 2.0 is released.

Finally, as soon as ViEmu 2.0 is ready, I will be focusing more in kodumi. Actually, part of the work for ViEmu 2.0 will indeed revert in kodumi. Even if I announced that I may release another derived product before kodumi 1.0, the core technology in that product is needed for kodumi, and I’m pretty much an expert now in building Visual Studio extensions, so it shouldn’t take as long to prepare as ViEmu has taken. On the other hand, I’m really excited to start working in this part of the code, as I will finally be working in an innovative area (a vi/vim emulator as a Visual Studio add-in is an interesting product, but it can hardly be called innovative). If everything goes well, I will be posting about it on the blog as I start working on it, so it will also bring some interesting technical content to the blog. Well, I will hopefully have the energy to post about it at the same time I’m developing it.

Thanks everyone for your continued support during this year.

As I mentioned in my last post, apart from ViEmu and the kodumi editor, I’m working in another product. I tend to concentrate on development most of the time, rather than marketing ViEmu. Don’t get me wrong, not only do I think that marketing is second only to product quality as the most important part of this business, but I enjoy marketing. The reason is that I think that ViEmu can not become a large business, because of the inherently small audience of a vi-vim emulator for Visual Studio. Thus, I think that the best way to grow the business is to release a product for a larger audience, rather than trying to squeeze every extra N% sales by implementing effective sales techniques.

Anyway, there are two phenomenons that push in the other direction. For one, pure coding of a product before it’s released lacks the thrill of direct feedback, so it’s very tiring. At least that’s how I experience it. And second, any improvements in ViEmu sales make it directly to the monthly bottom line, which is a pretty good motivator.

Thus, after a solid coding Saturday, I decided to dedicate some time to marketing ViEmu better. There were two main things that were irking me:

• I get pretty good feedback by e-mail and through the forums, but looking at the number of downloads it still looks paltry. Not having a super-easy way to get feedback (esp. criticisms!) leads to my ignorance about why those that don’t buy ViEmu don’t buy it.
• The main page of ViEmu (also doubling as landing page from Google adwords) was a bit dull. Too much text. Informative for those interested, but I don’t think it really “grabbed” visitors.

So, I dedicated all of Sunday to redesigning it, adding functionality so that visitors can send feedback from a simple form there, and making it more “catchy”. This ended up as an animated demo of Visual Studio running ViEmu. You can see the result here:

New main page

And, just for reference, the old one is still here:

Old main page

I’ll let you know how it turns out to work. I am planning to implement some other marketing “tricks” during this week, as well as releasing ViEmu 1.4.5, and then I’ll go back to more coding and support, coding and support, coding and support…

Fact #1: I haven’t posted on the blog for well over a month. With all the pending things I have (ViEmu 2.0, the text editor, my day job obligations, support, etc…), I can hardly find time to do so. Promises not kept: the “Friggin’ Darn Tough/Functional Dynamic Template-based C++” series, an article on ViEmu I promised to Keith Casey from CodeSnipers, an article with cool graphical charts on the digg effect as seen from viemu.com (more on this below), etc… Hopefully everything will come along. Until I build the business to the point where it will sustain me, I really just can’t afford do put my available energy in anything other than improving & supporting ViEmu, and preparing the next product.

Fact #2: The final name for the NGEDIT text editor will be kodumi. I wanted a name that sounded good, and which wouldn’t be limiting for the future evolution of the product. It means “hacking” in Esperanto, although Esperanto is not, like, so widespread that the meaning is the important part. I like how it sounds and I can identify with it. It still works when the product becomes more than a text editor. If the product is really good, which I’m hoping it will, this should ensure the name sticks. I’m open to feedback and criticism. I’m pretty stubborn and it’s unlikely I’ll change it, though.

Fact #3: The next product I release probably won’t be the kodumi text editor. There is quite some work yet to be done with kodumi before 1.0, and I’ll probably release another product based in another functional part of the editing core, as a VS add-in. Hopefully with a much larger appeal than a vi emulator. It will actually be based in one of the innovative features I’m planning for kodumi 1.0. It’s nice to have a product that has several offspring before being born.

On the other hand, given that I will probably be releasing this product, it may make sense to have a single site for all VS add-ins instead of a separate one for each product (such as viemu.com). Oh well… this right after moving to viemu.com… so much for my strategy forecast skills.

Fact #4: The amount of traffic you get from a reddit / del.icio.us / digg front page is amazing. I’ve also got thousands of visitors from StumbleUpon.

Here are some graphics that show it, as the graphical vi/vim cheat sheet I released made it (twice!) to those front pages. I apologize for not being able to write a full article on this, it would be worth an entire study.

In order to understand these properly, take into account that originally ViEmu was hosted at ngedit.com, and I moved it to its own domain viemu.com together with the release of the cheat sheet. The traffic graphs include both domains, as they’re served from the same account, but the Alexa graphs below show both domains with separate lines.

I released the cheat sheet on March 28. Here is the traffic for that day (click for a full sized image):

You can clearly see the moment it picks up to 100kbps sustained. The climb was caused by it getting to reddit’s homepage, which happened about half an hour after I submitted it (people liked it, so they voted for it, making it reach the front page – it’s not against their guidelines to submit your own stuff).

The traffic before the climb used to be typically low – very nichey product, a few blog readers, etc… enough to result in some sales, but nothing big.

I went to bed as soon as I saw it at the bottom of reddit’s front page. The next day would be crazier.

As a side effect, people started bookmarking it to their del.icio.us account for later reference. This is understandable given the “reference” nature of the cheat sheet. As soon as a fair number of people did this, it also appeared in del.icio.us’ popular page, thus getting more traffic from there.

This is the traffic on the 29th:

I apologize for not presenting a higher-resolution sampling, I forgot to save it from my hosting provider, and I can’t generate it again.

Anyway, please take into account that the lowest bar in the graph is as high as the 100kbps high in the previous one. It was pretty amazing. I first watched it for hours no end in reddit’s and del.icio.us’ homepage, and a lot of traffic coming. But then I submitted it to digg, and watched it play the voting game in digg’s “sub leagues” (the system is very different from reddit). And then the big spike came: it made it to digg’s front page. All hell broke loose, bandwidth requirements grew to 2Mbps sustained, and the number of visitors was amazing. It made reddit and del.icio.us look like a joke.

My hosting provider handled it without a hiccup. On the other hand, that very afternoon after submitting to digg, (1) there was a power outage at my building, (2) when it came back, my DSL service was down and unfixable according to my ISP, (3) I got a flat tire when driving to a friends’ in order to watch the digg effect, and mainly to be on the watch in case bandwidth went beyond the monthly limit, which happened, so (4) I had to upgrade my web hosting account. You can say I had all the hiccups web servers usually have in these cases.

Here you can see the traffic for the next two days:

You can see the long tail of the digg effect. Also, the cheat sheet got linked from many places around the web, and StumbleUpon started to pick it up as well.

Here you can see a graph of all of March’s traffic, a nice picture of the reddit, del.icio.us & digg effects:

And here is a glorious graph of all of 2006’s traffic:

I promise that I had traffic before March 29, even if here it’s squashed into oblivion!

Finally, I’ll bring you some captures of what alexa thinks of my domains (it doesn’t know they are related somehow).

First, here is the Alexa’s “Daily Reach” measure, for the last 12 months, 6 months and 3 months (just for your static zooming enjoyment):

I can almost tell you where each spike comes from: the first one, in May last year, comes from Eric Sink’s kind mention of my blog & NGEDIT. The second one comes after the release of ViEmu. The largish one before the digg effect comes from a mention in Bungie’s web newsletter (which, expectedly, led to thousands of hardcore gamers, only one of whom was courageous enough to actually download ViEmu), etc…

I chose to show the daily reach above just because it is the Alexa measurement which best shows the evolution of my web presence. Their best known stat is the “rank”, which ranks the site globally among all websites. They only plot it for the top 100,000 sites, but they give you the number in any case. Here are the graphs of the rank, for the last 12 and 3 months:

Actually, the second large spike you can see earlier this month was due to the cheat sheet making it once again to digg and del.icio.us’ front pages, this time as a direct link to the cheat sheet’s GIF file.

Amidst all of this traffic madness, there is another important source of visitors which is often overlooked. I know I did. The name is StumbleUpon. This is not a social links site, but a plugin that you install to your browser, and with which you both (a)vote sites up or down, and (b)discover sites other stumblers’ liked. The effect is much slower, but the amount of visitors it can bring during a few weeks competes with the likes of reddit and digg.

In order to show this better, I will show some visitor numbers by referrer (only for viemu.com). I’ve decided not to totalize them by domain, as the distribution of source pages also provides some interesting info. I haven’t included many other sources, generated from bloggers, news sites and site owners discovering it and linking to them.

March
Total unique visitors: 22,901

April
Total unique visitors: 20,429

May
Total unique visitors: 6,208 (this doesn’t count those coming through the GIF link as that is not considered a “page” by awstats)

Just for fun, I have included the links from several sites in Poland during April. For some reason it was very popular there during that month. Maybe vi/vim is better suited to heavily accented languages like Polish?

Fact #5: I’d need to sell about 1.5x to 2x as much as I’m selling now to live off of the income from ViEmu. Not a big success 10 months after release. It’s ok, as I’ve learned a lot from the experience, and I needed to do most of it for kodumi anyway, which is the main goal. At least for the kodumi I want to develop and release.

Fact #6: vi/vim emulation for VS is not for the masses. I have gotten over 50k visitors to the site in the past two months. This is about more than 20x as much as I was getting beforehand. I guess a product with a more general appeal would have noticed an enormous spike in sales. I’ve only seen a smallish upwards trend. Even VS users are a minority among vi/vim fans! I’ve sworn not to switch over to a Dvorak keyboard layout until the business really takes off, I could end up targeting an even smaller market!

Fact #7: I don’t understand Google results. I’m on page number one for “vim tutorial”, but nowhere to be seen for “vi tutorial”. I was extra careful to write “vi/vim graphical cheat sheet and tutorial” everywhere, so that I would be found by any of the likely keywords, and the result is so bad it’s sick. Searching for “vi emulation visual studio” gets the old page, even if there are links to www.viemu.com all over the place. If there’s a sandbox, I don’t understand why it affects some keywords and not others. Is “vi” too short? Then how did my SEO work before with the ngedit.com address? I’m starting to experiment with creative redirections to the new site, but I’m going to do it the slow way in order to cut the losses in case Google doesn’t like my playing around.

Fact #8: it was cool to have the vi/vim cheat sheet translated into simplified Chinese by Donglu Feng, a nice guy who sent it over to me. It makes regular vi/vim seem a piece of cake:

I think I promised a general strategy post & a status report, some time ago. Here goes.

Development strategy

I am currently sharing my efforts between two development efforts. One of them, ViEmu, has been available for almost 8 months now. It has improved, a lot, and sales have been steadily climbing. Although not a stellar success, it’s working well beyond my realistic forecasts (not beyond my wildest dreams), and I’m really happy that I decided to do it.

The second one, code-named NGEDIT, has been in development for a bit over a year, and it’s still not ready for release. In the time I’ve been developing it, both my belief in the concept, and my disrespect for my own time estimations, have grown a lot. I would be very happy to release 1.0 around July or August, one year after the release of ViEmu, but I know it’s still optimistic. And that’s after I’ve decided to cut out most of the stuff for version 1.0!

Of course, apart from these clear-cut fronts, and not including my day job, there are other fronts I have to attend. Customer support, for example, or this blog, for that matter.

I’ll try to summarize, in a general sense, what my current plans for the next few months are. What the main goals are, and how I’m planning to achieve them.

The #1 goal, as you can guess, is to release NGEDIT version 1.0. This is a bit trickier than sounds. The act of releasing it is, in a general sense, more important than the exact functionality it brings. I have come to this conclusion after over a year in development, and the experience of ViEmu. Emotionally, it’s much better to be working on improving an existing product than it is to be working on a product for its first release, with no users or customers. As long as you are not too impatient to get a lot of sales, having actual users & feedback is a big boost for motivation. Having a few sales helps, as well. And, as long as the product is good and there is a need, sales only get higher as you improve the product.

In order to get this process working, I’ve cut out many planned features from 1.0, in order to release it before long. You might ask, why don’t you already release it in its current stage?

A common answer, but not too informative, would be to answer that it’s still too basic, or unusable. Well, not completely true, as I use it. But a better answer would involve some thought on the market I’m getting in. The text editor market is pretty saturated, and most products out there have many man-years of effort built in. There is at least a general perception of things a text editor must have. I think releasing it without these features would be too much of a stretch. Rest assured, I’ve carefully removed everything which isn’t essential for 1.0. As with ViEmu 1.0, the first release will be pretty basic, but it will hopefully be a better tool for at least some people out there, and that should trigger the initial dynamic of usage-feedback-improvement.

Apart from these essential elements, NGEDIT 1.0 will also sport some interesting things that are well outside the minimum requirements list. The very complete vi/vim emulation, for one, or the native management of text in any format (no conversion on load/save). There are a few more, but these are probably the most interesting to talk about. There are two main forces that have resulted in this uncommon feature set. The first is that I’m building NGEDIT 1.0 as the core framework for the really advanced features, which have some unique requirements. And the second is that I’m building it to become my favorite editor first, and only then a commercial product. This results in the need of powerful vi/vim emulation, which is bound not to have much relevance as a commercial feature.

So, we could say the road to NGEDIT 1.0 is drawn by three guiding principles, listed in increasing priority:

• III: Build a good foundation for the future versions of the editor, if not fully realized, at least following a scalable design
• II: Release the minimum product that makes sense
• I: Build my favorite editor

This is not a list of principles I try to adhere to. It’s more of a recollection of the kind of decisions I’ve found myself taking on intuitive grounds. I’ve seen that I will trade the best design for some functionality, in order to be closer to release, and I’ve found that I’ve traded every sensible business principle by deciding to implement some very complete (and costly) vi/vim emulation. The fact that my sticking to vi/vim emulation has resulted in ViEmu, which is a nice product, (kind of) validates the principles. Actually, I think it validates them because I find myself enjoying the effort, which helps in sustaining the long term effort, and the business is gaining momentum. Apart from this, the ViEmu experience has been an incredible sandbox where to learn, and the lessons learned will play a nice role towards the actual release of NGEDIT. For example, the Google SEO front, and also the adwords & clickfraud front.

In a general strategic view, I’m meshing my efforts on NGEDIT 1.0 with steadily improving ViEmu. Even if ViEmu doesn’t have the business potential of NGEDIT, I think that making all the customers of ViEmu happy only helps with the later stages of building the business. One thing to which I haven’t paid too much attention is marketing ViEmu. I think I could easily improve the sales performance of ViEmu with some effort, but I also think this efforts falls on the other side of the line “makes sense over working on NGEDIT”. So far, a bit of Google-tweaking, a bit of adwords, a bit of word-of-mouth, and a deserted market have been successful in building up sales.

This is very different from what I think I should do if ViEmu were the product on which I wanted to base my business. I would have to be working 100% in promoting it while steadily improving it. But, frankly, I don’t think ViEmu would be a sensible sole-business product. Not everyone is dying for vi/vim emulation.

So, what do all the above principles result in, as practical acting? The first point is that, for the past few months, I’ve been (a)improving ViEmu little by little and releasing new versions, (b)designing and working on the core architecture of NGEDIT, and (c)crossporting ViEmu’s vi/vim core to NGEDIT. The reason for the third point was that, upon using NGEDIT myself, I was sorely missing good vi/vim functionality. It already had some nice vi/vim emulation, written in NGEDIT’s own scripting language, which was the seed for ViEmu, but ViEmu had grown way beyond this seed. Thus, principle (I) kicked in, and I started to crossport ViEmu’s vi/vim engine.

Why do I say crossport? The reason is that I have been rewriting the core in such a way that it can be used both within NGEDIT and within ViEmu. This has had some major requirements on the design of ngvi, as I like to call the new core, and it’s a reason it’s taken some serious time to develop. This effort has some nice side effects:

• I now have a super-flexible vi/vim core that I can integrate in other products, or use to develop vi/vim plugins for other environments (ah, if only solving interaction problems with other plugins weren’t the worst part!).
• I can now put in work that benefits both products.
• I’ll talk about it later, but I have come up with some neat new programming tricks due to this effort. The payoff for this will come later on, but it’s there anyway.

The new core is almost finished, with only ex command line emulation left to be crossported. For testing, this core is being used in NGEDIT. That way, ViEmu can advance as a separate branch. As soon as ngvi is finished, I will start implementing ViEmu 2.0 based on ngvi. This new core already brings some functionality that ViEmu is lacking, and I will be just plain happy that most of ViEmu is now officially part of NGEDIT.

And after this, I have a couple major features in NGEDIT that need to be implemented, and a gazillion minor loose ends. If you are an experienced developer, you’ll know it’s those loose ends that put the July/August release date in danger.

Names, names, names

As I mentioned recently, NGEDIT will not be the name of the final product. I already have the candidate for the name, and there’s only one thing pending before it becomes official: I need to check it with a Japanese person. I haven’t been very successful through asking here on the blog, or through asking the Japanese customers of ViEmu. Understandably, I haven’t insisted too much on my Japanese customers – they are customers after all!

I don’t want to reveal the name just yet, as I don’t want even more confusion if it ends up not being the final name. I would also like to have at least a placeholder page ready when I reveal the name.

Apart from this name change, I also intend to do something with the blog’s name. I plan to blog more and more in the future, as the business doesn’t critically require all my energy. I also plan to cover other areas: programming languages, software design, A.I., O.S.S., operating systems, I’d even like to write on things like economy or the psychology of programming! I think a more general name would be a good idea.

Given that the new editor will have its own new name, that I plan to move ViEmu to is own domain (viemu.com, already up with a simple page), and that the blog needs another name, ngedit.com will very likely end up pretty empty.

All that pagerank accumulated for nothing… sigh! In any case, now should be the best moment to do the deep reforms.

I’ll let you know as these names are ready for general exposure.

Tha blog

If anyone has been reading long enough, you will have probably noticed that I post less often that I used to. The main reason is that development itself already drains most of my available energy. There is not much I can do about that, except wait for days where I have more energy, and wait for the moment when NGEDIT is already released. I will feel much better when NGEDIT is out there, and I think I’ll be able to concentrate better on other things. Having put so much effort so far, and not having it available for download & for sale puts a lot of pressure.

But there are also other reasons. For one, I have many interesting topics I’d like to cover, but which I don’t want to cover just yet. I prefer to wait until I have a working product, before bringing up some of these areas. Should be better business-wise.

This ends up meaning that I don’t want to write about the stuff I want to write about. Ahem.

Anyway, I have come up with an area I’d like to cover with a series of posts. It’s about the techniques I have been using for the development of ngvi, which could be described as the application of dynamic & functional programming to C++. Part of the techniques will be applicable to C++ only, but many other apply to general imperative/OO programming. Hopefully it will be interesting to (some of) you.

Happy new year everyone!

I thought I’d write this post before the end of the first week, or else wishing a happy new year would sound completely stupid.

It is indeed not very nice that I haven’t blogged for most of December. For one, a couple of “real-life” events have helped in not being able to do so: I’ve moved to a new, much nicer, apartment, and I’ve had a quite intense family-and-friends Christmas week.

No, the moving to a new apartment doesn’t have anything to do with the income from ViEmu. The moving was planned before I even had the idea of developing ViEmu. And sales are picking up, but they’re not that strong!

Apart from those errands, I’ve been working tough for all this month. I think that it’s more important to advance in the development of my products than to blog more often – the blog is nice in several aspects: raising awareness, making interesting contacts, sharing the experiences… but as long as the product is not there, there is no business to be made!

Even during the family-and-friends week, back at my hometown, I’ve done quite a lot of work on NGEDIT and ViEmu. Mainly on NGEDIT, even if I’ve managed to release a maintenance release of ViEmu last week. NGEDIT is the main project, and the one which (I hope) will be able to generate a nice revenue stream. Anyway, there is a fair amount of synergy between the two products, and both will benefit from my current work. I’m ‘crossporting’ the vi/vim emulation core from ViEmu to NGEDIT – I say crossporting, because it’s actually a heavy reengineering of that core so that it is compatible with both a hostile environment like Visual Studio, and NGEDIT, which is hopefully better designed, and for sure better-behaved. The new shape of the emulation engine will allow improvements in ViEmu’s interaction with VS, and it will also power killer vi/vim emulation for NGEDIT.

Why am I doing this right now? vi/vim emulation for NGEDIT is surely nice, but vi/vim emulation could well be missing from NGEDIT 1.0 without significantly affecting sales – most of the potential customers of a new text editor are not vi/vim freaks.

The reason is that I need to develop NGEDIT as the best text editor for me first, and then that can be a good product. And given that I can’t use anything but vi/vim editing now, I need to have good emulation myself.

I’m currently using NGEDIT every day – not for programming, I do that in VS with ViEmu, but definitely for editing text files, keeping my work log, etc. It’s quite basic yet, but usable. And the part I was missing the most was better vi/vim emulation. It does have vi/vim emulation, written in NGEDIT’s own scripting language (actually, the module on which I built ViEmu), but ViEmu’s emulation is much, much more complete.

The new core is written using the same concepts as the C++ string class that doesn’t suck, that is, as very loosely-coupled template-based code. Of course, using those string classes themselves. And this, together with a powerful and flexible interaction design, allows it to be used both in VS an in NGEDIT.

Incidentally, I realized that, as soon as I finish the “crossporting”, I will have a fully portable vi/vim emulation core that I can use to produce a vi/vim emulator for any other environment quite quickly. I don’t think it can be a very profitable business, but it’s nice to have that possibility.

On the sales front, December was so-so, lagging a bit behind October and November, which were the strongest months. Actually, the web stats from the last two weeks of December were pitiful. I guess most developers around the world had better things to do 🙂

But January, so far, is being excellent. Sales during the first week so far almost equal those of the complete December! I had heard that the market for developer tools in January is strong, but I didn’t expect this much. It probably won’t, but if all of January stays at the same pace, it will be incredible.

And regarding the plans for NGEDIT itself, there are a couple of things to note.

One of the Christmas days, after having been coding for some hours, I was feeling tired. Programming is an activity that gets me really tired, so I often tend to stop doing it and find other activities. I actually have to press myself to get coding. Anyway, I stopped coding and started preparing a “mock-up” of how NGEDIT is going to look. It was some kind of procrastination, so I wasn’t feeling very happy about spending my time in such an activity. But the activity got me completely absorbed, and after some time I had a very, very nice image of how NGEDIT is going to look. I’m not posting it, as I want to keep the details for now, but it gave me the final idea of exactly what I want version 1.0 to be. And it also helped motivate myself. I had done a very rough mock-up when I started with the project, back at the beginning of ’05, but it was too basic, and didn’t do a good job of representing my plans. This one, though, is perfect in many aspects: it’s a source of inspiration, and it also helps clarify the development/release strategy.

All in all, I’m glad I procrastinated that day 🙂

And the second note regarding NGEDIT is that I’m seriously thinking about renaming it. The plans are for it to grow to be much more than a text editor, so the name is a bit limiting. And on the other hand, I plainly don’t like it so much now.

I’d be losing the pagerank, links, etc… to the current ngedit.com, but I think this is the moment I will be losing less with a full name/domain change.

Funny, because I decided to host ViEmu in ngedit.com instead of viemu.com in order to build-up some popularity for the site in preparation of NGEDIT, and now I will be losing that, for both products. Anyway, ngedit.com will keep pointing to the other domains in the future, and in any case viemu.com will become the main site for ViEmu once I release NGEDIT.

Regarding the new name, I have several candidates, and I already have a preferred one (which I stumbled upon today). I’ll wait until tomorrow, and if it sticks, I will register the name. I’ve already registered so many domains in the past few months, that I think I should restrain from doing so just a couple hours after thinking them up.

Anyway, I hope you all had some nice days for the end of the year, and I wish you an interesting 2006. It’s definitely going to be the year for me, I hope it is for you as well!

While I was doing some http logs analysis on the number of downloads of ViEmu, my commercial vi/vim emulator for Visual Studio, some interesting information turned up. Given that I think it may be useful to other entrepreneurs using adwords to promote their business, and that I have also received several requests for my experience with adwords, I’ll be sharing that information in this post. Hopefully it will save a few bucks for other fellow developers.

Some warnings are due before I delve into the details. First, I don’t really have any evidence of clickfraud – there simply are some things in my logs which look, hm, weird, and they may be a signal of something else. But it could all be due to my more-than-limited understanding of adwords. Given that I’m not spending much, I haven’t spent too much time investigating it. It would be wasting that time which is better spent in other areas.

As well, I haven’t taken the time to read all the information available on the net on these issues, so please feel free to point out the possible flaws in my reasoning.

As to the applicability of my case to other people, I guess my case is not the most common one, as I think I’m the only advertiser working on many of my keywords. Given this, there is hardly any bidding at all, and click prices are very cheap (5 euro cents/click). It must work very differently if you are advertising on keywords with a tough competition (I guess I will be able to comment on that once I release the NGEDIT text editor).

Anyway. I set up my google adwords campaing at the end of July, as I released ViEmu 1.0. It took a few hours or days for advertisements to appear for relevant searches, but it’s been working almost unattended since. I changed a minor detail in the ad text, and I added other keyword combinations as google searches reached my site and taught me what terms people actually use to search for in case they’re interested in vi/vim integration with Visual Studio.

Given I had hardly researched at all, I learnt stuff as things happened. When I set up the campaign, I saw I had to pay about 4 euro cents per click. But afterwards, I had to raise the bid to 5 cents/click, as google warned me and turned off advertising for some key phrases because of a price that was too low. This is pretty simple to see at your adwords.google.com account.

I also started getting hits from those clicks. I found them as hits referred from “pagead2.googlesyndication.com/…” where the “…” is a really long and complex reference. It actually took me a while to realize thouse clicks were from google adwords.

There have also been other weird hits, which had a referer address of “searchportal.information.com” followed by some kind of encoded ID (such as “UVsPWVALXVUMVV8LWQgQRggaCFIXE1Y_CFEIDA0BAQ”). These addresses took me to a search page which has the nasty habit of becoming a “frame parasite” to your web surfing, and used to encode URLs to those ID strings and route everything through their site. I had severe doubts that someone educated enough to use vi/vim would surf with such a bugger.

Anyway, back to my http log review, I started doing an analysis on my November data. I usually keep track of how many downloads of my product there are a month, and try to study the correlation with monthly sales (given the 30 day trial period, tracking is a bit difficult, but I think general trends are still there). I decided to classify all hits to www.ngedit.com in November to be able to tell how many of those came from IP addresses that ended up downloading the trial version of product.

I used vim on the log files to do this process. vi/vim is pretty good for this kind of text processing and I had the desired list in a short while, although it did involve some of that vi black magic.

Anyway, it turned out that, out of the ~20,000 hits of the month, over 6,000 belonged to IP addresses that downloaded ViEmu. As an aside, it was higher than I expected. But now that I could focus better on less information, I could start seeing some new information. I removed all lines not containing “pagead2” in this reduced hit log (ad-vi-tisement: “:v/pagead2/d”), and got myself down to just 11 lines – and to my amazement, there were only 3 IP addresses! One IP appeared only once, but the other two appeared 5 times each. In the full log, there were 24 hits from pagead2, and the repetition of IPs was kind of “hidden” (I hadn’t done a :!sort on them to see the unique addresses).

I nslookup’ed both addresses, which actually only differed in the last byte of the IP address, and only ‘localhost’ was returned from the reverse DNS lookup. I went back to the full hit log, removed everything but IPs belonging to the same subnetwork (n.n.n.*), and I also found out that some of the “searchportal.information.com” links belonged to them. Things started to make some sense.

Let me show you one of the googlesydication referers at this point (broken up in lines for nice display):

http://pagead2.googlesyndication.com/pagead/ads?
client=ca-pub-0919305250342516
&dt=1131084326621
&lmt=1131084326
&format=336x280_as
&output=html
&url=http%3A%2F%2F72.14.203.104%2Fsearch%3Fq%3Dcache
  %3Aq1fqcI2sut4J
  %3Awww.lyrics007.com
  %2FBeverly%252520Craven%252520Lyrics
  %2FPromise%252520Me%252520Lyrics.html
  %2Bpromise%2Bme%26hl%3Dvi
&color_bg=FFFFFF
&color_text=000000
&color_link=0000FF
&color_url=008000
&color_border=FFFFFF
&ref=http%3A%2F%2Fwww.google.com.vn%2Fsearch
  %3Fhl%3Dvi
  %26q%3Dpromise%2Bme
  %26btnG%3DT%25C3%25ACm
  %2Bki%25E1%25BA%25BFm
  %2Bv%25E1%25BB%259Bi
  %2BGoogle
  %26meta%3D
&cc=161
&u_h=600
&u_w=800
&u_ah=570
&u_aw=800
&u_cd=32
&u_tz=420
&u_his=12
&u_java=true

That’s a URL!

I started trying to decipher these URLs. Watching other pages that implement adsense, and how they appear on google’s cache, I deduced the referer for this click (for which I was being charged), came from a google cache page (“url=http://72.14.203.104/search…”). It was a cached page from www.lyrics007.com, which is a repository of song lyrics. The google cache had been accessed from a search at google Vietnam (a trip to www.google.com.vn showed that). The search seemed to include part of the lyrics and “vi” with some weird unicode characters in between (I’m as of yet unsure of whether those %25BB are geometric signs or diacritic marks).

Who gets payed for that click? I think the owner of www.lyrics007.com does. A whois look up showed that the hosting provider is located in Houston, Texas, and that it is registered by someone in Hong Kong.

If I visit any of the lyrics pages, sure, the Google ads are relevant to the content of the page. But it seemed that the ad for ViEmu appeared when looking at a google-cached copy of the page. It’s weird but it may happen. I think the “vi” with weird characters in between may have tricked the adsense engine into showing my ad.

The visits coming from ‘lyrics007’ showed different types of activity. Sometimes just a hit to the ‘html’ file, other times regular page viewing involving hits for the graphics on the page, and even downloading the product! I even found other hits from the same IP addresses coming from ‘searchportal.information.com’.

So what may have happened? I have two possible explanations.

One is that a developer in Vietnam was looking for the lyrics to some song, using www.google.com.vn. Developers also listen to music and check lyrics once in a while. He clicked on the google cache, in order to access the page, and Google picked my ad (as, obviously, adsense technology is imperfect and the relevance of the ad is just an heuristic). While humming to the tune of the song, the guy in question saw the ad to my product, and was excited to finally see vi emulation in Visual Studio. He clicked on the ad and came to my site. He even downloaded it.

This would mean that I payed google and lyrics007.com for reaching a potential customer of mine – someone who I wouldn’t have reached easily in another way. Fair enough.

The weird thing is that this same guy has some other friends using the computer (or sharing the IP address) who went through exactly the same process several other times during the month. With different song lyrics, of course. And some of the times, their browser crashed before even hitting the ‘css’ file or the page graphics (or maybe they surf with images deactivated?).

They even downloaded ViEmu several times during the month – they must have a messy download directory.

This also happenend from other domains, not only lyrics007. I haven’t researched them much, but they seem to come from nearby areas. If all of the cases have similar explanations, then the domain holders / adsense publishers are not to blame at all.

And then I have a second possible explanation.

Some guy in Hong Kong has set up several domains with song lyrics and other easily accessible content downloaded from other sites. As those guys are damn smart, they have figured a way to force a google cache access to their page into showing any adsense ad. I’ve been trying to do it myself, and haven’t been able to, but the cache does show weird adsense results. Then, they have some kind of bot which accesses those pages and simulates clicks on the ads. They probably click on many “cheap” advertisers & keywords like mine, but every once in a while they might click on a 50 cent or even a $1 ad. I guess they can make quite some cash that way, apart from the legitimate traffic that their site drives. They even use another method based on ‘searchportal.information.com’ URL hijacking, which hides even more information from advertisers. And they have even improved the bot to fake normal access to web sites.

I can’t know which one is the right explanation. But, I talked to Andy Brice of PerfectTablePlan, and followed his suggestion of turning off advertising on the “content” network (adsense). I’m only advertising on google’s own search results, for which only google gets paid, and which removes the clickfraud incentive for 3rd party publishers.

I’ve also limited ads to specific countries. Mainly, I’ve limited the countries to those on which I already have customers:

• USA
• Canada
• Russia
• UK
• Australia
• Netherlands
• Germany
• Finland
• Norway

I’ve also added other countries which I think are as likely as those to get me customers: Sweden, France, New Zealand, etc… but that’s about it.

Given that I have been spending less than €10 a month, the scam hasn’t been problematic for me. That’s the main reason it took me several months to investigate and optimize the issue – doesn’t make sense to optimize expenses when they are among the lowest ones.

I expect my adwords costs to go down to one tenth of what they have been – which I think amounts to pretty much the legitimate/interesting traffic that I was getting anyway.

I’m pretty happy for the result of google ads – one customer did tell me that they had found about ViEmu through an ad in google search. That single sale makes up for the rest, which I like to understand as the cost of my training with google adwords.

I hope this post doesn’t upset google – I believe it helps other people make a better use of adwords, and thus also helps google have more happy customers!