The growing pains of NGEDIT

I’m already using the scripting language extensively. I first had to complete array access code, and I also had to include “true”, “false” and “nil” (not “null”). I’m now fighting against the intrincacies of vi/vim emulation instead of against the language, so I’d guess it’s faring well.

I’m using the newly discovered style of avoiding all necessary syntactic sugar such as semicolons, commas, etc. The funny thing is that it feels comfortable everywhere except in a single place: I can not get used to leaving out the comma between function arguments. Given that arguments are typeless (the same as all other variables), it seems that something such as:

function IProcessKey ( key modifkey ) 

looks too much as if “key” was the type and “modifkey” the name, and my brain is already too hardwired to that.

On other issues, I’ve started working in a design for the blog – I already have a nice design, just need to work on some graphics elements and then work out how to integrate everything with WordPress.

And I’m also thinking about changing the name of the blog. I don’t like seeing “pains” up there prominently, it paints a negative picture of something which is a positive, enjoyable venture (if not completely painless).

Today, I’ve written the bytecode-based virtual machine that runs the scripts within NGEDIT. It didn’t take more than 6-8 hours. Complete with the access mechanisms to built-in functions, casting of values for inter-type operations, function calls, etc. It doesn’t amount to much more than 1,000 lines of code.

It always strikes me as weird that the kind of things that seem “complex” are often done in quite little time, while other parts which seem easier take so much more. The Undo system, selecting/copying/pasting etc… have taken much less time to develop than things such as the custom menus, tab bar, preferences dialogs, etc, which are supposedly much simpler. No surprise we’re so bad at forecasting development times. I almost dare predict that printing support will take double the time than syntax highlighting will.

I find that my productivity is much lower when I have to be talking to the Windows API, than when I’m writing self-contained code. Does the same happen to you?

My next goal is vi/vim emulation using the internal NGS scripting language, I’ll report how it fares.

The parser/code generator is already working very nicely. I’ve written a little disassembler to examine the output of the code generation phase – it has already helped me find a couple of bugs, and it’s crying for a peephole optimizer (although I’ll leave this for later). Here’s some sample input to the compiler and the output generated:

NGS source:

const
{
  NORMAL
  INPUT
};

function ProcessChar( Char )
{
  if (this.m_Mode == NORMAL)
  {
    var pfn = this.m_MapNormal.Get(Char);
    if (!pfn)
    {
      pfn = this.m_MotionMap.Get(Char);
      if (!pfn)
        ::Beep();
    }
  }
  else if (this.m_Mode == INPUT)
  {
    //TODO: implement insert
    //::Insert(Char);
    ::MessageBox("Insert", Char);
  }
}

Compiler output:

(002) ProcessChar (this, Char)
locals: this, Char, pfn
13           ENTER
02 00 00     PUSH_LOCVAR    0000 ; this
06 07 00     READ_MEMBERVAR 0007 ; m_Mode
01 00 00     PUSH_CONST     0000 ; 0 (NORMAL)
20           EQUAL
16 40 00     JNCOND         0040 ; L0
02 02 00     PUSH_LOCVAR    0002 ; Char
01 07 00     PUSH_CONST     0007 ; 1 (*unnamed*)
02 00 00     PUSH_LOCVAR    0000 ; this
06 09 00     READ_MEMBERVAR 0009 ; m_MapNormal
06 0A 00     READ_MEMBERVAR 000A ; Get
11           CALL
03 03 00     POP_LOCVAR     0003 ; pfn
02 03 00     PUSH_LOCVAR    0003 ; pfn
1E           LOGIC_NOT
16 23 00     JNCOND         0023 ; L1
02 02 00     PUSH_LOCVAR    0002 ; Char
01 08 00     PUSH_CONST     0008 ; 1 (*unnamed*)
02 00 00     PUSH_LOCVAR    0000 ; this
06 05 00     READ_MEMBERVAR 0005 ; m_MotionMap
06 0A 00     READ_MEMBERVAR 000A ; Get
11           CALL
03 03 00     POP_LOCVAR     0003 ; pfn
02 03 00     PUSH_LOCVAR    0003 ; pfn
1E           LOGIC_NOT      
16 09 00     JNCOND         0009 ; L1
01 09 00     PUSH_CONST     0009 ; 0 (*unnamed*)
0A 00 00     PUSH_BUILTIN   0000 ; Built-in
11           CALL
0C 01        STACK_DEL      01
14 1C 00 L1: JMP            001C ; L2
02 00 00 L0: PUSH_LOCVAR    0000 ; this
06 07 00     READ_MEMBERVAR 0007 ; m_Mode
01 01 00     PUSH_CONST     0001 ; 1 (INPUT)
20           EQUAL
16 0F 00     JNCOND         000F ; L2
01 0A 00     PUSH_CONST     000A ; Insert (*unnamed*)
02 02 00     PUSH_LOCVAR    0002 ; Char
01 0B 00     PUSH_CONST     000B ; 2 (*unnamed*)
0A 01 00     PUSH_BUILTIN   0001 ; Built-in
11           CALL
0C 01        STACK_DEL      01
09       L2: PUSH_NIL        
12           RET             

Ain’t it beautiful? Compiler writing has some “magic” feeling to it.

I am facing a little design decision: as you see in the source above, references to “this” are explicit within functions. The reason for this is that NGS is, the same as JavaScript, a class-less language. Each object can have its own set of members, be it values or member functions. For this reason, there is no way of telling during the compiling whether an identifier refers to a member of the “this” object. Allowing direct reference to member variables would turn off the detection of any undeclared identifier reference, turning them into accesses to the “this” object. This is bad, because a mis-spelling of a constant or module variable would silently be turned into a member-variable access (even worse given that all functions in NGS have a “this” object).

The way it is now, member accesses are unchecked at compile-time, but non-member accesses are checked and flagged as errors if not found either as constants, module variables, built-ins or local variables (this includes arguments to the current function).

JavaScript does exactly what I described above as “bad”. Any reference is checked not against the “this” object, but against the whole identifer resolution chain (which may include other stuff). In my experience developing in JavaScript (which consists basically in a FireFox extension), all syntax checking is very uncomfortable (I wonder how they do it, as most of the FireFox UI is actually implemented in JavaScript).

One possible solution is to use a single dot, elliding the “this” identifier, as an implicit reference to the this object. It reminds me of the “with” statement in Pascal, which I really liked (and miss in C++ etc). I could even include “with” as a feature in NGS, with “un-with’ed” code referring to “this” and explicit “with” referring to whatever the programmer wants.

I’ll leave this as it is now, and if it proves too cumbersome as I start using NGS to implement some functionality, I’ll get back to it.

By the way, the page is already in Google (although one needs to look explicitly for NGEDIT, of course), and no spam as arrived in my inbox as a result of publishing the e-mail address. I haven’t received any non-spam e-mail, either. A friend told me he’d dropped me a line, but it didn’t arrive. I checked sending one myself through the link and it worked. I hope it will work.

By the way, I’ve seen NGEDIT does mean something in some language (I think Indonesian, although I can’t be sure). If anyone who knows the meaning comes by, I’ll be very grateful for knowing what it means.

I don’t know whether this will get gigabytes of spam down my pop3 stream, but I’ve published my e-mail address on the sidebar. Feel free to get in touch.

Right now, I am writing the scripting language compiler for NGEDIT. It is loosely based in JavaScript (typeless variables, object oriented), and so it has a C-like syntax. It generates code for a simple stack machine. I’m using a handwritten recursive-descent parser to parse the code and generate the bytecodes as parsing is done.

I’ve noticed two interesting things in the process.

The first one is how much of C/C++ syntax is unnecessary. Python did away with a lot of it, but I never thought that even in C/C++ most stuff could be parsed without any problems with many commas and semicolons left out. Ever wondered about class/struct/enum declarations ending in a semicolon, while functions don’t have one? In the parser, it’s as simple as saying skip next token if semicolon after the decl/def. After most statements, there is no ambiguity if the semicolon is removed (expression parsing stops if the next token isn’t a possible continuation of the expression). It is possible to leave out the commas between function arguments and between identifiers in an enum definition, even if they carry an explicit = expr initialization.

The second one came onto me when I was writing the intermediate structures that hold the code generated for a full function and the code to evaluate an expression. I had to separate the code for the expression, as it can’t simply be added to the tail of the function (as can be done with the statements). It turned out that the structure to hold a compiled function and the structure to hold a compiled expression are identical! (at least, given that I was storing the function signature separately). In NGS (the NGEDIT scripting language) all function return values, so that is common as well. This reminded me of one extension GCC used to have years ago… a body of code surrounded by braces can be embedded in an expression, resulting in the value of the last statement. I don’t remember being able to use return inside the code, but it would make sense. In this way, we could write:

int a = { if (b) return 1; else { while(c--) do_something(d, e); return get_something(); } };

And if we used the parser with optional separators we could even write:

int a = { if (b) return 1 else { while(c--) do_something(d e) return get_something() } }

The optional separators are already working in the NGEDIT parser, but I don’t plan to implement the code-as-subexpression trick.

I just fixed the padding/border problems with the page, so it shows up fine with IE. The fix involves using Mozilla specific hacks to make it understand width/border/… in the same way as IE, that is, deviating from the W3C standard. It’s ugly, but it seems to be one of the standard solutions. The good thing with standards is that there are as many as you want, and if you don’t like them, you can always create a new one.

Now I just need to apply the design to the blog so that it doesn’t show the standard WordPress look, which cries amateur. Some more CSS hacking down the road.

After some design work with the Wacom tablet, some digging into CSS Zen Garden in order to learn some CSS, and some CSS trial & error, www.ngedit.com now shows a nice front page.

Unfortunately, it only displays well with Firefox and it looks really ugly on IE. I hope I’ll be able to fix it soon enough, before no one sees it like that.

My first experiences with browser compatibility. Nasty.

You have to see this keyboard. Lets you switch between Qwerty and Dvorak at will, apart from impressing anyone around. Now I want a laptop with the same concept.

What’s NGEDIT? NGEDIT is my project for a new text editor, which I’ve been developing for the past months, and which will hopefully be ready very soon.

You may be asking yourself, why another text editor? There are dozens of them in the market, of all types and sizes, prices and weights, colors and fits. Why would anyone in his right mind think on developing yet another one?

The answer is actually simple. It turns out that I have have a set of features in mind that will make current-day text editing look prehistoric. Of course, only time (and a lot of work on my part) will be able to prove or disprove this.

Before I can start writing all the innovative features, I need to implement a full-blown modern text editor: multiple text file formats and character sets (codepages, DBCS, Unicode anyone?), multi-file and multi-window support, fixed- and variable-width fonts, syntax highlighting, keystroke and programmable macros, multi-file find&replace, regular expressions, tons of preferences,… you name it.

I’m currently in the process of developing this first text editor, the first step towards the full NGEDIT. I plan to have a first release with:

• all the solid features of a modern text editor
• some advanced ones that are not very common
• some very carefully crafted usability-oriented features
• and some very specific and useful features that aren’t found in other editors

All in all, I think NGEDIT 1.0 will already be a winner over other editors out there, if not in all cases, at least in many of them. I hope to have a downloadable trial version soon enough that you will be able to see it yourself.

I will be posting about how development progresses, the process of setting up the web store infrastructure, there will probably be some rants, and miscellaneous other stuff. Feel free to comment on any issue.