IncursionScript and the Accent Compiler Compiler

When I got a few minutes, I spent a bit more time looking at replacing the Accent usage in Incursion.  For the most part, Accent works pretty much the same as yacc or bison, but does so in a much more user friendly way.  Yacc and bison use BNF grammars, where Accent uses an EBNF grammar.  This allows minor use of some of the regular expression syntax to avoid painful rule finagling completely.

This StackOverflow post quotes the type of work you have to do (from another web page), to convert from EBNF to BNF.  Looking at the one off case, it only seems mildly inconvenient, but when you have several variations, it becomes painful.  Especially, if you're considering converting ~100k of rules for parsing IncursionScript.

monster_def:
  MONSTER { theMon->Attr[0] = theMon->Attr[1] =
            theMon->Attr[2] = theMon->Attr[3] =
            theMon->Attr[4] = theMon->Attr[5] = 0;
            CurrAttk = 0; CurrFeat = 0; theRes = theMon; }
  LITERAL<name>  { theMon->Name = name; }
  ':' cexpr3<n>    { theMon->MType[0] = n; }
    ( ',' cexpr3<n2> { theMon->MType[1] = n2; }
    ( ',' cexpr3<n3> { theMon->MType[2] = n3; } )? )?
    '{' (mon_entry)* '}' { theMon++; };

That was the initial part of a monster type definition. Note that the Accent grammar syntax for the left hand side of rules, embeds the parameter name in < and >, whereas bison has a much more limited support and uses [ and ]. Also, the right hand side of a rule can use the previously mentioned aspects of regular expression syntax. In this case ( and )? are used to indicate 0 or 1 occurrences of a match, and additionally + and * can be used similarly to indicate one or more, and none or more matches.  Note also that the parsed parameter values are used directly in the code blocks, without syntactical sugar, where bison requires that n be $n.

Any use of these regular expression syntaxes would when rewritten for bison, require more and more complicated contrivances and would break up the whole rule making the grammar less readable.  In theory, it should be possible to construct a bison grammar that parses the Accent syntax, with a matching lexer, and convert an Accent grammar to a more convoluted bison grammar.  The bison grammar for the Accent syntax isn't hard to do, in fact, the Accent author provides one in their documentation.  The bulk of the work would be in writing some subset of C parsing, to transform the code blocks, and then there's the flexible Accent parameter lists which bison cannot support at all.

Old with dated pre-ANSI code and using a GPLv2 license, requiring a commercial license to be able to be used freely, Accent is a not just a compelling alternative to bison.  It's almost impossible justify downgrading from it, if one ignores the unfortunate license.  But to become something where any inspired player, can open up a script and start modding, it needs to downgrade from it.

Translating the grammar from EBNF to BNF?  Not enough.  Bison doesn't offer the additional support required.  This solution would still require hours of painstaking rewriting of the custom Accent grammar syntax.

The only remaining alternative, is to write an Accent replacement.  Or to enhance bison or byacc, to support EBNF, and the additional functionality.  But then how many hours would that require?  It might be worth downgrading that 90k Accent grammar to a bison grammar after all.  Something to think about it.

That reminds me, I've searched everywhere three times and I can't find my Dragon parsing book.  It's gone.  I imagine it's with all my compact flash usb adapter, my micro sd card, my issue of Dragontales, and the other things I haven't noticed going missing yet.

MSVC2013, Intellisense and anonymous struct/union bugs

One of the improvements in the C11 standard, is the addition of anonymous structs and unions.  This means that the unnamed element flattens, and all members of the struct or union can be referred to at the same level as the struct or union itself (as long as there are no naming conflicts).

A simplified example use I make of an anonymous struct inside an anonymous union is:

struct my_event_s {
    enum my_event_types event_type_id;
    union {
        struct {
            enum my_key_event_types key_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
        struct {
            enum my_mouse_event_types mouse_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
        struct {
            enum my_touch_event_types touch_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
    };
};

It's a pleasure to program with. But then you reach the stage where you have a different struct, and you want to incorporate a shared anonymous struct (and likely also union) in that. In my case, it was my complete key struct, excerpted from above:

struct {
    enum my_key_event_types key_event_type_id;
    union {
    /* Further fields here.. */
    };
};

One thing you might try, is what I tried. Give the struct a tag, and than just place an anonymous reference to it in the encompassing struct or union.

struct my_key_event_s {
    enum my_key_event_types key_event_type_id;
    union {
    /* Further fields here.. */
    };
};

struct my_event_s {
    enum my_event_types event_type_id;
    union {
        struct my_key_event_s;
        struct {
            enum my_mouse_event_types mouse_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
        struct {
            enum my_touch_event_types touch_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
    };
};

This compiles cleanly, but intellisense will choke on it, and find errors in the code. Intellisense will accept an anonymous struct or union member defined this way within a struct itself. But the moment you try it within a union, then it will choke on not only the struct definitions it accepted up until that point, it will also choke on the union definitions.

It turns out that this is a non-standard (not part of the official C11 standard) feature that Microsoft supports.  It might be argued that usage of it should be avoided, because of this, but with GCC supporting it via the -fms-extensions flag, it is pretty portable.

Personally, I see anonymous structs and unions as a great addition to the C language, that reduce friction without introducing complexity.  And this Microsoft extension should have been a natural part of the standard.  The only downside to it, is that it's actually almost unusable when you're using Microsoft's Visual Studio!  If intellisense chokes on this, then a lot of the value of the editor is undermined, as it's help becomes misleading.

For the record, I'm using Visual Studio 2013, community edition, update 4.  There's a new release of Visual Studio 2015 coming up, which will hopefully mean fixes for this (but who really knows?)  And ideally, a new release of the community edition.

Mysterious Incursion bug

In changes to Incursion back in August, I fixed compiler warnings related to casting. In theory, the changes should have been side-effect free. Testing of the specific lines of code, indicates this is the case. But people were reporting crashes related to the "room weights algorithm". In this case RM_Weights is an array of unsigned int, and c and tot are signed short.

    tot=0;
    for(i=0;RM_Weights[i*2+1];i++)
      tot += max(0,RM_Weights[i*2+1]);
    c = random(tot); 
    for(i=0;RM_Weights[i*2+1];i++)
      if (c < RM_Weights[i*2+1])
        goto RM_Chosen;
      else
        c -= RM_Weights[i*2+1];
    Fatal("Strange Error in room weights algorithm.");
RM_Chosen:
    RType = RM_Weights[i*2];
    RM_Weights[i*2+1] = -1;

I was unable to reproduce the crashes, although others reported doing so up to 15% of the time.

Were the crashes always there and were they merely hidden by all the other crashes, which have previously been fixed? Or are they caused by some unreproducible nuance of the casting? I'm guessing the former.

There's an obvious bug in the above logic. Points to whomever spots it without looking at my fixes on bitbucket.

IncursionScript & the Accent compiler compiler

When I started working with the Incursion source code, which Julian released, one of the problems was that not all the source code was available.  The bulk of the game data, both for d20 elements and dungeon generation, is specified in scripts.  These "IncursionScript" files are compiled into a module, which Incursion can have more than one of.

The problem was that they are compiled using the Accent compiler compiler.  And Julian had a version of this, which had been custom modified to produce C++ compatible code, but no longer had the source code.  I downloaded the original Accent source code and proceeded to try and get it working.

The Accent tool takes a ".acc" file, and transforms it into ".c" and ".h" files which you can compile against.  But it is also generated source code, created using some tool called Gentile.  The correct way to do this, which it appears Julian also avoided, would be to modify the Gentile grammar for Accent, and to generate a new version.  Instead, the approach we both took was to modify the generated Accent source code.

Accent is old school C code.  Seldom having return types in it's own source code, and rarely producing return types in the code it generates.  Occasionally using uninitialised stack variables, and expecting the value to be NULL.  And it generated argument typing for function declarations the other way, where rather than including the type with the argument name in the parenthesised list following the function name, the variable is redeclared fully (in the same way as stack variables) following that list and preceding the function body.

function(argname1, argname2, argname3)
    int argname1;
    char *argname2;
    void *argname3;
{

It is really the fixing of these three things that makes Accent C++ compatible, rather than making it output C++ specific elements.  In any case, a modified version of the Accent source code is now checked into the Incursion repository.  This is built as a dependency of the main Incursion project, ensuring that if there are any grammar changes, then they are rebuilt as needed.

C programming forgetfulness

There are a few nooks and crannies in C which I have to revise when I return to it.  StackOverflow covers them, once you eventually sort through the chaff there:

• A struct which can have pointers to other versions of the struct (link).
• Avoiding using void * pointer in list nodes (link).
• Writing cleaner loops with the comma operator (link).
• Recursive macro expansion and the # and ## operators (link).

I did a quick google for a wiki or web pages that summarised similar tips, but wasn't able to find anything suitable.

Finding help from StackOverflow is no longer as easy as it once was.  When googling for information, I now have to dig through sometimes dozens of low quality question pages.  Often the useful answer is not the accepted one, or there would be a secondary lower-voted answer which would link to a useful answer given to an earlier similar question.  Of course, it is still the best place for answers, but I think that I need to find a better way of finding them.