Mixing iteration and read methods would lose data

I just upgraded to Python 2.7.12 and my code is now erroring with rather unhelpful Mixing iteration and read methods would lose data errors. Googling for this yields a number of StackOverflow posts where people are iterating over a file, and then within the loop, doing calls on the file. This sounds like a good error for their case, but in my case the fixes are somewhat more inane.

This errors:

    persistence.write_dict_uint32_to_list_of_uint32s(output_file, { k: [] for k in read_set_of_uint32s(input_file) })

This fixes that error:

    set_value = persistence.read_set_of_uint32s(input_file)
    persistence.write_dict_uint32_to_list_of_uint32s(output_file, { k: [] for k in set_value })

Another error at another location:

    persistence.write_set_of_uint32s(output_file, persistence.read_set_of_uint32s(input_file))

Some simple rearranging makes this error go away:

    set_value = persistence.read_set_of_uint32s(input_file)
    persistence.write_set_of_uint32s(output_file, set_value)

And finally, the next error at another location:

def read_string(f):
    s = ""
    while 1:
        v = f.read(1) # errors here
        if v == '\0':
            break
        s += v
    return s

And the fix for this:

def read_bytes(f, num_bytes):
    return f.read(num_bytes)

def read_string(f):
    s = ""
    while 1:
        v = read_bytes(f, 1)
        if v == '\0':
            break
        s += v
    return s

This error seems like a horribly broken idea, that is popping up in places it wasn't intended.

Intellisense Bugs

I've posted about this before, but my use of C in Visual Studio isn't supported by Intellisense, however it is supported by the compiler.  This means that my code compiles without warning or error, but it's more difficult to edit as the editor falsely claims there are errors.

So I sent a report to Microsoft via their easy to use customer feedback menu:

Some time later, perhaps several weeks, I received email from a program manager on the Visual C++ team asking for further detail.  I compiled two cases, both in the same narrow usage of C, and send them in.

Here's one related to struct initialisation.

And another related to anonymous unions.

I don't hold out hope that Microsoft will fix them any time soon, or even fix them at all.  Their focus is on meeting more recent C++ standards, from reading their blog posts, and this is a niche part of their C support.

But fingers crossed.

MSVC2013, Intellisense and anonymous struct/union bugs

One of the improvements in the C11 standard, is the addition of anonymous structs and unions.  This means that the unnamed element flattens, and all members of the struct or union can be referred to at the same level as the struct or union itself (as long as there are no naming conflicts).

A simplified example use I make of an anonymous struct inside an anonymous union is:

struct my_event_s {
    enum my_event_types event_type_id;
    union {
        struct {
            enum my_key_event_types key_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
        struct {
            enum my_mouse_event_types mouse_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
        struct {
            enum my_touch_event_types touch_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
    };
};

It's a pleasure to program with. But then you reach the stage where you have a different struct, and you want to incorporate a shared anonymous struct (and likely also union) in that. In my case, it was my complete key struct, excerpted from above:

struct {
    enum my_key_event_types key_event_type_id;
    union {
    /* Further fields here.. */
    };
};

One thing you might try, is what I tried. Give the struct a tag, and than just place an anonymous reference to it in the encompassing struct or union.

struct my_key_event_s {
    enum my_key_event_types key_event_type_id;
    union {
    /* Further fields here.. */
    };
};

struct my_event_s {
    enum my_event_types event_type_id;
    union {
        struct my_key_event_s;
        struct {
            enum my_mouse_event_types mouse_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
        struct {
            enum my_touch_event_types touch_event_type_id;
            union {
            /* Further fields here.. */
            };
        };
    };
};

This compiles cleanly, but intellisense will choke on it, and find errors in the code. Intellisense will accept an anonymous struct or union member defined this way within a struct itself. But the moment you try it within a union, then it will choke on not only the struct definitions it accepted up until that point, it will also choke on the union definitions.

It turns out that this is a non-standard (not part of the official C11 standard) feature that Microsoft supports.  It might be argued that usage of it should be avoided, because of this, but with GCC supporting it via the -fms-extensions flag, it is pretty portable.

Personally, I see anonymous structs and unions as a great addition to the C language, that reduce friction without introducing complexity.  And this Microsoft extension should have been a natural part of the standard.  The only downside to it, is that it's actually almost unusable when you're using Microsoft's Visual Studio!  If intellisense chokes on this, then a lot of the value of the editor is undermined, as it's help becomes misleading.

For the record, I'm using Visual Studio 2013, community edition, update 4.  There's a new release of Visual Studio 2015 coming up, which will hopefully mean fixes for this (but who really knows?)  And ideally, a new release of the community edition.

stacklessemu

I use mobile broadband, and reducing bandwidth usage saves me a lot of money.  So when I wanted to play with pypes after hearing that Yahoo Pipes was being shut down, I somehow ended up taking Peter Szabo's greenstackless.py module, improving it a little, and putting it on a package up on PyPI.

It provides a good coverage of the Stackless API.  It's already been used as a backend for Peter Szabo's syncless project, and with the improvements I've made (adding the stackless.run() method, and fixing a greenlet/tasklet circular reference bug), it's even more capable.  But the devil is in the details.

It doesn't support the pre-emptive tasklet interruption that Stackless does, and can't as that is implemented via modifications to the internal Python VM source code.  Which is one of the few modifications Stackless makes in it's capacity as a fork of Python.  It is possibly possible to support this, as I believe Jeff Senn way back noted he had written a similar module with pre-emptive support based on the tracing hooks (with the downside that debuggers are unusable).

It doesn't support the threading model of Stackless Python.  This is derived by having a scheduler per thread, and most tasklets belonging to that scheduler at some level have slices of the stack of that thread and are therefore symbiotically attached to it.  It should be very possible to support this, with additional work, but that's something to set aside for when it's needed.

It doesn't support the module level properties, that Stackless implements.  And it can't unless it relies on hacks suggested in places like StackOverflow where classes are injected into 'sys.modules' to act as modules.

Future work I'm tempted to do is working out whether I can add the tealet project, as an alternative backend to the greenlet backend it currently has.  Greenlets are of course the stack slicing of Stackless extracted into an extension module.  tealet is the "next generation" version of the stack slicing intended for Stackless Python.  They're not using it already because it was such a wide ranging low level change to the workings of Stackless, that adopting it introduces potential instability which is not desirable at this time.

Kristjan Valur implemented a greenlet emulation module for tealet (in the same repo), so it may be that using it might be as simple as running stacklessemu on tealet.greenlets on tealet.  It's a pity it wasn't uploaded to pypi so I can try it out, but I don't think that's in Kristjan's wheelhouse.

Adding Curses support to Incursion

Incursion as long as I've been playing it, has always used a 2D graphical library to do it's graphics.  Before the open source release, it was Allegro, and after, it became libtcod which uses SDL2 behind the scenes.  But the problem with graphical libraries, is that the interface is presented on the screen as a static image.  And this means that people who can't see good, cannot play.

Would adding a terminal-based interface, like using curses, solve this problem? ...  Possibly.  Until it is using one, and a person who can't see good tries it out with their screen reader, I'm not sure it's possible to know for sure.

I use the premier operating system, Windows.  I know it's premier because the latest version does edgy shit, like cram a cheap looking tablet interface on the top of the new versions, and everyone not using a tablet gets a constrained and simple interface that doesn't do half of what it used to.  On Windows, we don't have a native terminal program that handles ANSI escape codes.  This is typically how terminals are implemented.  What we do have is a DOS console that does it's own special thing.

Shitty Windows 8 tablet UI

Fortunately, many many years ago, someone wrote a cross-platform curses implementation by the name of PDCurses, that has the bizarrely arcane and complex API that all curses programmers are familiar with, and that makes a DOS console look like it was doing all the nice stuff a Linux terminal window would otherwise do.  PDCurses is dated, but compiles easily, and integrates with little difficulty into existing projects.

The original PDCurses 3.4

Don't confuse it with the confusingly named PDCurses for "real" windows, which has had more recent work done on it.  This does not work in the DOS console, and opens a new window, where it does whatever it does.  Honestly, I have no idea whatever it is doing.  It maybe does it graphically?  The web page lists lots of changes, but isn't clear about whether this is a suitable replacement for the original PDCurses.

PDCurses for "real" windows

Besides allowing the tools (like screen readers) which people who can't see good use to access the console data, there are other advantages.  When people are playing in a Linux terminal, they can capture the data the terminal receives and displays.  This is an interleaved mix of ANSI escape codes and text.  It can be played back, to watch someone else playing.  On Windows, by using PDCurses, something like termrec polls the DOS console by the same Windows API that screen readers use, and recreates something resembling the best kind of low quality interleaved mix of ANSI escape codes and text that can be approximated from the data available.

I'm 90% of the way to implementing the Incursion curses UI.  It's completely untested, and based on Incursion's internal console implementation, it was likely the original implementation used curses.  There are few issues remaining before it can be tested.  Something that libraries like libtcod, SDL and Allegro do, is provide a higher level file system API.  This needs to be replaced.  And then there's the issue of how curses handles colours.  This is a complicated mess.  But I'm sure it's a solved problem that others can shine some light on, so I'll ask about it on the roguetemple forums.

At this point, I'm wondering if it would be better to implement a telnet server, and to have Incursion serve the stream of interleaved ANSI escape codes and text that Linux terminals expect.  Then anyone who plays it could use Putty to do so.

How did people ever complete Incursion?

Sometimes when I fix a bug, I wonder how people ever completed Incursion. Take this one for example.

The definition:

struct EventInfo {
    EventInfo& operator=(EventInfo &e) { /* Memory image copy for the int/bool/ptr members */
        // Implementation doing a quick copy, allocating instances of subobjects.
        return *this;
    }
    // Rest of the normal code for this struct.
}

One problem situation:

bool Magic::isTarget(EventInfo &_e, Thing *t) {
    EventInfo e = _e;
    // Rest of this function.
}

When isTarget exits, then _e is corrupted. This is because EventInfo only provides an assignment operator overload. However an assignment in a variable declaration uses the copy constructor, and in the absence of one, does a bitwise copy. Pointers to sub-objects then become shared, and the first version of EventInfo to be destroyed will destroy those sub-objects assuming itself to be the owner. In this case, it will be e which as a stack variable will be freed when the function returns.

Incursion is littered with perhaps a dozen cases where the bitwise copy constructor would have been employed. Each one a potential crash, if not source of corruption.

Good times!

planet-rldev

If you read this, and have a blog where you make posts about your work developing a roguelike, then consider submitting your blog for inclusion on Planet-RLDev.

Mysterious Incursion bug

In changes to Incursion back in August, I fixed compiler warnings related to casting. In theory, the changes should have been side-effect free. Testing of the specific lines of code, indicates this is the case. But people were reporting crashes related to the "room weights algorithm". In this case RM_Weights is an array of unsigned int, and c and tot are signed short.

    tot=0;
    for(i=0;RM_Weights[i*2+1];i++)
      tot += max(0,RM_Weights[i*2+1]);
    c = random(tot); 
    for(i=0;RM_Weights[i*2+1];i++)
      if (c < RM_Weights[i*2+1])
        goto RM_Chosen;
      else
        c -= RM_Weights[i*2+1];
    Fatal("Strange Error in room weights algorithm.");
RM_Chosen:
    RType = RM_Weights[i*2];
    RM_Weights[i*2+1] = -1;

I was unable to reproduce the crashes, although others reported doing so up to 15% of the time.

Were the crashes always there and were they merely hidden by all the other crashes, which have previously been fixed? Or are they caused by some unreproducible nuance of the casting? I'm guessing the former.

There's an obvious bug in the above logic. Points to whomever spots it without looking at my fixes on bitbucket.

Brogue v1.7.4

There's no public repository for Brogue, which means I maintain my own where I merge in each source release, which comes as part of each binary Brogue release.

Today I merged in v1.7.4.  It was a pretty straightforward merge, so should be an easy BrogueX release.

The only problem is that my BrogueX repositories are broken because of bad merging, where things have been merged in the wrong direction and I have no idea how to fix it besides throwing all my local repositories away and starting again.  It's strange how I keep doing this.

MIPS32 support

My satellite receiver has the odd bug, but otherwise it is generally a pretty good piece of hardware and software.  However, the bugs and the inability to fix them, are something I wish I could do something about.  The firmware, is written in MIPS assembly language, which runs on an Ali 3602 chip.  Scanning through it, you can see interesting things like the license for Linux-NTFS, something which was reported to the GPL violations mailing list several years back (with no action taken).

Anyway, I can't afford the main interactive disassembler out there, IDA.  So I have my own, which has a token amount of features comparatively.  Peasauce.  Up until now, it only disassembled m68k machine code, but I've just finished adding basic MIPS support.  It's nowhere near perfect, but it's a start.  And it shows how much work goes into IDA.

The next architecture is likely to be ARM support, although it like MIPS gets more complicated.  There's ARM and ARM thumb instructions, and they are different sized and mix together to some extent.  There's also MIPS32 and MIPS16, and they are different sized and mix together to some extent.  But that's a problem for another occasion.  The work on this could be endless, if I had the time.

IncursionScript & the Accent compiler compiler

When I started working with the Incursion source code, which Julian released, one of the problems was that not all the source code was available.  The bulk of the game data, both for d20 elements and dungeon generation, is specified in scripts.  These "IncursionScript" files are compiled into a module, which Incursion can have more than one of.

The problem was that they are compiled using the Accent compiler compiler.  And Julian had a version of this, which had been custom modified to produce C++ compatible code, but no longer had the source code.  I downloaded the original Accent source code and proceeded to try and get it working.

The Accent tool takes a ".acc" file, and transforms it into ".c" and ".h" files which you can compile against.  But it is also generated source code, created using some tool called Gentile.  The correct way to do this, which it appears Julian also avoided, would be to modify the Gentile grammar for Accent, and to generate a new version.  Instead, the approach we both took was to modify the generated Accent source code.

Accent is old school C code.  Seldom having return types in it's own source code, and rarely producing return types in the code it generates.  Occasionally using uninitialised stack variables, and expecting the value to be NULL.  And it generated argument typing for function declarations the other way, where rather than including the type with the argument name in the parenthesised list following the function name, the variable is redeclared fully (in the same way as stack variables) following that list and preceding the function body.

function(argname1, argname2, argname3)
    int argname1;
    char *argname2;
    void *argname3;
{

It is really the fixing of these three things that makes Accent C++ compatible, rather than making it output C++ specific elements.  In any case, a modified version of the Accent source code is now checked into the Incursion repository.  This is built as a dependency of the main Incursion project, ensuring that if there are any grammar changes, then they are rebuilt as needed.

Incursion stabilisation post 2

Suck-cess!  There's some problem with foreground and background colour not taking, but that's a minor detail.  Incursion is now basically playable with a libtcod backend.

Incursion stabilisation post 1

I've been working on replacing the use of Allegro in Incursion, as it only works in the Debug build.

Unfortunately, most of last night and today was spent trying to work out why at first calloc() was failing, returning NULL with an errno of 12 (NOMEM).  Luckily, after an eventual reboot that switched to the call to TCOD_console_set_custom_font() crashing on exit.

Finally, I noticed that I was passing 8 and 8 respectively, as my font's nb_char_horiz and nb_char_vertic values.  After correcting it, my code is now stable.  I debugged that function umpteen times, and am not sure how this makes that much of a difference.

Roguelike MUD progress #7 - Web-based client

Previous post: Roguelike MUD progress #6

I set a mini-goal for myself, to display a curses-based program on a web page.

The first step was to find a no frills, easy to adopt and extend Python websocket solution. One I've stumbled on  few times, is the simple websockets client/server gist. I used a slightly fixed version of it as a base. The next step was to find decent web-based terminal emulation. There are frameworks available which ditch the terminal window, but they come with larger problems.  The best candidate is term.js.

So taking the simple websockets simple_websocket_client.html and adding term.js, then modifying simple_websocket_server.py, it was easy to get a simple echo working with embedded ANSI control sequences.

First, I tried to proxy the input and output from the nano editor. Unfortunately, doing this with the subprocess module resulted in nano outputting a message about not supporting redirection.  There's probably a way to do this, but too much bother and it is not really my goal anyway.

Next, I went for something simpler.  I did a simple proxying of input and output to a telnet connection. Specifically my Roguelike MUD codebase.  There were some teething problems.  Text incoming from the browser-based terminal through the websocket, is UTF-8 encoded.  And similarly, outgoing text has to be as well.  But responses from the MUD contain both ANSI escape sequences and telnet negotiation sequences.  In order to work around this, I stripped the telnet negotiation out of outgoing text, and modified the relatively basic websocket code to also send binary frames.

A working connection:

In-game display:

As the picture above shows, the ANSI sequences are working perfectly.  But the character set does not seem to contain the line drawing characters.  My game interface already has debugging tools which display accessible character sets.

The main character sets:

As can be seen above, the only character set with even a hint of line drawing characters, is the one exposed through the ESC(0 ANSI sequence.  Most of these sequences are faked by term.js, where known character sets are mocked up from the underlying unicode-supporting font.

One solution is to do what I do with Putty, which is to identify the connected client, and if it's one which supports unicode, then switch unicode on using terminal control sequences.  Modifying term.js to respond to the ENQ character (0x05), and send it's terminal name (xterm) was easy enough.

This is what I should be seeing now:

This is what I am actually seeing:

The correct unicode characters are being sent, but not being displayed correctly by term.js.  I don't expect this will be that difficult to correct when I get some more time to work on this.

pysrd

One of my long-term projects is to programmatically generate game logic from the D&D 3.5 SRD.  The idea is that ideally if the required legwork was done, any time I wanted to prototype something around a game experience, it should be possible to plug in the processed data and be 98% of the way to having stock D&D gameplay.  There's a lot of hand-waving there around the phrase "required legwork", but that's not really important for the purpose of this post.

I've extracted the low level scripts I use to both extend the database and introspect it's contents, and published them on github under the project name pysrd.

The SRD is available from Wizards of the Coast as a set of RTF documents.  This of course, is not ideal, but fortunately it is possible to get the data from andargor.com in different database formats, including SQLite.  Unfortunately however, not all SRD data is included in these databases.  Fortunately, the OpenSRD project includes generated HTML versions of the RTF documents, which can more easily be processed.

Peasauce, interactive disassembler.. someday

From Resource to Peasauce

I have a list of hobby projects that I've always wanted to find the time and energy to work on, and one of these is an interactive disassembler.  Years ago when people used a type of computer called "the Amiga", it had an extremely usable interactive disassembler called Resource.  You would point it at a file and it would turn the file into assembly language code and data, after which you would interactively disassemble some more, and finally make it write out a file containing the assembly language source code.

Disassembled and commented code in Resource.

Recently I finally managed to start using some spare time to work on "Peasauce", and have made a pretty good start.  What it can currently do isn't really useful yet, given that editing a disassembled file or exporting it isn't implemented yet. But being able to select a file, see the disassembled results and then scroll around it qualifies as interactive disassembling in the loosest sense.  So it's a good start.

Disassembled code in Peasauce.

In order to help make the code more general, in addition to Amiga executable files, it also loads those from the Atari ST and X68000 platforms. I've never used either, but they're both based on the Motorola 680x0 CPU family so it's a minimal amount of extra work over and above parsing a few semi or undocumented file structures.

UI

I initially started usng the Tkinter, because it comes with the standard library.  Unfortunately, out of the box it just wasn't flexible enough to work in the way I wanted.  As an alternative, I first looked at PySide but wasn't able to locate documentation to make it easy to work out if it suited my needs.  The website looks like it is in a transitional phase and several links to useful sounding documentation were dead unfortunately.  This left wxPython, which I've used before.  As a fringe benefit of having a small play around with PySide, I first encountered crashes and then with Anselm Kruis' recent patch to Stackless Python was able verify this decade long ABI incompatibility was fixed and that the crashes went away.

With wxPython, I'm currently using the ListCtrl where it simply knows how many list items there are and requests list items as needed for display.  Strangely, as the number of list items increase it gets slower, but that's likely my fault for some reason. As a short term solution while there's minimal interactivity, it's good enough, but the time is almost right for something more dynamic.

The bisect module

I was iterating over ordered lists looking for the right entry, and this was profiling as the place most time was spent.  A quick browse of the Python documentation introduced me to a module I'd never heard of before, the bisect module.  By keeping two lists instead of one, bisect let me look up objects in the second list by passing it the first list.  That makes it sound rather complicated, a simplified piece of example code is a lot more helpful.

 lookup_index = bisect.bisect_left(lookup_keys, lookup_key)
 if lookup_keys[lookup_idx] != lookup_key:
  lookup_value = lookup_values[lookup_idx-1]
 else:
  lookup_value = lookup_values[lookup_idx]

While I haven't actually looked, writing this up has made me suspect that if I also bung judicial use of this in the list control virtual item fetching, it might even solve the slowness problem with larger files.  Actually going away and looking at the code confirms it.  I'll need another indexing list though, and to keep that synchronised with the other two.

Opensauce

My laptop is a little unreliable and addressing that isn't on the table, so a good way to ensure I don't lose the source code should anything happen, is to throw it up on github with everyone else's half completed projects where it can also sit mouldering and abandoned.  Or slowly be developed further, of course.

Link: Peasauce on github.

Android SDL 2 gesture tester app

I've written and uploaded a simple free app to the play store, which allows someone to create a selection of gestures, and then to draw and see how well they get matched (or indeed, if the right ones even get matched).

In an ideal world, I'd be able to use this to author things like the Google maps pinch to scale in and out, or swipes to go from one page to the next.  Unfortunately, while simple letter shapes like N, Z, C, W and more match perfectly fine, the functional gestures like pinch to scale and swipe to switch get confused and don't seem to reliably distinguish between the direction of the actual gesture.

Hopefully I'll get feedback from the mailing list, and it'll be something I am doing wrong.  Otherwise there's a load more work in my future to use the Android Java-level gestures.

EDIT (19/06/2013): Published source code of latest released version.

MUD development, wikis and useful info.

Only a few years ago, the various MUD forums were still pretty active.  These days, even the most recent popular forum MUDBytes, has stagnated.  If I had to speculate why, I think that in this day and age there is a larger variety of hobbiest programming activities to do.  And for that matter, a larger variety of general computer related distractions.  I know I have about 20 games I've bought from from either GOG or Steam, that I'd love to play but just can't summon the interest to do so.  I'm not saying MUDs are dying, text-based gaming will always have a niche -- all bets are off when the holodeck comes to fruition though..

Anyway.. I just wanted to post an interesting link which was posted to MUDBytes by Nathan.  It's a thread containing posts from Nick Gammon, and it basically enumerates a large number of the things that factor into the development of a decent game engine.  I know a  lot of them could be a summarisation of the code I wrote for an online 3D-based science-fiction outer-space situated game which has tens of thousands of concurrent players.  And I know a giant hunk of it was handed to me within the design for a system written by someone who previously worked on an online science-fiction first person game about a colony.

George, as you're the only person who reads this blog, imma gonna write this here instead of emailing you.  You've most likely read the same thread, but I think it's worth saying that this seems like a perfect piece of material for the wiki :-)

C programming forgetfulness

There are a few nooks and crannies in C which I have to revise when I return to it.  StackOverflow covers them, once you eventually sort through the chaff there:

• A struct which can have pointers to other versions of the struct (link).
• Avoiding using void * pointer in list nodes (link).
• Writing cleaner loops with the comma operator (link).
• Recursive macro expansion and the # and ## operators (link).

I did a quick google for a wiki or web pages that summarised similar tips, but wasn't able to find anything suitable.

Finding help from StackOverflow is no longer as easy as it once was.  When googling for information, I now have to dig through sometimes dozens of low quality question pages.  Often the useful answer is not the accepted one, or there would be a secondary lower-voted answer which would link to a useful answer given to an earlier similar question.  Of course, it is still the best place for answers, but I think that I need to find a better way of finding them.

Android development, SDL2 and crashes

Two more learning experiences with Android & SDL 2.0.

Crashes in native code

When your native code crashes, you get a PS3-style stacktrace with mystifying anonymous addresses representing each level of the call stack.  This is important because Eclipse cannot debug native code, it just exits debugging.  Much the same as on the PS3, you can run some tool on this stacktrace and get a readable form with file names and line numbers.  But you have to do it line by line manually, feeding addresses and static library paths into the tool.

This is a raw crash from the logcat output (with some duplicate lines trimmed):

08-01 19:36:56.899: I/DEBUG(31): Build fingerprint: 'generic/sdk/generic:2.3.3/GRI34/101070:eng/test-keys'
08-01 19:36:56.899: I/DEBUG(31): pid: 495, tid: 503  >>> org.disinterest.broguex <<<
08-01 19:36:56.899: I/DEBUG(31): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000020
08-01 19:36:56.899: I/DEBUG(31):  r0 00000000  r1 4419ac10  r2 ac411085  r3 00000001
08-01 19:36:56.899: I/DEBUG(31):  r4 0029e528  r5 002bdf60  r6 00000001  r7 43f37f3c
08-01 19:36:56.899: I/DEBUG(31):  r8 4419ac70  r9 43f37f34  10 00000000  fp 419507e8
08-01 19:36:56.899: I/DEBUG(31):  ip 441baf00  sp 4419ac10  lr ac411c93  pc ac411094  cpsr 00000030
08-01 19:36:57.139: I/DEBUG(31):          #00  pc 00011094  /system/lib/egl/libGLES_android.so
08-01 19:36:57.139: I/DEBUG(31):          #01  pc 00011c90  /system/lib/egl/libGLES_android.so
08-01 19:36:57.150: I/DEBUG(31):          #02  pc 00003844  /system/lib/libEGL.so
08-01 19:36:57.150: I/DEBUG(31):          #03  pc 00033004  /system/lib/libandroid_runtime.so
08-01 19:36:57.159: I/DEBUG(31):          #04  pc 00017d74  /system/lib/libdvm.so
08-01 19:36:57.159: I/DEBUG(31):          #05  pc 00048f54  /system/lib/libdvm.so
...
08-01 19:36:57.199: I/DEBUG(31):          #19  pc 000a3518  /data/data/org.disinterest.broguex/lib/libSDL2.so
08-01 19:36:57.199: I/DEBUG(31):          #20  pc 000a348e  /data/data/org.disinterest.broguex/lib/libSDL2.so
08-01 19:36:57.199: I/DEBUG(31):          #21  pc 0003142a  /data/data/org.disinterest.broguex/lib/libtcod.so
08-01 19:36:57.199: I/DEBUG(31):          #22  pc 00032e8e  /data/data/org.disinterest.broguex/lib/libtcod.so
08-01 19:36:57.209: I/DEBUG(31):          #23  pc 000112be  /data/data/org.disinterest.broguex/lib/libtcod.so
08-01 19:36:57.209: I/DEBUG(31):          #24  pc 0008e066  /data/data/org.disinterest.broguex/lib/libmain.so
...
08-01 19:36:57.219: I/DEBUG(31):          #31  pc 00029582  /data/data/org.disinterest.broguex/lib/libmain.so
08-01 19:36:57.219: I/DEBUG(31): code around pc:

Feeding each line into the tool is somewhat tedious and error prone.  Fortunately Ben van Daele has written a Python script to parse this raw data into a proper call stack.  With a few modifications, it even works on Windows and gives the output similar to the following:

d:/Code/android-ndk-r8/platforms/android-14/arch-arm/usr/include/jni.h:793:_JNIEnv::CallStaticVoidMethod(_jclass*, _jmet
hodID*, ...)
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/core/android/SDL_android.cpp:273:Android_JNI_SwapWindow
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/video/android/SDL_androidgl.c:108:Android_GL_SwapWindow
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/video/SDL_video.c:2592:SDL_GL_SwapWindow
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/render/opengles/SDL_render_gles.c:1062:GLES_RenderPresent
...
D:\Code\games\Brogue-v1.6.4-Android/jni/brogue/BrogueCode/RogueMain.c:38:executeEvent
D:\Code\games\Brogue-v1.6.4-Android/jni/brogue/BrogueCode/IO.c:637:mainInputLoop

My DOS window compatible version of the script, is available here.

Android rendering crashes

There's a page on the SDL wiki which says (or said, being unable to find it now) that as the SDL_APPACTIVE app state no longer exists, you should instead check the SDL_WINDOW_SHOWN window flag.  Unfortunately, on Android at least this is set even when the SDL_WINDOW_MINIMIZED flag is set.  Which might be when you hit the home button.  And if you try and render while your window is minimised, crashes ensue due to an invalid surface.

Something like this will precede one of those anonymous stacktraces:

09-20 01:59:06.702: ERROR/Surface(20116): surface (identity=...) is invalid, err=-19 (No such device)

This is what I use to check that it is okay to engage in normal rendering-related logic:

#if SDL_VERSION_ATLEAST(2,0,0) && defined(__ANDROID__)
 /* Abort if the window no longer has focus.  Otherwise a later draw operation will crash the app. */
 if ((SDL_GetWindowFlags((SDL_Window*)TCOD_sys_get_sdl_window()) & (SDL_WINDOW_SHOWN | SDL_WINDOW_MINIMIZED)) != SDL_WINDOW_SHOWN)
  return true;
#endif

I still need to properly handle pausing, and of course the special kind of pausing where the application gets killed and needs to persist state. But it's enough for now not to crash if the application is paused and resumed.

Supporting all Android architectures

Charilaos Kalogirou's post about porting your game from iOS to Android, recommends not just building code for the default ARMv5TE architecture, but all supported architectures.  To do this, in your "jni/" directory add the file "Application.mk" if it doesn't already exist.  And within this add the line "APP_ABI := all".  This will build all architectures, which according to the NDK documentation is ARMv5TE, ARMv7-A, x86 and MIPS.

You don't need to do anything else.  The presence of the binaries for the different architectures just works, and the Play store will now offer the app to those whose device was previously unsupported.  It did double the APK size of 1.1M to 2.2M, with MIPS architecture excluded for SDL-specific reasons.

The Play store does allow uploading multiple APK files, and filtering which of these is offered to different potential recipients, based on their differences.  But none of the filters allows you to upload a different APK per-architecture.

Getting more familiar with SDL 2.0

I'm a little confused by SDL 2.0.  On one hand, it has made getting libtcod working on Android very easy.  On the other hand, there is a range of functionality that was present in 1.2, which is.. vaguely alluded to as no longer being considered part of SDL and up to any end user to reimplement themselves.

One of these is key repeat rates.  SDL 1.2 had an API that let you customise this.  SDL 2.0 has a mailing list post suggesting that any end user wanting to modify it, should use the OS facilities.  Android does not have any OS facility to modify this.

Another of these is mapping key events to the character on the key the user actually pressed.  What I get as a developer when I press "~", is the keycode for "`" with a modifier indicating a shift key was being pressed at the time.  I get this on Windows, and I get this on Android.  SDL 1.2 had the unicode field, which was populated with the correct character through platform-specific logic that queried the keyboard mapping or code page, or whatever.  SDL 2.0 has the unicode field, but it never gets a value set to it, so remains 0.  It however has a comment indicating it is obsolete and that the text input API should be used.  Now, the text input API is a completely different non-key press based approach.  If you're writing a text adventure, then it's probably what you're looking for.  And if you are writing an arcade game, then the current SDL key press support is probably okay.  But if you want to have more than the limited SDL key press support, you need to (on Android at least) completely bypass the SDL event system and it's KeyEvent.  This is easy to do, but it feels like SDL is incomplete and that you're violating proper procedure.