Sunday 11 November 2012

Peasauce, interactive disassembler.. someday

From Resource to Peasauce

I have a list of hobby projects that I've always wanted to find the time and energy to work on, and one of these is an interactive disassembler.  Years ago when people used a type of computer called "the Amiga", it had an extremely usable interactive disassembler called Resource.  You would point it at a file and it would turn the file into assembly language code and data, after which you would interactively disassemble some more, and finally make it write out a file containing the assembly language source code.

Disassembled and commented code in Resource.
Recently I finally managed to start using some spare time to work on "Peasauce", and have made a pretty good start.  What it can currently do isn't really useful yet, given that editing a disassembled file or exporting it isn't implemented yet. But being able to select a file, see the disassembled results and then scroll around it qualifies as interactive disassembling in the loosest sense.  So it's a good start.

Disassembled code in Peasauce.
In order to help make the code more general, in addition to Amiga executable files, it also loads those from the Atari ST and X68000 platforms. I've never used either, but they're both based on the Motorola 680x0 CPU family so it's a minimal amount of extra work over and above parsing a few semi or undocumented file structures.

UI

I initially started usng the Tkinter, because it comes with the standard library.  Unfortunately, out of the box it just wasn't flexible enough to work in the way I wanted.  As an alternative, I first looked at PySide but wasn't able to locate documentation to make it easy to work out if it suited my needs.  The website looks like it is in a transitional phase and several links to useful sounding documentation were dead unfortunately.  This left wxPython, which I've used before.  As a fringe benefit of having a small play around with PySide, I first encountered crashes and then with Anselm Kruis' recent patch to Stackless Python was able verify this decade long ABI incompatibility was fixed and that the crashes went away.

With wxPython, I'm currently using the ListCtrl where it simply knows how many list items there are and requests list items as needed for display.  Strangely, as the number of list items increase it gets slower, but that's likely my fault for some reason. As a short term solution while there's minimal interactivity, it's good enough, but the time is almost right for something more dynamic.

The bisect module

I was iterating over ordered lists looking for the right entry, and this was profiling as the place most time was spent.  A quick browse of the Python documentation introduced me to a module I'd never heard of before, the bisect module.  By keeping two lists instead of one, bisect let me look up objects in the second list by passing it the first list.  That makes it sound rather complicated, a simplified piece of example code is a lot more helpful.
 lookup_index = bisect.bisect_left(lookup_keys, lookup_key)
 if lookup_keys[lookup_idx] != lookup_key:
  lookup_value = lookup_values[lookup_idx-1]
 else:
  lookup_value = lookup_values[lookup_idx]
While I haven't actually looked, writing this up has made me suspect that if I also bung judicial use of this in the list control virtual item fetching, it might even solve the slowness problem with larger files.  Actually going away and looking at the code confirms it.  I'll need another indexing list though, and to keep that synchronised with the other two.

Opensauce

My laptop is a little unreliable and addressing that isn't on the table, so a good way to ensure I don't lose the source code should anything happen, is to throw it up on github with everyone else's half completed projects where it can also sit mouldering and abandoned.  Or slowly be developed further, of course.

Link: Peasauce on github.

Saturday 10 November 2012

Extracting files from X68000 disk images

For testing purposes, I need some executable files from different m68k-based platforms.  One of these is the X68000, which like most old disk-based systems, mostly has downloads available as disk images in some obscure format.  There's a range of recommended ways of working with X68000 ".dim" images, but unfortunately I wasted 10 minutes on them before stumbling on this better way.

The first step is to download Virtual Floppy Image Converter.  Then simply run it, decide whether you wish to use it in Japanese or English.  At that point, it is rather straightforward.

Change the FORMAT "BKDSK FORMAT" setting to "FILE EXTRACT".


Simply drag the ".dim" disk image into the text area below, which instructs you to do as much.  You'll see a sub-directory get created in the same directory the disk image came from.


Apparently some of these games came with source code.


This was pretty painless as far as these sorts of technical tasks go, thanks to Virtual Floppy Disk Converter.

Wednesday 12 September 2012

Android SDL 2 gesture tester app

I've written and uploaded a simple free app to the play store, which allows someone to create a selection of gestures, and then to draw and see how well they get matched (or indeed, if the right ones even get matched).


In an ideal world, I'd be able to use this to author things like the Google maps pinch to scale in and out, or swipes to go from one page to the next.  Unfortunately, while simple letter shapes like N, Z, C, W and more match perfectly fine, the functional gestures like pinch to scale and swipe to switch get confused and don't seem to reliably distinguish between the direction of the actual gesture.

Hopefully I'll get feedback from the mailing list, and it'll be something I am doing wrong.  Otherwise there's a load more work in my future to use the Android Java-level gestures.

EDIT (19/06/2013): Published source code of latest released version.

Sunday 9 September 2012

Android, libtcod & gesture support

I've the beginnings of a gesture experimentation prototype running on Android within the libtcod sample program.  Currently it just allows the authoring of a gesture, but it's fairly straightforward from there to have the user author several gestures, then draw something and see which gesture SDL detects that it should be.


The SDL 2.0 gesture support works wonderfully.  But unfortunately a little too wonderfully, even unknown gestures are matched and the aspect that would distinguish a bad match, seems to be broken.  Currently every matched gesture has an error of -2.0.  Something I apparently have to debug myself, using printf-based debugging as proper debugging does not work at the C level.

More to come..

Wednesday 15 August 2012

MUD development, wikis and useful info.

Only a few years ago, the various MUD forums were still pretty active.  These days, even the most recent popular forum MUDBytes, has stagnated.  If I had to speculate why, I think that in this day and age there is a larger variety of hobbiest programming activities to do.  And for that matter, a larger variety of general computer related distractions.  I know I have about 20 games I've bought from from either GOG or Steam, that I'd love to play but just can't summon the interest to do so.  I'm not saying MUDs are dying, text-based gaming will always have a niche -- all bets are off when the holodeck comes to fruition though..

Anyway.. I just wanted to post an interesting link which was posted to MUDBytes by Nathan.  It's a thread containing posts from Nick Gammon, and it basically enumerates a large number of the things that factor into the development of a decent game engine.  I know a  lot of them could be a summarisation of the code I wrote for an online 3D-based science-fiction outer-space situated game which has tens of thousands of concurrent players.  And I know a giant hunk of it was handed to me within the design for a system written by someone who previously worked on an online science-fiction first person game about a colony.

George, as you're the only person who reads this blog, imma gonna write this here instead of emailing you.  You've most likely read the same thread, but I think it's worth saying that this seems like a perfect piece of material for the wiki :-)

Friday 3 August 2012

C programming forgetfulness

There are a few nooks and crannies in C which I have to revise when I return to it.  StackOverflow covers them, once you eventually sort through the chaff there:

  • A struct which can have pointers to other versions of the struct (link).
  • Avoiding using void * pointer in list nodes (link).
  • Writing cleaner loops with the comma operator (link).
  • Recursive macro expansion and the # and ## operators (link).
I did a quick google for a wiki or web pages that summarised similar tips, but wasn't able to find anything suitable.

Finding help from StackOverflow is no longer as easy as it once was.  When googling for information, I now have to dig through sometimes dozens of low quality question pages.  Often the useful answer is not the accepted one, or there would be a secondary lower-voted answer which would link to a useful answer given to an earlier similar question.  Of course, it is still the best place for answers, but I think that I need to find a better way of finding them.

Android development, SDL2 and crashes

Two more learning experiences with Android & SDL 2.0.

Crashes in native code

When your native code crashes, you get a PS3-style stacktrace with mystifying anonymous addresses representing each level of the call stack.  This is important because Eclipse cannot debug native code, it just exits debugging.  Much the same as on the PS3, you can run some tool on this stacktrace and get a readable form with file names and line numbers.  But you have to do it line by line manually, feeding addresses and static library paths into the tool.

This is a raw crash from the logcat output (with some duplicate lines trimmed):

08-01 19:36:56.899: I/DEBUG(31): Build fingerprint: 'generic/sdk/generic:2.3.3/GRI34/101070:eng/test-keys'
08-01 19:36:56.899: I/DEBUG(31): pid: 495, tid: 503  >>> org.disinterest.broguex <<<
08-01 19:36:56.899: I/DEBUG(31): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000020
08-01 19:36:56.899: I/DEBUG(31):  r0 00000000  r1 4419ac10  r2 ac411085  r3 00000001
08-01 19:36:56.899: I/DEBUG(31):  r4 0029e528  r5 002bdf60  r6 00000001  r7 43f37f3c
08-01 19:36:56.899: I/DEBUG(31):  r8 4419ac70  r9 43f37f34  10 00000000  fp 419507e8
08-01 19:36:56.899: I/DEBUG(31):  ip 441baf00  sp 4419ac10  lr ac411c93  pc ac411094  cpsr 00000030
08-01 19:36:57.139: I/DEBUG(31):          #00  pc 00011094  /system/lib/egl/libGLES_android.so
08-01 19:36:57.139: I/DEBUG(31):          #01  pc 00011c90  /system/lib/egl/libGLES_android.so
08-01 19:36:57.150: I/DEBUG(31):          #02  pc 00003844  /system/lib/libEGL.so
08-01 19:36:57.150: I/DEBUG(31):          #03  pc 00033004  /system/lib/libandroid_runtime.so
08-01 19:36:57.159: I/DEBUG(31):          #04  pc 00017d74  /system/lib/libdvm.so
08-01 19:36:57.159: I/DEBUG(31):          #05  pc 00048f54  /system/lib/libdvm.so
...
08-01 19:36:57.199: I/DEBUG(31):          #19  pc 000a3518  /data/data/org.disinterest.broguex/lib/libSDL2.so
08-01 19:36:57.199: I/DEBUG(31):          #20  pc 000a348e  /data/data/org.disinterest.broguex/lib/libSDL2.so
08-01 19:36:57.199: I/DEBUG(31):          #21  pc 0003142a  /data/data/org.disinterest.broguex/lib/libtcod.so
08-01 19:36:57.199: I/DEBUG(31):          #22  pc 00032e8e  /data/data/org.disinterest.broguex/lib/libtcod.so
08-01 19:36:57.209: I/DEBUG(31):          #23  pc 000112be  /data/data/org.disinterest.broguex/lib/libtcod.so
08-01 19:36:57.209: I/DEBUG(31):          #24  pc 0008e066  /data/data/org.disinterest.broguex/lib/libmain.so
...
08-01 19:36:57.219: I/DEBUG(31):          #31  pc 00029582  /data/data/org.disinterest.broguex/lib/libmain.so
08-01 19:36:57.219: I/DEBUG(31): code around pc:

Feeding each line into the tool is somewhat tedious and error prone.  Fortunately Ben van Daele has written a Python script to parse this raw data into a proper call stack.  With a few modifications, it even works on Windows and gives the output similar to the following:
d:/Code/android-ndk-r8/platforms/android-14/arch-arm/usr/include/jni.h:793:_JNIEnv::CallStaticVoidMethod(_jclass*, _jmet
hodID*, ...)
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/core/android/SDL_android.cpp:273:Android_JNI_SwapWindow
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/video/android/SDL_androidgl.c:108:Android_GL_SwapWindow
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/video/SDL_video.c:2592:SDL_GL_SwapWindow
D:\Code\games\Brogue-v1.6.4-Android/jni/sdl2/src/render/opengles/SDL_render_gles.c:1062:GLES_RenderPresent
...
D:\Code\games\Brogue-v1.6.4-Android/jni/brogue/BrogueCode/RogueMain.c:38:executeEvent
D:\Code\games\Brogue-v1.6.4-Android/jni/brogue/BrogueCode/IO.c:637:mainInputLoop

My DOS window compatible version of the script, is available here.

Android rendering crashes

There's a page on the SDL wiki which says (or said, being unable to find it now) that as the SDL_APPACTIVE app state no longer exists, you should instead check the SDL_WINDOW_SHOWN window flag.  Unfortunately, on Android at least this is set even when the SDL_WINDOW_MINIMIZED flag is set.  Which might be when you hit the home button.  And if you try and render while your window is minimised, crashes ensue due to an invalid surface.

Something like this will precede one of those anonymous stacktraces:
09-20 01:59:06.702: ERROR/Surface(20116): surface (identity=...) is invalid, err=-19 (No such device)
This is what I use to check that it is okay to engage in normal rendering-related logic:
#if SDL_VERSION_ATLEAST(2,0,0) && defined(__ANDROID__)
 /* Abort if the window no longer has focus.  Otherwise a later draw operation will crash the app. */
 if ((SDL_GetWindowFlags((SDL_Window*)TCOD_sys_get_sdl_window()) & (SDL_WINDOW_SHOWN | SDL_WINDOW_MINIMIZED)) != SDL_WINDOW_SHOWN)
  return true;
#endif
I still need to properly handle pausing, and of course the special kind of pausing where the application gets killed and needs to persist state. But it's enough for now not to crash if the application is paused and resumed.

Supporting all Android architectures

Charilaos Kalogirou's post about porting your game from iOS to Android, recommends not just building code for the default ARMv5TE architecture, but all supported architectures.  To do this, in your "jni/" directory add the file "Application.mk" if it doesn't already exist.  And within this add the line "APP_ABI := all".  This will build all architectures, which according to the NDK documentation is ARMv5TE, ARMv7-A, x86 and MIPS.

You don't need to do anything else.  The presence of the binaries for the different architectures just works, and the Play store will now offer the app to those whose device was previously unsupported.  It did double the APK size of 1.1M to 2.2M, with MIPS architecture excluded for SDL-specific reasons.


The Play store does allow uploading multiple APK files, and filtering which of these is offered to different potential recipients, based on their differences.  But none of the filters allows you to upload a different APK per-architecture.

Getting more familiar with SDL 2.0

I'm a little confused by SDL 2.0.  On one hand, it has made getting libtcod working on Android very easy.  On the other hand, there is a range of functionality that was present in 1.2, which is.. vaguely alluded to as no longer being considered part of SDL and up to any end user to reimplement themselves.

One of these is key repeat rates.  SDL 1.2 had an API that let you customise this.  SDL 2.0 has a mailing list post suggesting that any end user wanting to modify it, should use the OS facilities.  Android does not have any OS facility to modify this.

Another of these is mapping key events to the character on the key the user actually pressed.  What I get as a developer when I press "~", is the keycode for "`" with a modifier indicating a shift key was being pressed at the time.  I get this on Windows, and I get this on Android.  SDL 1.2 had the unicode field, which was populated with the correct character through platform-specific logic that queried the keyboard mapping or code page, or whatever.  SDL 2.0 has the unicode field, but it never gets a value set to it, so remains 0.  It however has a comment indicating it is obsolete and that the text input API should be used.  Now, the text input API is a completely different non-key press based approach.  If you're writing a text adventure, then it's probably what you're looking for.  And if you are writing an arcade game, then the current SDL key press support is probably okay.  But if you want to have more than the limited SDL key press support, you need to (on Android at least) completely bypass the SDL event system and it's KeyEvent.  This is easy to do, but it feels like SDL is incomplete and that you're violating proper procedure.

Friday 27 July 2012

BrogueX released

I've published my port of Brogue as an Android paid app.  It's basically playable with a little difficulty, and requires a little work to fix bugs, and a lot of work to enhance the Brogue libtcod-based UI for tablets and mobile devices.

All the details are on the link above.  Buying it supports the work required to make this a first class Android experience, and you get to direct the efforts involved should you choose.

Tuesday 24 July 2012

Android development, Eclipse and SDL2

I've started playing with developing an Android application.  Here are my notes for future reference.

Debugging emulated C code in Eclipse

If I was developing in Java, Eclipse would be an easy choice.  When a breakpoint is hit or an exception happens within the Android emulator, execution will pause at the given point in whatever code is executing.  But only for Java.  If I step into C code, the debugger simply proceeds to run the program.  If the C code crashes, the debugger exits.  The only help is the LogCat window, and printf driven debugging.


No amount of web searching has located anyone else having this problem.

Eclipse's pre-compilation code checker gets in the way

Eclipse has this feature called Code Analysis.  It appears to be an editor level tool that shows you errors in your code before you actually compile.  The only problem is that it is unable to properly parse C code.  This means that not only do you get the wrong parts of preprocessor directive #if/#ifdef clauses being highlighted, you get officially lodged errors you have to fix before you are allowed to do things like debugging your code - even though it was just compiled without error!

Web searching reveals people who suggest clearing the analysis results seemingly every time you make a code change.  Others suggest rerunning the indexer.  Then there are suggestions to manually enter the mis-indexed symbols in a settings tab which does not exist for me.  And finally, some people hack the Eclipse project to specify the symbol, to get it to work.

I ended up disabling this feature via the Window->Preferences menu, within the C/C++->Code Analysis pane.  It sounds like it just isn't reliable enough to spend the time required to get it to work.


Android file access and SDL

An Android application is packaged in a zip archive, suffixed with ".apk".  One of the your directories, named "assets", is placed in this archive and can be accessed in a read-only fashion.  To access this, you need to make calls into the Java VM and your application object running there.  You grab an AssetManager, and from there can make calls on it to list whatever files and subdirectories are present within this directory, and of course can open and read from them.

Luckily, I am using SDL2 and can avoid this by just using the SDL_rwops abstraction layer.  This is not very well documented like much of the still in-development SDL2, but only provides access to the asset manager, and the assets directory within the apk file.  File system access I do not have much experience with, and still need to research.  But it looks like the standard C file-related functions are used.

Android asset access and libpng

I have a png image within my apk file, but libpng internally uses standard C file IO functions.  Working around this was a simple matter of taking advantage of the ability to specify read and write callbacks to libpng, which are used in place of the standard functions.
static void png_read_data(png_structp png_ptr, png_bytep data, png_size_t length) {
 SDL_RWops *rwops = png_get_io_ptr(png_ptr);
 rwops->read(rwops,data,length,1);
}
static void png_write_data(png_structp png_ptr, png_bytep data, png_size_t length) {
 SDL_RWops *rwops = png_get_io_ptr(png_ptr);
 rwops->write(rwops,data,length,1);
}
static void png_flush_data(png_structp png_ptr) {}

/* ... */

 SDL_RWops *rwops;
 if ((rwops = SDL_RWFromFile(filename, "rb")) == NULL)

 png_set_read_fn(png_ptr, (voidp)rwops, (png_rw_ptr)png_read_data);

There are some macros I should be using, which allow me to get rid of the rwops->opfunc(rwops pattern.  And what happens if I want to read or write png files from external storage, is another complication yet to be addressed.

Emulator logging, printf and Eclipse's LogCat tab

If I write to standard output from your application, it effectively goes nowhere by the looks of it.  So as advised by web searching, I use Android logging functions like __android_log_print whose output appears in the LogCat window in Eclipse  Better yet, I just bung these defines in a global include file -- which every project seems to have.

#ifdef __ANDROID__
#include <android/log.h>
#ifdef printf
#undef printf
#endif
#ifdef vprintf
#undef vprintf
#endif
#define printf(args...) __android_log_print(ANDROID_LOG_INFO, "libtcod", ## args)
#define vprintf(args...) __android_log_vprint(ANDROID_LOG_INFO, "libtcod", ## args)

#ifdef assert
#undef assert
#endif
#define assert(cond) if(!(cond)) __android_log_assert(#cond, "libtcod", "assertion failed: %s", #cond)
#endif
And I can just proceed to observe the existing printf calls in the LogCat tab in Eclipse.  I don't know why the Android assert function takes in the condition as a string for the first argument, as it doesn't appear in the Eclipse LogCat tab.

SDL2 also has logging functions which on Android, end up going through the Android logging API.  But as my application already has pre-existing printf calls, and I don't need the complication of an extended logging system, whatever.

Touch events and screen coordinates

Touch events have x and y coordinates.  But these are relative to the resolution of the touch device, and are not screen coordinates as events normally give.  It is an easy matter to fetch the device resolution and to scale it to screen coordinates.
 SDL_TouchFingerEvent *tfe=&ev->tfinger;
 SDL_Touch *touch=SDL_GetTouch(tfe->touchId);
 dx += (tfe->dx * windowWidth)/touch->xres;
 dy += (tfe->dy * windowHeight)/touch->yres;
 x = (tfe->x * windowWidth)/touch->xres;
 y = (tfe->y * windowHeight)/touch->yres;

Android "offline" SDK documentation

Being on Windows, I downloaded the SDK documentation through the SDK Manager program which comes as part of the SDK.  However, it has hard-coded references to Google javascript and font related web sites.  This means I get is a 10-20 sec loading delay every time I bring it up in a browser, or click a link within it to go to another page.

Android application's initial cwd

Porting an existing program to an Android application, being based on SDL it generally just works.  But what does cause crashes is the presumption that the current working directory is somewhere files can be stored.  What is the initial cwd for an Android application?

#include <errno.h> /* errno */
#include <unistd.h> /* getcwd */
#include <dirent.h> /* opendir / readdir */

/* ... */

 char cwdpath[100];
 DIR* pd;
 struct dirent *cur;
 if (getcwd(cwdpath, 99) == NULL) {
  printf("getcwd() failed and gave: %d\n", errno);
 } else {
  printf("getcwd() gave: %s\n", cwdpath);
  pd = opendir(cwdpath);
  if (pd == NULL) {
   printf("opendir() failed and gave: %d\n", errno);
  } else {
   while (cur = readdir(pd)) {
    printf("\"%s\"\n", cur->d_name);
   }
   closedir(pd);
  }
 }
Results in LogCat rows containing:
07-21 14:30:50.227: I/Brogue(923): getcwd() gave: /
07-21 14:30:50.227: I/Brogue(923): "."
07-21 14:30:50.227: I/Brogue(923): ".."
07-21 14:30:50.227: I/Brogue(923): "config"
This makes sense.  The install location of the apk file is not really relevant.  So the developer needs to query where to store these files, which may be either on internal or external storage.

SDL2 hard-coded namespace

Java code is referred to by namespace, and so JNI calls into C are made on the naming of that namespace.  This is a problem with SDL2 because the implementation of SDL2 relies on the use of the namespace "org.libsdl.app".  Anyone publishing their own app would want to do so in their own namespace, and if you change all the namespace references (whether in code, directory structure or configuration files), the JNI calls try and find the SDL C methods using your namespace.  This means that the internal SDL functions don't get called, and your app doesn't load due to the first UnresolvedLinkError.

This forum thread discusses it, and this blog post provides a file you can link against (with changes), in order to work around this problem. The changes required are replacing the use of this namespace with your own, and also verifying that all the SDL methods are wrapped.  I had to add several more.

Sunday 15 July 2012

Compiling Brogue v1.6.4 with MinGW

Brogue is a very user friendly roguelike.  It implements a lot of the things that I have always wanted whenever I play a "tile-based" text game of this kind.  That includes support for mouse over events, and giving visual indications of where on the map the selected item is, and the like..



It comes with source code, which should allow me to make my own changes, but unfortunately with programming.. it's never that simple.
In an ideal world, I could just create a new project in Visual Studio, but the code is not platform independent and requires dirent.h and associated functionality.  太麻烦了... so MinGW it is.  The option of using the Code Blocks IDE and it's version of MingW is present, but that's just more complication of a different kind.
  1. Have a MinGW/msys compilation environment set up.
  2. Open a MinGW shell.
  3. Go to the "Brogue Source" directory that comes with the Windows Brogue binaries.
  4. Download and extract: libtcod-1.5.1-mingw32.zip (from here).  It is already compiled.
  5. Download and extract: SDL-devel-1.2.14-mingw32.tar.gz (from here).  It is not necessary to compile it as the headers are all that are required.
  6. Apply patch below for "tcod-platform.c".
  7. Edit "makefile".
  8. Line 3: Correct CFLAGS with -DBROGUE_TCOD instead of -DBROGUE-TCOD.
  9. Line 23: Add the missing "line continuation" character '\'.
  10. Line 32: Correct the g++ linking command modifying "-lSDL.dll" to be "-lSDL".
  11. Line 32: Correct the g++ linking command adding "-L$(LIBTCODDIR)/" so that g++ can find "SDL.dll" and "libtcod-mingw.dll".
  12. Line 32:  Extend the g++ linking command adding "-static-libgcc -static-libstdc++" so that the compiled executable can be run without a dialog box saying "The program can't start because libgcc_s_dw2-1.dll is missing from you computer. Try reinstalling the program to fix this problem.".
This gives you a replacement "brogue.exe" binary which can be copied up to the outer "Brogue v1.6.4" directory, and does not have any further cumbersome MinGW dependencies.

Patch for "tcod-platform.c":
< #include <SDL/SDL.h>
---
> #include <SDL.h>
293c293,295
<               bufferedKey = TCOD_console_check_for_keypress(TCOD_KEY_PRESSED);
---
>               TCOD_sys_check_for_event(TCOD_EVENT_KEY_PRESS | TCOD_EVENT_MOUSE, &bufferedKey, &mouse);
>       } else {
>               TCOD_sys_check_for_event(TCOD_EVENT_MOUSE, &bufferedKey, &mouse);
390c392
<               key = TCOD_console_check_for_keypress(TCOD_KEY_PRESSED);
---
>               TCOD_sys_check_for_event(TCOD_EVENT_KEY_PRESS | TCOD_EVENT_MOUSE, &key, &mouse);

The patch makes up for what seems like custom changes to the libtcod DLL that comes with Brogue.  Out of the box, libtcod does not update the last known mouse state unless the using application polls mouse state.

Edit: Added all the other extra steps I had forgotten to mention, as I have gone through this process many times now.

Sunday 8 July 2012

Android NDK & Windows symbolic links

There are projects out there that have symbolic links in them.  Because they're stored in GIT, you get these files and directories in a corrupted form.  They just don't work, so you write a script to convert them to Windows symbolic links.

#!/bin/sh

git ls-files -s | awk '/120000/{print $4}' | while read FILEPATH
do
    if [ ! -e $FILEPATH ]; then
        echo "Please check that GIT core.symlink for this project is 'false'" 1>&2
        exit 1
    fi

    if [ ! -L $FILEPATH ] && [ -f $FILEPATH ]; then
        wc -l $FILEPATH | while read LINES DISCARD
        do
            if [ "$LINES" = "0" ]; then
                FILEDIR=`dirname $FILEPATH`
                FILENAME=`basename $FILEPATH`
                LINKPATH=`cat $FILEPATH`
                pushd . > /dev/null
                cd $FILEDIR
                if [ -e $FILENAME ]; then
                    if [ -e $LINKPATH ]; then
                        echo "Repairing symlink: $FILEPATH"
                        if [ -f $FILEPATH ]; then
                            rm $FILENAME
                            cmd /c "mklink ${FILENAME//\//\\} ${LINKPATH//\//\\}"
                        else
                            rm $FILENAME
                            cmd /c "mklink /d ${FILENAME//\//\\} ${LINKPATH//\//\\}"
                        fi
                    else
                        echo "$FILEPATH: symlink path '$LINKPATH' invalid" 1>&2
                    fi
                else
                    echo "$FILEPATH: problem finding this file"
                fi
                popd > /dev/null
            else
                # ERROR: Expected a file with one line for the link path
                echo "$FILEPATH: bad link file: expected 0 lines, got $LINES" > /dev/null # 1>&2
            fi
        done
    else
        echo "$FILEPATH: already a symlink" > /dev/null # 1>&2
    fi    
done

You run it within MinGW Shell in the top level directory of the project, but MinGW Shell is run as Administrator so that it has permissions to create Windows symbolic links.  And it appears to work.  You can see the linked files or directories looking like shortcuts in explorer, and you can open them.  But then you try and compile something with the Android NDK.

Compile thumb  : sdl-1.2 <= SDL_androidinput.c
/d/Code/android-ndk-r8/toolchains/arm-linux-androideabi-4.4.3/prebuilt/windows/bin/arm-linux-androideabi-gcc -MMD -MP -MF ./obj/local/armeabi/objs-debug/sdl-1.2/src/video/android/SDL_androidinput.o.d.org -fpic -ffunction-sections -funwind-tables -fstack-protector -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__  -Wno-psabi -march=armv5te -mtune=xscale -msoft-float -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -Ijni/../jni/sdl-1.2/include -Ijni/../jni/sdl-1.2 -DANDROID -O3 -DSDL_JAVA_PACKAGE_PATH=net_sourceforge_gemrb -DSDL_CURDIR_PATH=\"net.sourceforge.gemrb\" -DSDL_TRACKBALL_KEYUP_DELAY=1 -DSDL_VIDEO_RENDER_RESIZE_KEEP_ASPECT=1 -DSDL_VIDEO_RENDER_RESIZE=1 -DSDL_ANDROID_KEYCODE_0=LCTRL -DSDL_ANDROID_KEYCODE_1=c -DSDL_ANDROID_KEYCODE_2=NO_REMAP -DSDL_ANDROID_KEYCODE_3=NO_REMAP -DSDL_ANDROID_KEYCODE_4=e -DSDL_ANDROID_SCREENKB_KEYCODE_0=LCTRL -DSDL_ANDROID_SCREENKB_KEYCODE_1=c -DSDL_ANDROID_SCREENKB_KEYCODE_2=NO_REMAP -DSDL_ANDROID_SCREENKB_KEYCODE_3=NO_REMAP -DSDL_ANDROID_SCREENKB_KEYCODE_4=e -Wa,--noexecstack -O0 -g -O2 -DNDEBUG -g -I/d/Code/android-ndk-r8/platforms/android-8/arch-arm/usr/include -c  jni/../jni/sdl-1.2/src/video/android/SDL_androidinput.c -o ./obj/local/armeabi/objs-debug/sdl-1.2/src/video/android/SDL_androidinput.o && ./obj/convert-dependencies.sh ./obj/local/armeabi/objs-debug/sdl-1.2/src/video/android/SDL_androidinput.o.d
jni/../jni/sdl-1.2/src/video/android/SDL_androidinput.c:41:30: error: SDL_androidvideo.h: No such file or directory
...

Observing the file access in Process Monitor, access flags it as a directory in some instances. Even though file access appears to work outside the NDK case.  MinGW shell almost believes it works when you use the file command on it.
$ file project/jni/sdl-1.2/src/video/android/SDL_androidinput.h
project/jni/sdl-1.2/src/video/android/SDL_androidinput.h: ASCII C program text, with CRLF line terminators
If you try and copy the file that it links to over top of it, that seems to think they're the same.
$ cp -f project/jni/sdl-1.3/src/video/android/SDL_androidinput.h project/jni/sdl-1.2/src/video/android/SDL_androidinput.h
cp: `project/jni/sdl-1.3/src/video/android/SDL_androidinput.h' and `project/jni/sdl-1.2/src/video/android/SDL_androidinput.h' are the same file
Something shows as broken if you try and delete it.
$ rm -f project/jni/sdl-1.2/src/video/android/SDL_androidinput.h
rm: cannot remove `project/jni/sdl-1.2/src/video/android/SDL_androidinput.h': Not owner
A bit of googling suggests trying the rmdir command.
$ rmdir project/jni/sdl-1.2/src/video/android/SDL_androidinput.h
$ file project/jni/sdl-1.2/src/video/android/SDL_androidinput.h
project/jni/sdl-1.2/src/video/android/SDL_androidinput.h: ERROR: cannot open `project/jni/sdl-1.2/src/video/android/SDL_androidinput.h' (No such file or directory)
It looks like the best approach is to copy files rather than link to them. Linking to directories might be stable, but just because it seems to work for now, doesn't mean that problems might be present in certain use cases. As file linking had the above NDK compilation use case, for instance.

Saturday 7 July 2012

msysgit & corrupted symbolic links

Using msysgit on Windows?

Let's say the project you are cloning has:

real-directory-1/
real-directory-2/linked-directory-1/
Where linked-directory-1/ is a symbolic link to real-directory-1/.

What you will find in your clone of this project is:
real-directory-1/
real-directory-2/linked-directory-1
Where linked-directory-1 is a file containing the local path of what it's target would be if it were the symbolic link it were supposed to be.


The problem?

Cloning projects in a broken state just seemed.. ridiculously unusable.  I've wasted hours on errors that appear to be in one place, but are actually because of these broken symbolic links.  With a bit of googling, I encountered someone suggesting that someone else set their core.symlink GIT setting.  Out of curiosity, I decided to check the .git/config file in my cloned project.

[core]
        repositoryformatversion = 0
        filemode = false
        bare = false
        logallrefupdates = true
        symlinks = false
        ignorecase = true
        hideDotFiles = dotGitOnly

I don't understand..  The default value is to create broken clones, with missing symlinks and junk files littered throughout them?  Is there something I am missing here?

The solution?

A bit of further searching revealed that you can set this setting globally.  Something like:
git config --global core.symlink true
After executing that, and then modifying the red highlighted project-specific setting..
$ git checkout -f
error: unable to create symlink AndroidAppSettings.cfg (Function not implemented)
error: unable to create symlink project/jni/application/atari800/atari800 (Function not implemented)
 
Now the files are missing, but at least I know that the clone is bad.

I think the best approach is to have it work this way by default, then cloned projects can have their local setting changed in their configuration file.  Then the corrupted clone can be processed with a script to turn the corrupt files into the links they should be.

The following almost does the trick:
#!/bin/sh

git ls-files -s | awk '/120000/{print $4}' | while read FILEPATH
do
    if [ ! -e $FILEPATH ]; then
        echo "Please check that GIT core.symlink for this project is 'false'" 1>2
        exit 1
    fi

    if [ ! -L $FILEPATH ] && [ -f $FILEPATH ]; then
        wc -l $FILEPATH | while read LINES DISCARD
        do
            if [ "$LINES" = "0" ]; then
                FILEDIR=`dirname $FILEPATH`
                FILENAME=`basename $FILEPATH`
                LINKPATH=`cat $FILEPATH`
                pushd . > /dev/null
                cd $FILEDIR
                if [ -e $FILENAME ]; then
                    if [ -e $LINKPATH ]; then
                        ln -s -f $LINKPATH $FILENAME
                        echo "Repairing symlink: $FILEPATH"
                    else
                        echo "$FILEPATH: symlink path '$LINKPATH' invalid" 1>2
                    fi
                else
                    echo "$FILEPATH: problem finding this file"
                fi
                popd > /dev/null
            else
                # ERROR: Expected a file with one line for the link path
                echo "$FILEPATH: bad link file: expected 0 lines, got $LINES" > /dev/null # 1>2
            fi
        done
    else
        echo "$FILEPATH: already a symlink" > /dev/null # 1>2
    fi    
done
What it does not do, is create a symbolic link within the same directory, to that same directory the link is intended to reside within.

Conclusion

Do not use Git for Windows AKA msysgit.  Even if you work around the corrupted files with the above script, you cannot make the links yourself because some cases are not supported by the ln command.  You can get a little further by switching over to a MinGW msys installation (the above script works in the "does not do" case), but there are still problems.  It is probable that MinGW ln is copying, rather than linking, given the time the script takes to run.

Cygwin has the same problem, when using it's git command.

Portable Ubuntu has been recommended, but I haven't tried it yet, or looked at it's suitability.

Next: Android NDK & Windows symbolic links.

Tuesday 3 July 2012

Broken gǔzhèng strings

This post is intended to record what I have learnt about tuning my gǔzhèng (古筝), and sourcing new strings in New Zealand.

When you transport your gǔzhèng, you remove the bridges to prevent the strings from breaking if weight is placed on them.  The next time you want to play it, you replace the bridges and retune each string.

2012-06-29 - Gu Zheng - 03 - Bridges
Gǔzhèng bridges
Replacing the bridges and tuning the strings, I snapped three of the higher ones (1, 6 and 7).  These are the thinnest, and are more likely to break.  With my bad tuning technique, I think some of the lower strings are also damaged but are thankfully still usable and whole.

2012-06-29 - Gu Zheng - 05 - Instrument
Tuning knobs and strings
The best solution would have been to find suitable strings in local music stores.  I tried two local stores and it became clear this wasn't something they dealt with ever.  The first store was staffed by a Chinese lady who knew what I was talking about, and later called to suggest looking in one of the many local Chinese newspapers for gǔzhèng teachers as they often sell parts or instruments (as teachers also do in China).  The second showed me a selection of strings for other instruments that might be suitable, but we concluded they weren't and they kept my number in case they located any.  When they later called back, they had found a local teacher who would sell individual strings at $5 NZ a piece.

In the meantime, I had ordered and received several sets (a full 21 piece, and two sets of 1-10 strings) from a store in the USA called Sound of Asia.  This came to about $70 NZ including shipping, and was much cheaper than buying from a local teacher would have been given the price above.  It also arrived extremely quickly, within 3-4 days of ordering.

2012-06-29 - Gu Zheng - 02 - Strings
Sound of Asia sourced strings
With the new strings and a desire not to have to order more, I had researched (and learnt in the practice of doing this) some guidelines to follow when replacing the bridges:
  • Bridge placement: Use the tuning chart from the Sound of Asia web site, which illustrates best bridge placement for each string.  You can see white tape on the left side of my gǔzhèng, in the first picture.  This is where I've measured and marked each of the distances.
  • String fitting: There's a trick to properly winding the new string onto it's tuning knob.  Wind the string around the knob several times before pushing the end of the string through the hole.  If you do not do this, then you'll run out of tightening room before your strings are tightened enough.
  • String tuning/tightening: When turning the tuning knob for a given string tighter, take the string off the bridge, tighten and then having done that replace the string on the bridge.
I've included the tuning chart below, as I am unable to locate it on the Sound of Asia site any more.

Good luck!  And here's a YouTube video of someone playing Richard Marx's Right Here Waiting on one of these instruments, just for the sake of it.  I've got to learn more about music so I can try playing some western songs.


Thursday 28 June 2012

Recording Internet Radio & Chinese Learning

This post is intended to show how to use VLC to make MP3 recordings of streaming internet radio stations.  The hardest part is locating and copying the streaming address for your station, once you have this it's very simple from there.  VLC is a free and fully featured video and audio playing tool.  It also has a wide range of related functionality, like playing and recording internet streams.

I record the local Mandarin language radio station AM936 so that I can:
  • Have access to pre-recorded Mandarin language listening material wherever I am.
  • Listen to a particular part of the broadcast as often as I need to better understand it.
  • Save money due to the cost of streaming in New Zealand.
The first step is to open VLC and select the Open Network Stream option in the Media menu (or just hit Ctrl-N, as shown).


Then select the Network panel, paste in the streaming address and click on Stream.


At this point you should hear the streaming radio.  To start recording, click on the record button (the red dot).


VLC will then record the stream with a useful file name within a standard folder.  In my case, and most likely any case, it was in the Music folder under the Libraries folder on the Desktop.


That's pretty simple.

You can make it more complicated by choosing the Convert / Save option shown in the menu in the first screenshot.  But that gets overly messy and buggy.  When I tried it, it refused to encode the stream as an MP4 file, giving an unhelpfully vague message.

Wednesday 27 June 2012

Google Translate & Chinese Learning

One of the complaints I had about (perhaps misuing) Google Translate as a language learning tool was that when I looked up words, it just generically listed the meanings.  Did it mean future as a noun or adjective?  I've just noticed that it has been upgraded at some point, and now gives this information.

In this case, I was listening to a recording of a local Mandarin AM radio station.  The speaker kept repeating weilai.  Looking it up in Pablo gave me the match "wèilái"  (未来), but the definitions it gave are the same non-grammatically specific ones that you get in all the free dictionary based products (and up to this point, also Google Translate).


For the same of future reference, here's the current Google Translate output with separated nouns and adjectives:


On a related note, two of the cheap MP3 playing devices that I have accrued over the years also receive and play FM radio.  I'd love to find one that also receives and plays AM radio.

TiddlyWiki Recovery

TiddlyWiki is a extremely handy way to store notes locally.  As a one file wiki, with no install process -- you just run it -- it is as low overhead as you get.  Gone are the days when I run a local web server, so that I can run Trac, MoinMoin or whatever.  But like all software projects, it has it's downsides.

One downside is that occasionally the manual saving simply fails.  You get a dialog box with some unhelpful text simply stating "It is not possible to save changes.."  If you search for this text, all results suggest that you have removed the drive which you are saving to.  But this is not necessarily the case, I save to my internal laptop hard drive and this occasionally happens to me regardless.

What to do?

If you search further, there are references with links to a recovery process.  You'll notice that text is in the Wayback Machine.  It's links are dead.  But if you search for "FAQ_RescueStoreArea" there is perhaps one existing TiddyWiki instance which still contains the content you need.  It contains the following link:
Link to copy.
  1. Right click on the above link and copy it.  In Chrome this is the "Copy link address" menu item.
  2. Switch to the browser tab where your unsaveable TiddlyWiki is located.
  3. Paste the copied link into the address bar.
  4. Hit enter! Observe the new window with your raw wiki contents.
  5. Edit a local copy of your wiki ".html" file.
  6. Paste the raw wiki contents within the div HTML tag with the "storeArea" id.  You can get further detail on this from the "one existing" link above.
  7. Save your modified HTML file.
  8. Open it and verify that it contains your most recent changes.
  9. Close the old unsaveable TiddlyWiki instance tab and pretend this horrible affair never occurred, that the TiddlyWiki powers that be document a clear recovery process and forget about your wasted time.
To be clear about where to insert the recovered raw wiki contents, here is the quoted text from the "one existing" link:
<!--POST-SHADOWAREA-->
<div id="storeArea">
.... *** replace this part *** ...
</div>
<!--POST-STOREAREA-->
In theory, I would hope you could copy the tiddler into your own wiki so it is available for future use.  But it appears to be plugin-based, so this will not work and the raw pre-generated javascript link needs to be used instead.


Good luck!

Tuesday 26 June 2012

Stackless Python and Stack Overflow

I often search for programming related information, and gladly use the Stack Overflow based results.  More often than not, the answer I am looking for is already there.  This unfortunately means that people blindly post their questions there, without it occurring to them that there might be a better place where actual experts are present.  This better place is the Stackless Python mailing list.

For projects where the number of knowledgeable users present on Stack Overflow is high enough, it obviously serves as an invaluable resource.  For a lesser used niche project like Stackless Python, not so much.

Monday 25 June 2012

Shanghai water delivery

I found this in my drafts folder.  Have I already posted a version?  Beats me, but between my cold and my desire to go to bed.. I'm going to assume I haven't and just post it.

---

If you're drinking water in Shanghai, you have several options. You can boil the tap water to kill the germs and drink that, but then you still have the heavy metals. You can buy bottles from the supermarket, but then you're killing the environment with all that discarded plastic and it gets old carrying the bottles home. And you can also obtain or buy a water dispenser and have 5 gallon bottles delivered to your home.

I'm at the point where I am buying a water dispenser. The word on Shanghai Expat was that you can get one for around 100 RMB at a store like Gome. So I located my nearest Gome store on Changshou road and went there.

2011-04-04 - Water delivery - Gome - 01 - Storefront

The process of buying something at a store like these, let alone a water dispenser, is long and complicated. You start off buy going to find the good you want on the shelf, then you negotiate in limited Mandarin with all the old ladies standing around. Then one of them goes off and fills in a form, and gives it to you. Then you head up stairs to the payment counter, and pay the girls there. Then they give you two slips, one of which (the white one) is to give to the collections counter and the other (the pink one) is your fa piao / receipt.

So you head over to the collections counter, where someone takes the white slip and goes to fetch whatever you bought.

2011-04-04 - Water delivery - Gome - 02 - Pickup counter

Then you grab it and leave.

2011-04-04 - Water delivery - Gome - 03 - Takeaway

The next step is to get the water bottles to accompany it. On Shanghai Expat people seem to talk about preferring three brands mostly. Nongfu Springs, Watsons and Nestle. Nongfu Springs is a local cheaper brand, but no-one seems to talk about getting it delivered. I normally drink Nestle, so I decided to stick with it and order that. After a bit of Googling, I managed to find a Chinese language Nestle web page. Life must have been so much harder for people living in China before tools like Google Translate were around.


There are three types of water offered. Distilled at 28 RMB, mineral water for 19 RMB and purified water at 17 RMB. It is kind of confusing which to choose, but Shanghai Expat comes to the rescue again. People advise against the distilled water because it lacks the minerals and salts of normal water, which causes problems in the long run. The word is to buy mineral water.

So I called the listed number, pressed 2 for English. Then gave my address, named how many bottles I wanted, agreed to pay a 40 RMB deposit for each, chose which type of water I wanted and they will be delivered tomorrow between 12 PM and 5 PM. It is not exactly ideal to have to stick around home for five hours!

I've since left Shanghai, and didn't bother getting the ~40 RMB refund per bottle.  The Nestle delivery service was consistently helpful, and went out of their way to get me the water.  If they came at the wrong time, or when I popped out, they'd ring and sort out a new time.  I'd recommend anyone living in Shangai use them.  Many of the locals simply boiled the water, maybe that's good enough for you.. but maybe you should think twice.

Wednesday 20 June 2012

Mnemosyne 2 finally released

Just noticed there was a new release of Mnemosyne, the spaced repetition flashcard program I use which also happens to be written in Python.  It's ten times larger than the previous version, but at 20 megabytes that's not a big deal.  A quick look suggests that this is due to the inclusion of mplayer and the updated version of QT being used.


There was no information I could locate about the repercussions of installing 2.0 over an existing 1.x installation, so I went ahead and installed it under a different menu and installation directory name.  Either as part of the installation process, or when first run, it located my existing cards and imported them.


The revision interface looks pretty much the same.  The card browsing interface which I didn't screenshot, is much improved however.


There's also new support for synchronisation with mobile devices built in, and a whole range of other more standard features used in the implementation of the new version.  I haven't had a chance to explore these yet, but am looking forward to it.

The older version had support for user authored plug-ins written in Python, which I only used to convert Chinese pinyin numbers to tone marks.  Unfortunately, there's no matching support in the new version yet.  For a long time, it seemed like this was one of those projects where the author had given up on it and was effectively distracted by a never-ending "rewrite".  Glad to see it's not the case.

Sunday 20 May 2012

Planet MUD-Dev update

One new blog has been added to the aggregator:

Two blogs have died and been removed from the aggregator:
  • Lysator.  An extremely productive developer, inspired me to work on my own MUD.  Kicked off the one he worked on because of politics, presumably the blog has been discarded.
  • Ruby MUD.  This one has been dead a long time I believe.  The link was there in case it came alive.
Updates:

  • This blog.  Turns out it was pointing to the awkward feedburner thing I have stopped using because it got in the way of subscribing to labels.

Frankly it is a miracle the aggregator is working.  Bluehost has changed something and python no longer works in the shell, however the cron script appears to bypass the shell configuration and uses the default Python installation.. and hence things still work.

If you know of any MUD blogs which meet the criteria and are not already here, then please sing out!  Also, if you're involved in any community projects like the Imaginary Realities thing, some blog posts about the progress on a weekly basis might be a good idea.

Wednesday 25 April 2012

Paint.NET file type plugin development

Paint.NET isn't very well documented in terms of what you need to do to develop a plugin.  And if your plugin is intended to add support for a new file type, rather than add a new image-related effect.. then there's even less information.  Luckily you can download a third-party template and use that as a base.

Ensure the project is set to use the .NET 3.5 framework (mine defaulted to 4.0).  If this is not done correctly, then your plugin will be ignored by Paint.NET.


Copy your compiled DLL from the Visual Studio project build directory to the FileTypes folder within your Paint.NET installation.


And that's pretty much the compile and test (or installation) cycle covered.  Debugging is another matter.  I'm not sure its possible with Visual Studio Express, but I think it is with the full version.  I had to make do with custom message boxes, but it could be worse! :-)

Tuesday 24 April 2012

An alternative lossless video codec

I've been wanting to install a lossless video codec so that I can record something or other.  The recommended codec to use is the Techsmith Camtasia screen recording codec.  However, this only decodes and lacks the support for encoding.  Trying to encode gives the following dialog:


Installing the trial of Camtasia Studio unfortunately doesn't magically open up the encoding functionality for the trial period.  Luckily there's a free alternative in the confusingly named CamStudio.  CamStudio provides its own lossless video codec based on both LZO and GZIP.

There are a few teething issues.  I use Windows 7 64 bit and rhe codec doesn't appear in the list offered within the application I want to capture video within.  But thankfully moisamva has provided instructions on how to manually correct the installation, and they work for me.

Saturday 21 April 2012

Why, oh why, iPad/iPod?

I've never really been a fan of either my iPad or iPod. Each of them gets one use. My iPad is used for reading ebooks, and the iPod for daily Mandarin revision with a flashcard application. They're clunky, arcane devices, hidden behind the almost unusable and arbitrary experience that is iTunes. 

Something else I have been using them for is to take advantage of free internet bandwidth. Internet in New Zealand is metered, and a gigabyte can cost something like $2 NZ. It all adds up quickly. So when someone offers free internet, instead of using their limit of 100 MB a day, I can use 300 MB a day with my three devices. I can download a podcast over a couple of days, resuming the download on additional days.

Or I could, if Apple's iTunes downloading wasn't incomprehensibly badly designed or broken. Pause a download? Chances are that it will start downloading from the beginning again when you resume it. Download get interrupted? Chances are that it will start downloading from the beginning again when you resume it. This is basic functionality - I've downloaded iOS updates several times over, again and again, when it happens on the PC iTunes application. Search the internet, people have been complaining about it for years..

I look forward to the day I can replace these devices with something more user friendly.

Edit: And that day may come sooner!  I dropped my iPod on the pavement and the screen shattered - replacing the screen costs a good fraction of an Android phone..

Saturday 3 March 2012

pyuv experimentation

The appeal of libuv, and therefore it's Python binding pyuv, is that it provides asynchronous access to a wide range of functionality.  I don't think there is any documentation for libuv itself, if you want to understand the nuances of how to use it, you probably need to jump back and forwards between C source files interpreting the use which has been made of various macros.  The pyuv documentation however, provides a decent overview of the functionality available and how it can be used for Python.

Unfortunately, there is still a degree of libuv source code reading involved, working out what happens in various situations. In this situation, I was curious what happened when the server disconnected a client that was trying to read from its accepted connection.

Server Code

The code starts by locating or creating a loop, which is responsible for managing the events and callback dispatching related to the IO that is performed with respect to that loop. Next a timer is created, this in theory every 20th of a second calls a function. In reality, it is a hack to work around the fact that you can't directly specify a maximum timeout to the run_once method. Instead, the timeout enables the run_once to exit rather than blocking indefinitely.

Next, a TCP connection is created. It is set to be allowed to reuse the same port without waiting, bound to a listening address and then the loop is run until the callback indicating the listen operation occurs.
import pyuv
loop = pyuv.Loop.default_loop()
timer = pyuv.Timer(loop)
timer.start(lambda *args: None, 0.0, 0.05)
listen_socket = pyuv.TCP(loop)
listen_socket.nodelay(True)
listen_socket.bind(("0.0.0.0", 3000))
had_listen_callback = False
def listen_callback(*args):
    global had_listen_callback
    print "listen_callback", args
    had_listen_callback = True

listen_socket.listen(listen_callback, 5)
while not had_listen_callback:
    timer.again()
    loop.run_once()
The listen callback is called everytime there is an incoming connection so that it can be accepted and handled however. The code now proceeds to accept the connection, write token data to it, and then to wait until the data has been sent.
incoming_socket = pyuv.TCP(loop)
listen_socket.accept(incoming_socket)
had_write_callback = False
def write_callback(*args):
    global had_write_callback
    print "write_callback", args
    had_write_callback = True

incoming_socket.write("DATA", write_callback)

while not had_write_callback:
    timer.again()
    loop.run_once()
And finally, once the data has been sent, the socket is closed.
had_close_callback = False
def close_callback(*args):
    global had_close_callback
    had_close_callback = True

incoming_socket.close(close_callback)
while not had_close_callback:
    timer.again()
    loop.run_once()
Client Code

The client code has the same boilerplate to begin with, but then tries to connect to the listening address, and waits for the connection to be established.
import pyuv
loop = pyuv.Loop.default_loop()
timer = pyuv.Timer(loop)
timer.start(lambda *args: None, 0.0, 0.05)
client_socket = pyuv.TCP(loop)
had_connect_callback = False
def connect_callback(*args):
    global had_connect_callback
    print "connect_callback", args
    had_connect_callback = True

client_socket.connect(("127.0.0.1", 3000), connect_callback)
while not had_connect_callback:
    timer.again()
    loop.run_once()
Next the client starts reading, and prints out the arguments everytime it receives data in it's callback.
had_read_callback = False
def read_callback(*args):
    global had_read_callback
    print "read_callback", args
    had_read_callback = True

client_socket.start_read(read_callback)
while True:
    timer.again()
    loop.run_once()
Results

The answer to the question of what happens when the client is disconnected mid-read, is that it receives a data value of None and an error value of UV_EOF. Pretty simple actually.

My main reason for playing with pyuv, is to hopefully replace the monkey-patching framework that is stacklesslib, with one that can cover a wider variety of thread blocking functionality. The existing Stackless-compatible socket module is based on asyncore and by extension it wraps the existing Python networking functionality in an asynchronous manner, and it can handle the more complicated functionality like makefile and ioctl because of this.

Unfortunately, libuv and by extension pyuv, abstracts away or does not provide access to much of the standard API. This means a lot more work if those aspects are to be emulated. Sure, it might be an easier approach to simply not implement them, but that loses a lot of the benefit of writing a monkey-patched module. If the replacement socket module provides makefile, then it is compatible with a wider range of other standard library modules.

I've written the most straightforward part of a socket module for Stackless based on pyuv, and it can be seen within a Google hosted project here.

Monday 20 February 2012

LPMuds.net: VM to Demo LPC Muds

Link: VM to Demo LPC Muds

This is more a response to Drakkos' post on the subject, as I can't work out how you register to post comments there.  Drakkos, someone actually started a project like this back in August 2008 (see link above).  I think it would be a good idea for it to be revived and actively mantained, although how much work or what is involved.. I do not know.

Friday 17 February 2012

mudbytes.net: Excel World Builder

Link: Excel World Builder An interesting thread on MUDBytes, where JohnnyStarr shows the progress of his Excel-based building tool. I'll link directly to his image here, so he can switch it out for something obscene.

Tuesday 24 January 2012

DarkBasic and Notepad++

Syntax highlighting is my #1 requirement from an editor.  If anyone else wants to view DarkBasic source code in Notepad++, you can use the following UDL to get most of the way there.

  1. Save the XML below to "something.xml". If you mouse over the top right hand corner of the highlighted source area, you can find buttons that let you view the source and copy/paste it.
  2. Open "User-Defined Dialogue" in the view menu.
  3. Dock it.
  4. Click on the "Import" button and select the saved XML file.
  5. Close any open DarkBasic files and reopen them to ensure they get the hightlighting (needed? dunno).

    
        
            
            
            
        
        
            000000
            
            
            ' - ! " # $ % & ( ) * , . / : ; ? @ [ \ ] ^ { | } ~ + < = >
             1 2 0`
            dim as dword for to rng rnd sync on rate skipsyncmode set display mode desktop width height autocam hide next position inc prepare matrix texture trim make object box sphere update add delete rotate limb offset do loop if endif else text end curvevalue camera point constant function create bitmap endfunction ink get image save and or write memblock from atanfull sin cos circle wrapvalue mouse color backdrop aspect tile show mesh