you gotta be effin kidding me

While trying to solve this issue, i have found a document from RIM's Knowledge Base about verification errors in Java packages.
This is it.
Some choice tidbits:

7. Comment out any non-executable code. Verification errors might be related to the size of the main code file and the library files. If you comment out non-executable code, the file sizes change, which may correct the problem.

2. Remove any System.out.* calls. These generally do nothing on the BlackBerry smartphone, but they might cause verification errors.
3. Remove unused import statements.

I have nothing. My mind is blown. How can anyone ever develop anything for this device?

few notes on Samsung's IO

J2ME implementation on new Samsung phones has a few peculiarities. Yes, they're doing everything right by the spec - but they do it just slightly differently than most of the others.

First of all, InputStream.read() method, the one that reads single character, seems to be rather slow. When you say something like:

while (true) {
    int ch = istream.read();
    if (ch == -1) throw new IOException ("end of stream");
    if (ch == '\n') break;
}

then the phone will do what you want, ssssssssllllllllllloooooooooowwwwwwwwwwwwllllllyyyyyyy.
Instead, make use of InputStream.available(), for example like this:

int av = istream.available();
if (av > 0) {
    byte[] buf = new byte[av];
    istream.read(buf);
} else {
    int ch = istream.read();
}

The else is there for a reason. Two reasons, actually. First, some phones won't tell you what is available. So you should try to read anyway. Second, if nothing is actually available at that moment, read() will block and hopefully provide opportunity for other threads to run. (That is, if your phone implementation isn't completely retarded. Which, unfortunately, some can be.)

Now you've seen how Samsung reads from streams in large chunks. As it turns out, the chunks aren't as large as they could be.
Let's say you want to read a chunk of data from a file. Consider this:

DataInputStream dis = filehandle.openDataInputStream();
int length = dis.readInt();
byte[] buf = new byte[length];
dis.read(buf);

Can you spot the problem? Of course, the read(byte[]) method doesn't guarantee that it fills the buffer. But interestingly enough, on a vast majority of phones, it will actually do that when you're reading from a file.
Not on a Samsung.
So remember, kids, always check the return value of read(...) calls. Or, if you're using DataInput, as shown here, just readFully(...).

tidbits

S40 Nokias do weird things when you put a StringItem on a form, give it a label but no text. It will prevent users from scrolling that form.
S60 Nokias do some weird stuff too, but only sometimes. Best not to do this.

SonyEricssons, on the other hand, retain image size on ImageItem when you setImage(null). You have to set a dummy 1x1 image first (or instead).

And last but not least, with certain bluetooth devices (GPSr's, in this case), Nokia phone would perform a bluetooth service search, but return error instead of the service.
When you're searching for a serial port service (UUID 0x1101) and get an error instead, just take "btspp://" + RemoteDevice.getBluetoothAddress() + ":1" as a connection url and you have a solid chance of connecting successfully.
(That means BT address + channel 1. On single-purpose devices, you'll most likely find the only service on channel 1. Makes sense, eh?)

thoughts on Nokia class loading mechanism

situation:
Imagine you have your basic midlet class (say gui.Midlet). This class references another class (gps.InternalProvider) which implements a specific interface (gps.LocationService). Note that there is no reference to the actual class gps.InternalProvider, except for instantiation, and that is pretty well hidden in a function under several ifs and switches. Everything else is done through the interface.
Class gps.InternalProvider references classes from an api (JSR-179) that might or might not be present on your target device (S40 Nokia). It uses them rather extensively, but of course, it can't use them until it's instantiated. (no static codeblocks, nothing like that)

issue:
When such midlet is started on such device, it instantly dies with NoClassDefFoundError on the JSR-179 classes. It doesn't even start, I'm pretty convinced that none of my code is executed.
Definitely not the part that would instantiate the offending classes.

solution:
Create a distraction. Add a class (let's call it gps.InternalProviderRedirector) that has only one static method:

public static LocationService instantiate() {
  return new InternalProvider();
}

Then instead of doing this directly in the Midlet class, call InternalProviderRedirector.instantiate(). Magically, it will work.

thoughts:
This is easy to grasp intuitively (for me, at least), but in some situations, that is not enough. So i studied the JVM specification, especially the parts about class loading and linking, and tried to come up with a scientific explanation to this phenomenon. Here's my best effort - note that it is only an educated guess and might not relate to reality in any way.

The spec says that when you are loading a class, you get symbolic references to all classes in use. Then, in the linking step, you can (but don't have to) resolve those symbolic references by trying to load the referenced classes.
I say that Nokia does this. That means that when linking the Midlet class, the class file for InternalProvider (or InternalProviderRedirector) is already loaded.
Then, either in the initialization phase or when the first code from a class is run, Nokia JVM attempts to link all the referenced classes. That means that those classes now try to load and resolve their symbolic references.
When you start the midlet, the class Midlet is loaded. Then it's linked, triggering loading of InternalProvider. And then it's instantiated, triggering linking of InternalProvider. That triggers loading of JSR-179 classes, which are not present, so the instantiation itself fails.
When you insert InternalProviderRedirector into the chain, InternalProvider is never linked (only loaded), so JSR-179 is never loaded. And all is well.

how to send JAR files to a Nokia from linux

Just so i don't forget next time.

1. install gammu
2. create a .gammurc:

   [gammu]
   # your phone BT address:
   port=00:02:5B:00:A5:A5
   connection=bluephonet

3. cd /directory/with/jar/and/jad
   gammu --nokiaaddfile Application NameOfJarWithoutExtension

internal affairs filed under JSR179

OpenWIG 0.3.07 is out and it supports JSR-179 style "Location Providers" a.k.a. internal GPS.

Getting that to work was no picnic, though.

Here's a basic overview of how it's supposed to work:

1. choose a list of things you want from your GPS device and set it all up in a Criteria object
2. instantiate a LocationProvider by calling LocationProvider.getInstance(myCriteriaObject);
3. periodically ask for a new location by calling provider.getLocation(timeout)
4. or implement a LocationListener interface, and register that with the provider. You can then specify some refresh intervals and timeouts and whatnot. Fortunately, you can also leave it on default values.

Sounds rather easy, no?
Well, yes, but with a caveat: there's no listProviders or equivalent method. If the phone has more than one provider, it can return pretty much anything it wants.
And it turns out that e.g. Nokia N95 has more than one provider, and one of them is very dumb and coincidentally that's the one that is selected when you specify "empty" criteria (the default Criteria instance means basically "anything will do")

It seems to work when you say that you want to get speed, course, altitude and you allow it to cost money:

Criteria c = new Criteria();
c.setAltitudeRequired(true);
c.setSpeedAndCourseRequired(true);
c.setCostAllowed(true);
provider = LocationProvider.getInstance(c);

Oh and then there's the fun with invalid locations that can either have isValid flag set to false or be null. But that is easy to get right. Finding out that a phone is giving you a bad provider is the real pain.

testing FileConnection with the J2ME emulator

...is difficult at best, and impossible most of the time.
UPDATE: ...CAN be very easy, IF you "specify the security domain" in NetBeans (and choose Manufacturer, for best results). All the problems mentioned below disappear, because the emulator stops asking questions. But if you want to test behavior on regular phone with no rights, you'll have to test on an actual phone - read on.

one: unlike most (if not all) phones, the emulator runs confirmation windows in the same thread as commandActions.
That means that if you try to do anything file-related from a commandAction, a security confirmation pops up and you can't dismiss it, because your command thread is (dead)locked waiting for the result of this confirmation.
Oh well, I made my file browser run the file commands in a background thread.

two: unlike all phones, the emulator wants a security confirmation for each and every method call on the FileConnection object. And it does not remember permissions. Therefore, listing a directory takes three confirmation clicks, and every time you want to re-read a file, another confirmation (which is likely to deadlock you, for reason one).

three: the emulator (at least version 2.5.2 in linux) creates a brand new tree structure for each run. And it appears that you can't choose what will appear in that structure. So if you want to test opening a specific file, you have to copy it into the proper location every time.
(although i remember that it behaved slightly differently on my notebook. i'll have to check that out too)

and all's good when the end's good, four: do not ever in your life try to "mount" parts of your filesystem into the emulator by linking them to the temporary filesystem root.
Upon exiting, the emulator happily descended into my directory and recursively deleted it. Fortunately, there wasn't anything irreplaceable, or i haven't noticed yet; but i lost a good hour of work.

Good to know, though, is that there is now a way to undelete files from an ext3 filesystem. Didn't help me, though; the software insisted on segfaulting. It's apparently a known problem with "large partitions on some 32bit systems".

today's most interesting bug

Which, as stated before, didn't look like one.
True, the midlet would die on me when the GPS was forcefully disconnected ... but isn't that what midlets do? You take away their bluetooth connection, they die, right?
Wrong, apparently. And it wouldn't make sense, too - why should an application die just because of lost bluetooth connection?

For some reason, i started thinking about it yesterday, spent a good hour googling for the explanation and found none, so i finally gathered my courage and posted a question on SonyEricsson developer forums.
First answer solved my problem.

Two lessons to be learned here, apart from the "if google doesn't know that a problem exists, there's something wrong with the problem"...
First, Exception isn't the root of all that can go wrong. Throwable is. Replacing my Exception handling code to handle all Throwables revealed that there actually is a Throwable being thrown, and it's an OutOfMemory one.
Which also explained why the application died - when an OutOfMemory condition happens, dying is the sane thing to do. (Really. Trust me on this one.)
And second... well, not really that much of a lesson, except maybe "don't trust (foreign) code, even if you read it and thought that it's fine". That OutOfMemory was caused by a loop that continuously read from the bluetooth link and saved results into a ByteArrayOutputStream (which was rather stupid thing to do in itself, but oh well), until it found a newline.
Except that when the link died, stream did not vanish (that was handled, but honestly, what would you think if a stream just vanished from under your program? i wouldn't write something like that, but when i was reading it, it looked like proper error handling) and instead started reporting "end of stream". Which it did by returning -1.
The loop happily filled the byte array with -1's and happily ran out of memory in the middle of doing that.
Simple bug, simple fix.
Once you realize that it is a bug and not a status quo.

four quick notes about j2me

one: Obfuscation must be handled with care.
If you choose to obfuscate your midlet, you gain two things: the final size will be reduced, and the contents will be hard to disassemble.
But. If you turn on obfuscation in NetBeans, it will be used for your local debug builds as well. Due to that, you will lose debugging information, notably filenames and line numbers in stack traces.

two: Version strings must be handled with care.
My phone (SE K610i) refused to install a midlet with version "0.2.991", or anything longer than 6 characters. A different phone (Siemens CX65) installed a midlet with version "0.2.99sync1", which then mysteriously crashed with a NullPointerException that wasn't there with version "0.2.99".
This is bad magic. If anyone can explain it, please do.

three: From what i gathered, there is no way to list files contained in a JAR at runtime.
If you want to do that, create a text index at compile time and include it in a known location.

and four: Some older phones, even though MIDP 2.0, don't call Canvas.sizeChanged event.
Or maybe they do, but not (citing the spec) "at least once before the Displayable is shown for the first time." [here] Or maybe they even do that, but don't specify the proper size.
So don't rely on it.

good news about j2me's I/O

Well yes, it is horrible, but not nearly as horrible as I originally thought.
Once again, I misinterpreted a part of the spec. You do need to reopen the InputStream if you want to seek() back, that's true, but. If you keep a FileConnection object for the file in question, you can call getInputStream() on it as many times as you want - the security prompt is shown only for first access, then the phone is supposed to remember that you already have permission to do that (and it appears that many (if not most) phones actually adhere to this).
I assume that this holds true for other functions on a FileConnection as well, but I didn't test that yet.

Oh, and that RMS caching idea I had was slow and stupid.

j2ME has no seek() method

let me stress this: J2ME doesn't have seek() method.
How gay is that? (pardon my french)

File I/O in Java has been traditionally too generic to be of any use (as are most parts of its standard library), but there was always a way to do what you need without excessive hassle.

Not true in J2ME.
Imagine that you are trying to read data from a file in a certain format. The file contains three distinct JPEG images. First one starts at offset 500 and has 1000 bytes. Second one starts at offset 1501 and has 1000 bytes as well, and the third one starts at offset 2502 and has also 1000 bytes, not that it matters here. Let's pretend that you know all this beforehand.
When you open the file, you get only an InputStream. If you want to read the second JPEG, you call stream.skip(1501), which, hopefully, brings you to the start of the data. However, unlike traditional seek(), this means that you are actually reading the first 1501 bytes and silently discarding them.
Let's just hope that it's optimized to do a seek() under the hood.
There are basically three options what to do next. In order of increasing stupidity:

1. read() 1000 bytes into a byte array and work with that. If you happen to be using an API function expecting an InputStream, construct a ByteArrayInputStream over your array and use that.
2. Implement your own LimitedLengthInputStream, which allows API functions to read up to 1000 bytes from the original stream, while catching all the crucial calls like close() in order to preserve the stream for further use.
3. Hand over the original stream as-is and hope that it will "just work" with all the trailing garbage in the stream. Added value of this approach is that it effectively kills the stream and you need to open a new one.

All was fine up to this point, now let's say that you want to return to the first image and read that. How do yo go about it?
The answer, as is common to Java, is surprisingly simple: you don't.
That's right. There is no reliable* way to go back in an InputStream. And, in J2ME, you have nothing but InputStreams. Bad for you, go open a new one.

* Well yes, there is this mark()/reset() function pair. In theory, you mark() a position in a stream and then reset() when you want to return to it. In practice, my phone said "mark/reset not supported".
(This was on a resource stream, maybe it will work on file streams, but somehow i doubt it. Plus, if it failed on one phone, it will fail on others.)

Oh, and about opening a new stream - did I mention that if your midlet isn't certified, each attempt to getInputStream() gives a pop-up warning to the user? As does basically any other operation on the filesystem?
That has enough material for a medium sized rant on its own.

So, for now, my solution is to walk through the file in one pass and save the individual files from it to a RecordStore. Then retrieve them at will.

UPDATE:
The situation with pop-ups is not nearly as horrible as it appeared at first. See my new article on the subject.

expert at making things more complicated

Oh Me Oh My.

For whatever reason, Wherigo is designed in a way that perfectly maps to J2ME's gui handling. Commands, actions, dialog boxes, it all looks like somebody designed it to work in a midlet.

Except for one thing.

To quote Wherigo Advanced Concepts on the subject of MessageBoxes and Dialogs:

The Wherigo Player automatically cancels any message box that is buried by another user interface element. This means that the message box will not remain on the screen even after all other UI elements have been closed. However, the message box script (if one exists) will still execute. The author can override this by creating a condition to specify what should happen if the 'Cancel' button is pressed. These conditions will work regardless of whether the player intentionally presses 'Cancel' or the message box is automatically canceled by the application.

Funny, I thought, this is going to be rather tough. You have to keep a return value from the message box, but anything can close it anytime. Hello and welcome aboard, race conditions. And what's even better, Lua VM is thread-unsafe. And it would be dumb to lock the whole VM for as long as the dialog is on, because that can be quite a long time.

On the other hand, it's quite a dumb idea to automatically cancel the dialogs when another comes. What if the user doesn't have time to read it? But ... who am I to judge :P

So let's restate the problem.

1. Midlet UI works in non-blocking, fire-and-forget mode. Which is basically a good thing for an UI to do, but...
2. ...we need to keep return values. Therefore we must block on the dialog.
   
3. Anything can close the dialog anytime. I.e., another thread must tell the first one to continue.
4. Lua VM is thread-unsafe, therefore the two must somehow arrange locking. Plus locking of the dialog threads, because a third dialog can come at any time and kill the two.

I devised a clever scheme, where the VM is locked by a mutex (did you know Java doesn't have mutexes?), but the dialog unlocks it, and after it's canceled, reacquires the lock. Plus some regular java synchronization stuff to display the dialog properly, make it block etcetera.

Strangely enough, it worked, and it worked really well.

Too well, actually.

I apparently missed some of the Advanced Concepts.

in Wherigo, a message box does not pause the cartridge. Instead, the game continues to run while the message box is still displayed on the screen.

Just like midlets do. Fire-and-forget, anyone?

It actually does work like that. Dialog is put on screen and the script continues happily on its way. Return values? No sire. There are callbacks instead. Who could have thought?
(anyone who would try it in the builder ... oh well)
In fact, the very Wherigo Level Two cartridge I use for testing is relying on this "feature". Instead of if/elseif conditional blocks, it just displays messages over one another, stopping at the right time.
I was very surprised that OpenWIG showed more messages than the emulator. I even thought that it's a bug in the emulator, or some kind of unspecified if/elseif handling - which would be very, very bad.

Nope, none of that. Once again, I made the problem more difficult for me. If I didn't want to do it "properly", it would have been easier, and correct at the same time. Folks at GroundSpeak apparently don't do things the "proper" way, but the "practical" (easy for the programmer) way. I'll have to get used to it.

But auto canceling the dialogs? Come on. That is dumb. Really.

How to crash a J2ME emulator (and Nokia phones)

When working on OpenWIG, the mobile phone emulator would always crash (and i mean crash hard, that's segfault for you) before running anything useful. Weird, I thought, and reinstalled NetBeans, then switched to a different JDK, then different operating system, then different WTK (not really different, though; turns out that SonyEricsson's J2ME SDK contains basically unmodified WTK from Sun). Everything would crash. I had to upload the midlet to my phone (SE K610i) for each testing cycle. That worked without problems.

But "debugging" on the phone, without debugger at the ready, is a whole lot of fun. Mainly because on the phone, you don't have backtraces.
(Next time I might try OTA debugging, but right now I don't even know what exactly that is...)

The Lua VM must be triggering some bug in the emulator, I thought, too bad for me. And I released a testing version that worked perfectly on my phone.
Oddly enough, the Lua VM managed to trigger a bug in Nokia phones too. They would freeze or reset when attempting to run the midlet.

Solution? Let me quote a mail from Kristofer Karlsson, developer of Kahlua:

Someone previously mentioned that the function System.identityHashcode
crashed for some strings such as "error"

Isn't that funny?
Fortunately, the solution is simple: instead of System.identityHashcode(a), use a.hashCode().

Apparently, the identityHashcode function was slightly broken in older JVM's (and the WTK too, apparently. oh, and in Nokia phones. but oh well, i never liked Nokia anyway) and it couldn't cope with too many internalized strings.

Wherigo midlet ... or is it?

Once upon a time, there was Wherigo. It was closed-source, closed-format, and worked only on PocketPCs (and some kind of obscure navigation device). It did not work on cellphones and the developers apparently weren't interested in making it work.

Turns out, however, that the Wherigo Builder produces plain Lua source files. It can't be too hard to make that work on a mobile phone, now can it?

Well, let's hope not. Enter matejcik, who just recently bought a bluetooth GPS module to use with his phone. And whose idea of fun is, among other things, staying up late (or early, you might say) coding weird stuff.

Working name of the project is OpenWIG. As of now, it can run nearly all of Zooventure Level Two from the tutorial.
No GPS connectivity yet, but you can input coordinates by hand. I might keep this feature, as it would enable folks without GPS-enabled mobiles to play. Then again, I might not - you know how things go, cheaters are everywhere and this would only make it easier for them.

You can't load your own cartridges yet, and you will probably never be able to load gwc files, as the format is closed and I have no experience in reverse engineering. But once the loading functionality is there, I will probably set up a web service that would take your sources and compile them to OpenWIG-friendly format.

Heart of the midlet is called Kahlua and it is a Lua virtual machine for J2ME. It eats Lua 5.1 bytecode compiled files and works pretty damn well, considering its size. It is also fast enough on all phones I tested.
(Then again, it's not like the Lua scripts in the cartridges do a whole lot of work. Setting variables, conditional branch here and there, that's about it.)
And as an added value, the guy who wrote it was very helpful and the (two) bugs I reported were fixed immediately.

No download link today, but you can find some testing versions in this thread (in Czech).

Update: check out OpenWIG's Google Code page.