2015-07-25

JyNI status update

While for midterm evaluation the milestone focused on building the mirrored reference graph and detecting native reference leaks as well as cleaning them up I focused on updating the reference graph since then. Also I turned the GC-demo script into gc-unittests, see test_JyNI_gc.py.

32 bit (Linux) JNI issue


For some reason test_JyNI_gc fails on 32 bit Linux due to seemingly (?) a JNI-bug. JNI does not properly pass some debug-info to Java-side, and causes a JVM crash. I spent over a day desperately trying several workarounds and double and triple checked correct JNI usage (the issue would also occur on 64 bit Linux if something was wrong here). The issue persists for Java 7 and 8, building JyNI with gcc or clang. The only way to avoid it seems to be passing less debug info to Java-side in JyRefMonitor.c. Strangely the issue also persists when the debug info is passed via a separate method call or object. However it would be hard or impossible to turn this into a reasonably reproducible JNI-bug report. For now I decided not to spend more time on this issue and remove the debug info right before alpha3 release. Until that release the gc-unittests are not usable on 32 bit Linux. Maybe I will investigate this issue further after GSOC and try to file an appropriate bug report.

Keeping the gc reference-graph up to date


I went through the C-source code of various CPython builtin objects and identified all places where the gc-reference graph might be modified. I inserted update-code to all these places, but it was only explicitly tested for PyList so far. All unittests and also the Tkinter demo still run fine with this major change.

Currently I am implementing detection of silent modification of the reference graph. While the update code covers all JyNI-internal calls that modify the graph, there might be modifications via macros performed by extension code. To detect these, let's go into JyGC_clearNativeReferences in gcmodule. This is getting enhanced by code that checks the objects-to-be-deleted for consistent native reference counts. All counts should be explainable within this subgraph. If there are unexplainable reference counts, this indicates unknown external links, probably created by an extension via some macro, e.g. PyList_SET_ITEM. In this case we'll update the graph accordingly. Depending of the object type we might have to resurrect the corresponding Java object. I hope to get this done over the weekend.