Autocxx and g2o

We are experimenting with various VSLAM algorithms, and a big part of those is always "bundle optimization". While it is just spicy Least Squares with sparse matrices, I couldn't find a well-maintained Rust library to do this. I, however, found g2o, a c++ library which is both very well structured, easy to use, and flexible. The only problem is that it's not Rust, it's C++...

Interfacing Rust and C++

First of all, this blogpost about interoparability is way better than what I've written, so I suggest reading it instead. I'll just provide a summary here.

Rust itself is designed to be somewhat compatible with C (without the ++), in that you can declare C-compatible data structures, call C functions directly, and expose Rust functions as C-compatible ones. Usually you don't want to do this by hand, so there's the excellent bindgen crate, which generates rust code from C headers.

C++ is a different beast. First, it's hard to call a C++ function (especially methods) directly, because of the name mangling, ABI stuff, etc. You can either try to call mangled names directly, or use a small C shim which bridges the rust and C++ parts. Obviously this also needs code generators, because doing it by hand would make me suicidal real fast.

Rust <=> C++ bridge code generators

I found 4 ways to bridge Rust and C++ code:

• bindgen: Apparently bindgen has some limited support for C++, but you have to do some gymnastics, like calling constructors and destructors manually
• cxx: The cxx crate lets you generate both the C++ and Rust interface side, and it does a lot of the error-prone stuff (construction, destruction, cloning, pinning) automatically. It is useful if you fully control the API through which the two parts communicate
• cpp: The cpp crate lets you write C++ code directly in rust sources. During build time, the C++ parts are copy-pasted into a separate C++ file, C bindings are generated, and only a C FFI call is left in the "final" rust source.
• autocxx: Generates cxx-style structs and bindings from existing headers.

autocxx sounded like the easiest solution, so I went with that. The word count of this article probably spoils how it went...

Initial steps

First, I cloned and built g2o. It's easy enough, the most generic cmake-style building worked on the first try:

git clone https://github.com/RainerKuemmerle/g2o.git
cd g2o
mkdir build
cd build
cmake ..
make -j

So far so good, let's do the same in a rust environment. I made a quick lib crate, added the cmake crate as a build build dependency. I added g2o as a git submodule, and set up cmake in build.rs according to the tutorial, and it worked out of the box.

Integrating autocxx

The next step was using autocxx. I added the autocxx crate to the deps, the autocxx-build crate to the build deps, copy-pasted the S2 example but with g2o...:

build.rs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

fn build_g2o() -> PathBuf { ... }

fn run_autocxx(g2o_include_path: &Path) {
    autocxx_build::Builder::new(
            "src/lib.rs",
            [g2o_include_path]
        )
        .build()
        .unwrap()
        .compile("g2o-bridge");
    println!("cargo:rerun-if-changed=src/lib.rs");
}

fn main() {
    let g2o_install_path = build_g2o();
    let g2o_include_path = g2o_install_path.join("include");
    run_autocxx(&g2o_include_path);
}

lib.rs:

1
2
3
4
5
6
7

use autocxx::prelude::*;

include_cpp! {
    #include "g2o/core/optimization_algorithm_factory.h"
    safety!(unsafe_ffi)
    generate!("g2o::OptimizationAlgorithmFactory")
}

Cross fingers, hit cargo build, and...:

error: couldn't read /home/alex/projects/ar/g2o-sys/target/
debug/build/g2o-sys-9096d6b66137e0a2/out/autocxx-build-dir/
rs/g2o/core/optimization_algorithm_factory.h:
No such file or directory (os error 2)

Ugh.

Side note: abstractions

Humans think using abstractions all the time. It doesn't make sense to keep all the little details, all atoms and interactions in mind when doing something. Instead we assume that things are black boxes with easy to understand properties, and have a simpler mental model of how it works.

An example is driving a car. While you're driving, you are not thinking about how the internal combustion engine combusts internally. In fact, you don't even keep the fact that there is an engine in mind. You push the gas pedal, and the car will start accelerating, simple as that.

The car (and its controls) abstracts a lot of thing away from you... as long as it works.

Once the car stops working, you have to pop the hood, look behind the abstractions, and start caring about the specifics. Perhaps you should've added "engines need engine oil" to your mental abstraction of the car.

(Image by ASphotofamily on Freepik)

Popping the hood on autocxx

Back to the error. What even is going on?

Apparently the build script itself ran successfully, so the C++ files were parsed; it's the actual compilation that fails.

The file target/debug/build/g2o-sys-9096d6b66137e0a2/out/autocxx-build-dir/rs/g2o/core/optimization_algorithm_factory.h really does not exist. In fact, out/autocxx-build-dir/rs contains a single rust source file, autocxx-ffi-default-gen.rs. Why does it want to find headers relative to itself, instead of in the include dirs I told it to use?

Inside the generated file, we find

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

#[cxx::bridge]
mod cxxbridge {
    ...

    unsafe extern "C++" {
        fn autocxx_make_string_0x3ed014fe952b5583(str_: &str) -> UniquePtr<CxxString>;
        pub unsafe fn OptimizationAlgorithmFactory_alloc_autocxx_wrapper_0x3ed014fe952b5583(
        ) -> *mut OptimizationAlgorithmFactory;
        ...
        include!("g2o/core/optimization_algorithm_factory.h");
        include!("autocxxgen_ffi.h");
    }
    extern "Rust" {}
}

So it uses cxx's include! macro, ok. It's also used in the cxx examples, so it should probably work. I guess it's a special syntax that's parsed by the cxx:bridge proc macro? Let's see the macro backtrace, using

RUSTFLAGS="-Zmacro-backtrace" cargo +nightly check
    --> /home/alex/projects/ar/g2o-sys/target/debug/build/g2o-sys-2948130ad04363a8/out/autocxx-build-dir/rs/autocxx-ffi-default-gen.rs:1:11471
     |
1    | ...ty , other : & OptimizationAlgorithmProperty) ; include ! ("g2o/core/optimization_algorithm_factory.h") ; include ! ("autocxxgen_ffi.h") ; } extern "Rust" { } } # [allow (unuse...
     |                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ in this macro invocation
     |
    ::: /home/alex/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/macros/mod.rs:1379:5
     |
1379 |     macro_rules! include {
     |     -------------------- in this expansion of `include!`

It uses the built-in include! macro! Huh. I guess the proc macro does not replace it? I removed the includes from the generated file and invoked rustc again.

error: extern block cannot be declared unsafe
   --> /home/alex/projects/ar/g2o-sys/target/debug/build/g2o-sys-9096d6b66137e0a2/out/autocxx-build-dir/rs/autocxx-ffi-default-gen.rs:178:9
    |
178 |         unsafe extern "C++" {
    |         ^^^^^^

error: functions in `extern` blocks cannot have qualifiers
   --> /home/alex/projects/ar/g2o-sys/target/debug/build/g2o-sys-9096d6b66137e0a2/out/autocxx-build-dir/rs/autocxx-ffi-default-gen.rs:180:27
    |
178 |         unsafe extern "C++" {
    |         ------------------- in this `extern` block
179 |             fn autocxx_make_string_0x3ed014fe952b5583(str_: &str) -> UniquePtr<CxxString>;
180 |             pub unsafe fn OptimizationAlgorithmFactory_alloc_autocxx_wrapper_0x3ed014fe952b5583(
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
help: remove the qualifiers
    |
180 |             fn OptimizationAlgorithmFactory_alloc_autocxx_wrapper_0x3ed014fe952b5583(
    |             ~~

What. The cxx:brige macro is not working or something? Looking at my Cargo.toml, and also at the autocxx tutorial( to which I should've paid more attention) I noticed that I've missed a crucial entry: the cxx = "1.0" in the dependencies.

I added it, and viola![sic] Everything started working.

Weird how I didn't even get a warning for using an unknown proc macro. I guess rustc tried expanding things from inside out and didn't even get to it.

Implementing the simplest example

To test out the integration, I decided to port the simplest example in g2o in an even simpler form.

Build.rs changes

First, some more changes needed to be made to build.rs. Namely:

• C++17 needed to be enabled during the parsing step (with .extra_clang_args(&["-std=c++17"]))
• Eigen needed to be added to the include dirs ("/usr/include/eigen3")
• I removed the "from git" g2o include entry, since every important header is copied to the out/include dir anyway, so it's redundant

build.rs, run_autocxx part:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

let eigen_path = std::path::PathBuf::from("/usr/include/eigen3");
autocxx_build::Builder::new(
        "src/lib.rs",
        [&eigen_path, g2o_include_path],
    )
    .extra_clang_args(&["-std=c++17"])
    .build()
    .unwrap()
    .compile("g2o-bridge");
println!("cargo:rerun-if-changed=src/lib.rs");

Object creation, pointers and references

Now let's start the rewrite. The first C++ code part we will do is creating the sparse optimizer, the root object of everything useful in g2o:

1

SparseOptimizer optimizer;

Easy enough:

1
2
3

let mut optimizer = 
    SparseOptimizer::new()
    .within_unique_ptr();

What's with within_unique_ptr? Well, SparseOptimizer is not a regular rust object, it's more like a proxy. And we can choose if we want it in a Box (rust heap) or a proxied C++ unique pointer (C++ heap). Which heap is kind of important: g2o sometimes takes pointers which it frees when the containing object is released. Releasing a pointer that lives on a different heap instance is bad juju, so we will definitely want to avoid that. And while this is an opt-out behavior in g2o (see G2O_NO_IMPLICIT_OWNERSHIP_OF_OBJECTS), not having this ownership is even worse, because while the pointers themselves are stored, their lifetime is not tracked, because C++. So we either check ownership manually, or leave it g2o.

BTW, I found one of my favorite warnings this month in the cxx docs:

Take care in doing this because thread safety in C++ can be extremely tricky to assess if you are coming from a Rust background.

Next, we create the optimization algorithm itself:

1
2
3
4
5

g2o::OptimizationAlgorithmProperty solverProperty;
optimizer.setAlgorithm(
    g2o::OptimizationAlgorithmFactory::instance()
        ->construct("lm_var",solverProperty)
);

In rust:

1
2
3
4
5
6

let mut solver_property = OptimizationAlgorithmProperty::new().within_unique_ptr();
let_cxx_string!(optimizer_name = "lm_var");
let algorithm = unsafe {
    (*g2o_sys::OptimizationAlgorithmFactory::instance())
        .construct(&optimizer_name, solver_property.pin_mut())
};

There's a lot to unpack.

1. First, we create an OptimizationAlgorithmProperty object. This is a mutable object that gets filled with some information about the chosen optimizer, like matrix solver type or block dimensions. We don't need it, but the factory does.
2. let_cxx_string: We have to give the optimizer_name parameter as a C++ const std::string&, and this is the only way to create such strings, because of pinning reasons.
3. Since instance() function returns a pointer on the C++ side, it gives us a raw pointer on the rust side, that needs to be dereferenced manually before we can call its methods. There is no -> operator in rust.
4. &optimizer_name: Since optimizer_name is actually a Pin<&mut CxxString> object, it needs to be "converted" to &CxxString. This is done by dereferencing the pin (which is done automatically), and then getting a reference to it.
5. solver_property.pin_mut(): The solver_property is a cxx:UniquePtr object, and construct needs Pin<&mut ...>, we have to pin it for the call.
6. And a lot of unsafe, because we are working with pointers.

Phew. Looks like autocxx has all these rules about when something is a pin, a raw pointer, a reference, and we will have to do a lot of conversion between all these, and all of the conversions are going to be unsafe.

This should probably be wrapped in a safe rusty bubblewrap as soon as possible.

Anyway, let's give our algorithm to the optimizer:

1

unsafe { optimizer.pin_mut().setAlgorithm(algorithm) }

Fortunately (?) we got a pointer from the construct call, and setAlgorithm also operates with pointers (and takes ownership), so no magic here (other than having to pin the optimizer.)

All this work for 5 lines of C++ code. This approach really does combine the disadvantages of both Rust and C++.

Calling a superclass method

The next important line in the example is something like this:

1

optimizer.load("in.g2o");

When I tried to call load() on optimizer, I ran into two problems:

1. The smaller one: load() is an overloaded function, so it's actually called `load1()
2. The bigger one is that optimizer is a SparseOptimizer instance, but the method is declared (and implemented) in its superclass, OptimizableGraph

Turns out, the support for superclass methods in autocxx is... patchy at best. The only thing we have is an AsRef conversion, but unfortunately load() is a very mut kind of method. Apparently due to pinning stuff, the cast is actually hard?

Well, we already have the metaphorical hood of autocxx popped, and we saw that while the AsRef stuff has all kinds of rusty typing, in the end, all it does is call this code in cpp (cleaned for readability):

1
2
3
4
5
6
7

inline const
g2o::OptimizableGraph&
cast_SparseOptimizer_to_OptimizableGraph(
    const g2o::SparseOptimizer& arg0
){
    return arg0;
}

Which, if you pop the hood of C++ itself, is a noop if there is only one ancestor, and a simple pointer offset if there are more.

So what we are going to do some absolutely unholy pointer casts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

trait SuperclassHack {
    fn as_superclass_hack<T>(self: Pin<&mut Self>)
        -> Pin<&mut T>
    where
        Self: AsRef<T>,
    {
        let converted_ref = (*self).as_ref();
        let immutable_ptr = converted_ref as *const T;
        let mutable_ptr = immutable_ptr as *mut T;
        unsafe { Pin::new_unchecked(&mut *mutable_ptr) }
    }
}

impl SuperclassHack for SparseOptimizer {}

fn main() {
    ...
    optimizer
        .pin_mut()
        .as_superclass_hack::<OptimizableGraph>()
        .load("in.g2o\0".as_ptr() as *const i8);
    ...
}

Fun fact: if you do the & to &mut conversion with pointers like above, but in one line, clippy actually complains that you should probably not do that.

Also notice how the filename (which is a char* originally) has to be passed as a * const i8. Also remember to manually null-terminate the string, because rust's str is stored as length + chars, and is not guaranteed to be terminated.

We're almost there.

The next line to convert is the following:

1

optimizer.initializeOptimization();

And the converted version is:

1

optimizer.pin_mut().initializeOptimization2(0.into());

Ok, the 2 at the end of initializeOptimization2 was covered; this is how autocxx handles overloads. But what's the parameter? Well, autocxx (and rust for that matter) doesn't handle default parameters either. So the metaphorical hood pops on g2o's method declaration, and the default parameters turns out to be 0.

And to spice things up, autocxx uses newtype wrappers for ints, so we need an into() too.

The last three lines in C++:

1
2
3

optimizer.initializeOptimization();
optimizer.optimize(10 /* max iterations */);
optimizer.save("out.g2o");

Or, as we say in rust:

1
2
3
4
5
6
7
8
9

optimizer.pin_mut().initializeOptimization2(0.into());
optimizer.pin_mut().optimize(10.into(), false);
unsafe {
    optimizer
        .as_ref() // Returns Option<&SparseOptimizer>
        .unwrap()
        .as_ref() // converts to &OptimizableGraph
        .save("out.g2o\0".as_ptr() as *const i8, 0.into());
}

The last line shows how the official method of casting to a superclass works.

Linking and initializers

Let's try compiling the thing. It compiles fine, but does it actually do anything? How should we test it?

g2o itself doesn't have a set of data files, but gs-rs (the "g2o partially rewritten in rust" crate) does. I went with pos2d_only_0.g2o

Aaaand... It didn't crash. But the output was empty. This is because the data types used in the file weren't actually linked.

Apparently g2o defines a macro called G2O_USE_TYPE_GROUP, which basically makes a static variable that makes sure that those specific types actually get linked, and they are registered as factories (through macros like G2O_REGISTER_TYPE).

In our case, I did a --whole-archive (todo, I guess?), and put a bunch of these entries into the lib.rs where the autcxx macro is:

1
2
3
4
5
6

#[link(
    name = "g2o_types_slam2d",
    kind = "static",
    modifiers = "+whole-archive,-bundle"
)]
extern {}

Running it for real

Now that everything is done, I just hit run on the example.

cargo run --example=simple_optimize
cat out.g2o

The output was

VERTEX_SE2 0 1 0 2.5 
FIX 0
VERTEX_SE2 1 1 0 3.10011 
VERTEX_SE2 2 0.5 -0.5 0 
VERTEX_SE2 3 1 1 -3.1 
VERTEX_SE2 4 1 5.87341e-18 -1.5 
VERTEX_SE2 5 -18 7 -1.57001 
...

Which looks somewhat similar to the expected output:

VERTEX_SE2 0 1.0 0.0 2.5
FIX 0
VERTEX_SE2 1 1.0 0.0 -1.5
VERTEX_SE2 2 0.5 -0.5 0.0
VERTEX_SE2 3 1.0 1.0 -3.1
VERTEX_SE2 4 1.0 0.0 -1.5
VERTEX_SE2 5 -18.0 7.0 -2.8

It's not the same, but I trust the original g2o more, and this test was more about seeing if it blows up or not.

Come to think of it, the rest of the cases look pretty complex... if these are gs-rs testcases, then maybe gs-rs is actually good enough to be used for our purposes...

...meh. Maybe next time.

If you need Augmented Reality problem solving, or want help implementing an AR or VR idea, drop us a mail at info@voidcomputing.hu