Hi! I'm trying to implement some means of self-therapy to prevent me from spiraling into mental decay. The critical bit is keeping occupied and focused, so I'm thinking of working on finally rewriting my vector graphics library in a native, portable language.

By portable, I mean:

• Can compile without tremendous effort on just about any non-esoteric platform combination (hardware and software). This means x86 desktop platforms, such as modern Windows (anything beyond XP), Linux distributions, BSD (especially BSD, because I'm going to be switching to FreeBSD shortly for philosophical and personal reasons). It also means popular ARM-based mobile platforms (iOS, Android, Windows Phone...) and various embedded platforms (consoles namely, such as Wii U/PowerPC and its x86 counterparts, PS4/Xbox One).

• Can interface with any competent language by exposing a C-style interface, enabling efficient usage from other environments (i.e., using LuaJIT's FFI bits, or Java's JNI, or using it in a C++ application, or .NET using P/Invoke).

And by native, I mean:

• Low-level memory management when necessary. If the user of my library has their own memory allocation needs, my library should be able to make use of it. Very important for mobile and embedded targets, as well as high-performance desktop needs (e.g., rendering engines in games).

• No hidden performance penalties.

But most importantly, I'd like minimal antagonistic language philosophies . I don't want to become overly stressed because of poor language choices that rear their heads at inconvenient times. Verbosity isn't a major issue if it has a purpose, either--I actually prefer a more rigid and explicit syntax, especially when reading code by others ("others" includes me after looking at months-old, or years-old, code!).

The library aims to be performant enough to enable real-time vector graphics in user interfaces and games. Allowing integration into existing projects is also crucial. I actually won't be implementing rendering code in the library itself; instead, it would spit out resources for various APIs and hardware (i.e., optimized meshes and textures). Efficiently marshaling these resources (essentially binary blobs at the end of the day) is very important, as well as being able to clearly define ownership. A solid threading model could be beneficial, depending on future developments (e.g., developing parallel-friendly logic to take advantage of multi-core systems).

I've previously attempted writing a vector graphics library in C++. A simple C shim would enable the library being used in other projects and some subset of C++ is extremely portable (this subset generally eliminates C++11 and beyond, though). It fits the memory management requirements somewhat--except when you want to use custom allocators with the standard library, at which point all the data structures are off-limits. Any C++ code that doesn't play nice with custom allocators (a large amount of libraries) also becomes difficult or impossible to work with. Not to mention managing ownership and all is left to the developer and isn't integrated into the language, making it much too easy to make a mistake that won't blow up for some time. And language quirks pop up at the most inopportune times. Verbosity is needlessly terrible and useless, especially with modern computers (namely the archaic headers/source distinction; I shouldn't have to write so much boilerplate code that doesn't offer any incentives). And threading is a joke.

I've skimmed over Rust and find it very refreshing. Looking at larger projects written in Rust isn't an obstacle (I spent the better half of a day looking at Mozilla's Servo browser engine). I am very fond of the memory features (especially if many memory errors are found during compile time, which seems to be a major point/philosophy?), and it seems I can have fine control of allocation strategies, all the while benefiting from a solid ownership model. Creating a C shim seems well supported (judging by Servo and documentation I found otherwise). It seems Rust uses LLVM as a back-end, which should ensure the portability requirements. I know a few users here have used Rust to varying degrees, so advice and anecdotes would be nice! It also seems to have a good threading model.

I've used D in the past. I'm very fond of the power and syntax D brings. For this project in particular though, it seems counterproductive despite D being a native language because of the memory requirements. As far as I can tell, D's garbage collector, despite being optional, is pretty much necessary when using the standard library and some language features. I'm also not sure how well supported D is on less common platforms (mobile and embedded). No idea about threading, having never worked with threads in D. edit: I also can't find any documentation on making a C shim for a D library, which is absurd. I found a snippet about using Haskell's FFI with a D library, but nothing on marshaling or potential ABI issues.

And there's probably other options that I haven't discussed... So any advice?

Portability ? C.

The verbosity of C is worse than C++, especially math code. Considering my library would be very dependent on matrices and vectors, I'd rather not. Not to mention Microsoft's C compiler has no support for C99.

And regarding C++ portability: there's many useful niches of this library where only a subset of C++ would be used where C++11, if supported, isn't amazingly useful. For example, consoles developers, or a console SDK itself (i.e., PS4), often disables support for exceptions (making the C++ standard library counter-productive). Unreal Engine 4 disables exceptions, as well, on the platforms it supports. C++ containers have terrible support for custom allocators (even in C++11), and the language itself makes using custom allocation strategies a nightmare if using any other libraries.

And thank you for your advice, SiegeLord ! I think I'm going to try Rust for this project. Ownership models will be very simple as it stands, so I don't suppose I'll run into extreme issues with the borrow checker at this moment.

what for?

I just want a null pointer exception. :/

A few reasons really.

I (very) briefly looked into other engines and they all seem to be designed for a specific use case (FPSs mostly) and are not very good at working with dynamic/modifiable worlds.

Also, this is something I want to do, for learning purposes.

Just considered you might be asking what the library itself would be for, its games mostly.

The core idea is that everything, including the core frame building bits, is data driven and can be changed at runtime (can call it a framegraph). Including the entire main renderer, or adding some pre or post processing, maybe add some overlays, whatever you want really. I can't take credit for the idea though. The rest of the engine is a scene graph of sorts, also data driven and easily mutate-able. There's a concept of a "Query", where things can ask the engine to let them know if some condition is met, and the engine then fires off an event when it happens. It should avoid the need for individual bits to scan the world for things. Everything will be scriptable. It will also be highly parallel. That may make some things a bit "interesting" to do (events of any kind will usually be fed into a job queue that can run on a number of cpus/cores)

I have somewhat grandiose plans we'll see if I can make it work the way I want, and if i can get some kind of game on it working before I die

I think it's laughable to disregard C on the premise of MSVC not supporting C99, yet look towards far less portable platforms like D or Rust. It sounds less like a legitimate question and more like you're looking for somebody to support your bias. To each his own. I'm not heavily invested in this sphere.

I have little use for writing low-level, blazing fast to run, slow to write code. New players sound like really great ideas until you look into actually targeting them and then it's really not worth it anymore. It's a pain in the neck just for a newcomer to build a program for these platforms. They aren't readily available. I'd probably appreciate some of the latest C++ features, and at the same time probably curse them for their rough edges compared with both modern and ancient options. C is nice in theory for being pure and simple, and horrible for being too close to the metal and far too much effort to get even a relatively simple job done properly (which C++ also suffers and probably D and Rust et. al. too).

I imagine that deploying a D or Rust program are also a bit of a pain right now. I think it's cool that somebody is taking on the problem of fast and smart coding platforms, but I think the evidence shows that it's still a few decades away from being free. Until then you and likely your users will have to do a lot of hard work, or you're stuck with the tried and true subset of C or C++ that is truly portable or with preprocessor magic where that fails.

The really interesting thing is how much is possible with very high-level code these days. It really makes me wonder how much value there is investing in such low-level code. It should be enough to keep somebody like the OP busy, which is arguably a good thing. That said, his efforts may be better placed trying to contribute to the dream projects he proposes to use rather than attempting to merely use them (albeit, using them is probably a contribution in itself).

It makes perfect sense. With D and Rust you won't have MSVC users trying to compile your code and failing. It's less of a hassle to get a user to install a Rust or a D compiler than it is to make a MSVC user install GNU tools.

There's lots of C# and Java games now though, because those languages are "productivity first" programming languages.

My games engines will run D or C++ as a back-end, with Python as a scripting language.

I've opened the can of worms for Ruby extensions. I think it's viable to write a complete backend in C/C++ and have the content creation and scripting wrapper in ruby.

I just don't know what the core module components might be.

   class Entity
   end
   class Menu
   end
   class World
   end
   class Scene
   end

?

It just seems like so much overhead.

So it's structured more like a packed database table than a series of scattered and uncertainly sized derived objects. I recently discovered "object slicing" about a month ago and thought dammit, now I'm going to have to rethink this whole architecture.

Is the bottleneck of cache misses and cache coherence really that substantial? I wish I knew more about this.

Holy balls, yes. RAM is what... 200 times slower.. worse when you multiply every redirection you have to do!

This talk opened my eyes:

https://youtu.be/rX0ItVEVjHc?t=30m24s

It was great. He was like "screw templates. screw RTTI." to a bunch of C++ elites who seem to get off on the idea of writing templates for everything. They were speechless.

Another great point he drove home in another one is how we're all brainwashed into resolving problems into OOP. "I have a car, I make a car class. I have a bird, I make a bird class." That's stupid. A computer doesn't know what a bird is, and you have assumed that the problem you're trying to solve will neatly fit into your chosen concepts, and then you wonder why it's all slow as balls--because of the ridiculous amount of redirections going on.

Like, why the hell should a "car" object operate on itself? What happens when a car has to talk to another car, like during collision detection? Who owns "collision"?

Moreover, you can't pack a bunch of "Car classes" together. You can, but they're going to be bigger than the cache lines and certainly not a multiple of the cache line size. With data-oriented design, you can keep the data separate from the code. You can not only put cars together, but you can also put like-attributes of cars/birds/etc like "position" data packed together. Why? Because you're almost always using the same data in the same way. Positions are compared to positions, not a position+sound+texture data. So why have them next to each other in memory? If they're next to each other it becomes trivial to unroll a loop to the size of your typical cache lines and presto, wizard-level speed increases. You can also do things like keep the position data of cars, dogs, worlds, and tacos all next to each other instead of trying to collide a taco (position + taco_specifics) and a car (position and car_specifics) when all you really care about is the position data of each member.

And what about cache lines? "I can't hardcode that! What if there's another architecture?" He goes on to say that "No matter what you're doing there is ALWAYS a subset of architectures you are targeting. So find your subset, and optimize for that. Nobody is writing code to run on PCs and commodore 64's."

This was an edit, but it makes more sense as another post:

----

This also works GREAT for static pools. You almost always know the maximum amount of objects you're going to have in a game. A world mesh isn't going to have more polygons than when it started (in most games). So you throw everything into static pools tied to scene data. (Global data + level specific data) Want to free all the data with a level? Well, free the [objects_in_the_level] static pool and you're done. That's it. The thing that's really great about static pools is... you're not deleting anything. You're re-using the skeleton of objects. Essentially, everything becomes a particle system. There's no need to "delete" an object from the heap when it's dead. Mark it as dead, move on, and use the corpse for your next created object.

Another thing is: Branching is evil. OOP leads to tons of branching and redirection. In a real game by AAA-programmers, they don't branch... they sort. That way, all objects of a specific branch all hit over and over so the branch prediction doesn't have to constantly try and re-learn. It goes from 1111111111... to ...1110000000 instead of say 10011110001000101011110010101001.

One example he lists in the video is literally write out your branches to a file, and then zip it. The compression ratio is an approximate indicator of how much time you're wasting branching.

Basically, anything you read under data-oriented design, or from Mike Acton of Insomniac Games is absolute gold and eye opening.

He did some other great talks on game development at a college, as well as Cell processor programming. Apparently, DoD is required for getting any real power out of a Cell so Sony put tons of time and money into making DoD talks.

Here's a great PPT of a talk:

http://harmful.cat-v.org/software/OO_programming/_pdf/Pitfalls_of_Object_Oriented_Programming_GCAP_09.pdf

I can't find the video of the talk yet. But it does break it down into numbers to prove all of their claims.