A C compiler, a CPU emulator, an IDE…

…are you getting it? They’re one project.

The two devices I have been most excited about in many years are iPad Pro and Vision Pro. However, the software support for writing systems software on these devices have historically been lack luster: many programming apps either only support traditional scripting languages (such as Lua, Python, or JavaScript), require an internet connection to phone home to a server that performs compilation and execution, or even opt to embed the entirety of Clang and LLVM in interpreter mode. The most competent attempt at meeting the needs and wants of engineers I have seen so far is Swift Playgrounds, with its ability to publish iPad apps directly from the app itself, and providing access to lower level frameworks. However, it still only supports compiling Swift.

So I have set out to build a suitable environment to write low-level systems programs directly on these platforms.

IDE

On the surface, the IDE app is fairly bare bones: there is a single “tab” for editing just one open file, a file tree, and an output window.

Errors

Code is being compiled and checked for errors on a continual basis. The IDE integrates deeply with the compiler to provide an error list window, and clicking on an error sets the editor’s selection to the token where the error was found. The editor also underlines all tokens where errors are currently being encountered.

Environment

What makes the IDE special is the runtime environment. Vertical integration with the CPU emulator enables user code to call special API functions in the IDE to do things such as rendering 3D graphics.

Sample code: https://gist.github.com/machinamentum/0c0c12c973915e9bbf45e16fd22eaf08

In the future, one can imagine even extending the editor via plugins writen in C via an API exposed in this way.

CPU Emulator

The intepreter engine emulates a RISC-style CPU with 16 integer registers, and 16 float registers.

Special built-in functions enable the emulated program to call standard C API’s, or call virtual machine functions that call into the app frontend. These mechanisms are not dlopen/dlsym wrappers, the runtime implements standard C API functionality, implementing checks and managing resources to prevent errors and leaks. printf, for example, is entirely implemented within the interpreter engine, including parsing the format string, and formatting the arguments (which are retrieved from VM registers); the app frontend need only provide a single putchar callback for the engine to report output characters to.

VM functions (those provided by the app frontend) are registed with the interpreter engine and called via a hash lookup of the name of the function. This is done in user code by calling a built-in function, call_vm_function, that takes the name of the VM function and performs a hash lookup to find the frontend callback to call.

[[builtinfn("call_vm_function")]] void call_vm_func(const char *name, ...);

int main () {
	// Call the open_window VM function with the given title "My App Title"
	call_vm_func("open_window", "My App Title");
}

The attribute [[vmfn("name_of_function_here")]] may be declared on a function declaration; this tells the code generator to call call_vm_function and to pass the given name as the first argument.

[[vmfn("open_window")]] void openWindow(const char *title);

int main () {
	// Call the open_window VM function with the given title "My App Title"
	openWindow("My App Title");
}

Note that in either case, the identifier of the declaration does not need to match the registered name of the VM function being called.

Memory is entirely sandboxed: the program sees “virtual addresses” that the emulator translates and bounds checks on loads and stores, catching memory access errors in the process. Under the hood, the interpreter manages regions of system memory with an assigned virtual address range, and access permissions. When the user program accesses memory, the virtual address, and size of the access, are checked against the ranges of the mananged memory regions; thus it is impossible for the user program to read/write memory outside of the sandbox.

C Compiler

The C compiler is entirely new; this is not another consumer of clang. Compilation speed is critical in this project; compilation, and execution, should always feel instantanous (even a simple Hello, World compiled with clang takes too long to compile and execute). As a poor example, a file containing 1 million printf calls takes 0.7 seconds (O2, 1.5ish in debug) to both compile and execute in this system, clang takes 7.6 seconds just to compile the file (M2 MB Air, plugged in to power).

The C compiler currently supports many foundational C language constructs including trickier-to-implement elements such as typedef and macros (with arguments). However, there is still a very long way to go to support compiling large and complex C programs.

On Clang and LLVM

Many projects opt to embed Clang to compile C code; sometimes utilizing Clang to parse C and translating the Clang AST into a format native to the project; sometimes embedding LLVM to compile and execute code. Why build a new C compiler? Weight, Velocity, and Fun.

For reference, the following LLVM build has been configued with cmake -S llvm -B build -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="AArch64;ARM" -DLLVM_ENABLE_PROJECTS="clang". AFAICT, this is a best-case scenario for these tools.

Running ls -l in the LLVM 18 build directory in Release configuration, we can see that the clang binary is 110 megabytes, and the lli interpreter is 36 megabytes. It is unclear to me how much of the bloat can be stripped if one just wants the clang compiler and the LLVM interpreter in one binary, but I think it is safe to assume it would be on the order of 10s or 100s of megabytes. That’s far too large, by default.

machinamentum@Joshuas-MacBook-Air build % ls -l bin/clang-18 bin/lli
-rwxr-xr-x  1 machinamentum  staff  110452360 May 20 01:06 bin/clang-18
-rwxr-xr-x  1 machinamentum  staff   36207360 May 20 00:56 bin/lli
machinamentum@Joshuas-MacBook-Air build % 

In comparison, the current size of the new compiler and interpreter is 206 kilobytes. That’s several orders of magnitude smaller, and I don’t expect the code size to increase by more than one order of magnitude by the time the project is in a shippable state.

machinamentum@Joshuas-MacBook-Air cvm % ls -l cvm
-rwxr-xr-x  1 machinamentum  staff  206096 Nov 15 00:28 cvm
machinamentum@Joshuas-MacBook-Air cvm % 

So, what is an acceptable size budget for our tools? How long should they take to download and install? How long should it take to build the tools from source? In an ideal world, the answer to these things would be small enough to download, build, and install instantaneously; however, a reasonable metric I now use is the time it takes to make a cup of coffee using a modern pod coffee system. Timing myself as I am writing this article, it takes just under 3 minutes to go to the kitchen, brew 3oz of coffee through an overpacked 3rd party pod (Nespresso Original Line style), and return to my desk. With an official pod that is packed properly, it would likely take closer to 2 minutes in total. The 2-3 minute range should be plenty of time for any toolchain to download+build+install on a modern computer. So how long does it take to build Clang + LLVM in the configuration mentioned above?

It takes 30 seconds just to configure the project:

cmake -S llvm -B build -DCMAKE_BUILD_TYPE=Release    12.30s user 13.84s system 88% cpu 29.592 total

And another 16 minutes to compile the tools on a modern, fast machine (a M2 MacBook Air with 8GB of RAM, not a ridiculously powerful machine, but not an unreasonably low end machine either). This isn’t actually that bad; I’ve experienced many times in my life having to wait in excess of 30 minutes for an LLVM build.

cmake --build build -j8  7885.73s user 449.68s system 718% cpu 19:20.42 total

To build the new project, it takes just 3 tenths of a second.

machinamentum@Joshuas-MacBook-Air cvm % time ./build.sh
./build.sh  0.22s user 0.08s system 104% cpu 0.287 total
machinamentum@Joshuas-MacBook-Air cvm % 

• Added support for default parameters:

#import "LibC";

func test(a: int32 = 10, b: float = 1, c = "hello") {
    printf("a: %d\n", a);
    printf("b: %f\n", b);

    printf("c: %.*s\n", c.length, c.data);
}

// We don't currently error on the case that two or more functions
// cause an ambigous overload situation. Currently, the behavior
// is that the functions with fewer needed default params is prefered.
// This means you cannot call the above function via test(2, 3), or test(1), as
// the function below will be chosen instead since it requires less parameters
// to be filled in.
func test(a: int32, b: float = 1) {
    printf("Dang\n");
}

func main() {
    test();
    test(1);
    test(2, 5, "test");

    // Will print "Dang"
    test(3, 5);
}

• Added support for binary literals, starting with the prefix 0b or 0B.
• Added support for switch-case control flow:

 #import "LibC";

func main() {

    enum Test {
        A;
        B;
        C;
        D;
    }

    var i = Test.D;

    switch i {
        case .A:
            printf(".A\n");
        case .B:
            printf(".B\n");

        case .C, .D:
            printf(".C or .D\n");
    }

    let Y = 10;
    var x = 2;

    // Currently only integer and enum (integer derived) types are support.
    switch x {

        // Currently only expressions that resolve to integer literals are supported.
        case 1:
            printf("1\n");
        // Multiple case conditions can be declared per case-block.
        case 2, 3:
            printf("2\n");
        case Y:
            printf("Y\n");
    }

    printf("Done\n");
}

• Added support for overloading the array index operator [], and the lvalue-assignment array index operator []=:

operator[] <T, R>(arr: [..] T, index: R) -> T {
    assert(index >= 0 && cast(int64) index < arr.count, "Array index out of range!");
    return arr.data[index];
}

operator[]= <T, R>(arr: [..] T, index: R, value: T) {
    assert(index >= 0 && cast(int64) index < arr.count, "Array index out of range!");
    arr.data[index] = value;
}

• Added support for template structs:

#import "LibC";

struct Poly<T> {
    var a: T;

    func do_thing(this: *Poly<T>) {
        if (T == int)   printf("%d\n", this.a);
        if (T == float) printf("%f\n", this.a);

        // Unfortunately, we cannot straight up do this because this is invalid when T == int or T == float
        // if (T == string)printf("%.*s\n", this.a.length, this.a.data);

        // But if we added another static-if construct like `when` in Odin then
        /*
            when (T == string) printf("%.*s\n", this.a.length, this.a.data);
        */
        // would work. I do not want to make #if this generous as #if operates directly on
        // the scope it is declared in, `when` on the other hand would work exactly like a
        // a regular `if` except the body is only evaluated when the condition is a literal
        // `true` (or folds to).
    }
}

func main() {
    var a: Poly<int>;
    a.a = 10;
    a.do_thing();

    var b: Poly<float>;
    b.a = 1345.234;
    b.do_thing();

    var c: Poly<string>;
    c.a = "Hello, Pilot!";
    c.do_thing();

    printf("%.*s\n", c.a.length, c.a.data);

    // We currently do not have a way to parse TypeName<Typearg, Typearg2, etc>
    // in regular expressions, so currently one cannot use templates in first-class
    // uses unless one uses a typealias (since the right hand side of a type alias is
    // always a type instantiation).
}

• Added an intrinsic function for inserting a debugging breakpoint in code __builtin_debugtrap().
• castano added support for filling function types in polymorphic parameters:

#import "Basic";
#import "LibC";

func test<T> (a: T, b: T, comp: (a: T, b: T) -> int) -> int {
    return compare(a, b);
}

func compare(a: int, b: int) -> int {
    return a - b;
}

func bar<T> (foo: () -> T) -> T {
    return foo();
}

func zero () -> int {
    return 0;
}

func main () {
    var i = bar(zero);
    var j = 1;

    var result = test(i, j, compare);
    assert(result == -1);

    printf("result = %d\n", result);
}

• castano added support for declaring float literals in scientific notation.
• Added support for anonymous unions and anonymous structs:

#import "LibC";

struct Test {
    var d: float;

    union {
        var a: int;
        var b: int;
    }

    var c: int;
}

func main() {
    var t: Test;

    t.a = 10;
    t.c = 20;
    t.d = 15.5;

    printf("t.d: %f\n", t.d);
    printf("t.a: %d\n", t.a);
    printf("t.b: %d\n", t.b);
    printf("t.c: %d\n", t.c);
}

• Upgraded to LLVM 9. This change has been made primarily to pull in newer features from libclang inorder to properly detect and import anonymous unions/structs.

Additional changes include various fixes.

Projects

One of the main reasons I am building this language is to have fun programming in the language myself. In early February, I had decided to start a new project that could be built within a reasonably short amount of time, but still do something that is interesting and relatively nontrivial. I chose to build an Atari 2600 emulator since I have had previous experience building this type of emulator, and it fits within this criteria: the 6507/6502 has a relatively small instruction set and is well documented, the 2600 has a fairly small number of hardware interface registers, and thus has a small surface area to implement for each of the onboard chips, and the hardware’s behavior is fairly simple. That being said, the hardware does have a large number of minor quirks, but if you are not aimaing for 100% accuracy, then a mostly naive implementation of the hardware from the perspective of the Stella programmer’s manual is enough to get a good number of games working in a relatively small amount of time.

I am planning to do a separate write up on the project in the near future. The source code can be found here. I’ve uploaded a Windows binary here.

Conclusion

I want to again thank the contributors for picking up this project and helping to improve it!

The code for the compiler can be found at: jiyu. Pull requests, feature requests, and issue reports are all welcome.

• There is now support for harnessing LLVM’s JIT engine in user code

#import "Compiler";

// Since metaprograms are just fully JIT-ed LLVM modules,
// we have been internally using the JIT-er to lookup the "main" symbol
// in a program and call that... We no longer do that internally.
// There is a legacy function in Compiler to mimick the way this worked
// previously, but now we allow user code to fully leverage this functionality.

func @metaprogram main() {
    var options: Build_Options;
    options.only_want_obj_file = true;
    var compiler = create_compiler_instance(*options);

    if compiler_load_string(compiler, CODE_TO_COMPILE) != true return;
    if compiler_typecheck_program(compiler) != true return;
    if compiler_generate_llvm_module(compiler) != true return;

    // New function to JIT the entire program.
    if compiler_jit_program(compiler) != true return;

    // Once compiler_jit_program passes, we can now arbitrarily query symbols
    // and call into them.
    var do_a_thing = cast(() -> void) compiler_jit_lookup_symbol(compiler, "do_a_thing");
    do_a_thing();

    var do_a_thing2 = cast((p: string) -> void) compiler_jit_lookup_symbol(compiler, "do_a_thing2");
    do_a_thing2("Howdy :)");
}

let CODE_TO_COMPILE =
"""
#import "LibC";

// We are using the @export tag here for the sake of demonstration, otherwise
// we would need to lookup the proper jiyu-mangled names.
func @export("do_a_thing") do_a_thing() {
    printf("Hello, Sailor\n");
}

func @export("do_a_thing2") do_a_thing2(str: string) {
    printf("Hello, Pilot! %.*s\n", str.length, str.data);
}
""";

Additionally, we now provide a library version of the compiler that is built alongside the compiler driver program. The API is exactly the same whether the user code is running as a metaprogram or as a compiled binary. This allows users to effectively implement custom driver frontends that support arbitrary operations that the official driver does not. This also allows compiled user code to JIT-compile Jiyu code at runtime, enabling games to embed Jiyu as a robust scripting system, or enabling IDEs to embed Jiyu in environments where running external executables and toolchains is not allowed.

• Support for operator overloading has been added. Currently, only +, -, *, and / can be overloaded, but support for more operators will be added in the future.

#import "LibC";

struct Vector3 {
    var x: float;
    var y: float;
    var z: float;

    func make(x: float, y: float, z: float) -> Vector3 {
        var v: Vector3;
        v.x = x;
        v.y = y;
        v.z = z;
        return v;
    }
}

operator+(b: float, str: string) -> float {
    return b + cast(float) str.length;
}

operator+(a: Vector3, b: Vector3) -> Vector3 {
    return Vector3.make(a.x+b.x, a.y+b.y, a.z+b.z);
}

operator-(a: Vector3, b: Vector3) -> Vector3 {
    return Vector3.make(a.x-b.x, a.y-b.y, a.z-b.z);
}

func print_vector(v: Vector3) {
    printf("vector3: %f, %f, %f\n", v.x, v.y, v.z);
}

func @metaprogram main() {
    var a = Vector3.make(1, 2, 3);
    var b = Vector3.make(4, 5, 6);

    var c = a + b;
    var d = b - a;

    print_vector(a);
    print_vector(b);
    print_vector(c);
    print_vector(d);

    printf("1.0 + \"Hello\": %f\n", 1.0 + "Hello");
}

• castano has implemented support for typeof() operator that resolves to the type of an input declaration.

• castano has implemented support for enums:

#import "LibC";
#import "Basic";


enum Day : uint {
    Monday;
    Tuesday = Monday + 1;
    Wednesday = 2;

    Thursday;
    Friday = 1 + Thursday;

    Saturday;
    Sunday;
}

enum WeekendDay {
    Saturday = cast(WeekendDay)Day.Saturday;   // This should require a cast. Do not allow implicit cast from one enum to another.
    Sunday;
}

enum Bool : uint8 { False; True; }

typealias DAY = Day;

var another_day : DAY = Day.Monday;

var monday = Day.Monday;
var tuesday : Day = Day.Tuesday;
var sunday : Day = cast(Day)WeekendDay.Sunday;
var my_day : Day;

func test(day : Day) -> int {
    return cast(int)day + 1;
}

func main() {
    {
        var day : Day;

        let Monday = Day.Monday;
        let Tuesday = Monday + 1;
        assert(typeof(Monday) == Day);

        let MONDAY = cast(int)Day.Monday;   // Type of MONDAY is int
        assert(typeof(MONDAY) == int);
    }
  
    printf("Monday = %d\n", Day.Monday);
    printf("Tuesday = %d\n", Day.Tuesday);
    printf("Wednesday = %d\n", Day.Wednesday);
    printf("Thursday = %d\n", Day.Thursday);
    printf("Friday = %d\n", Day.Friday);
    printf("Saturday = %d\n", Day.Saturday);
    printf("Sunday = %d\n", Day.Sunday);

    printf("test(Day.Monday) = %d\n", test(Day.Monday));

    // Implicit calls from integer to enum:
    // Currently these are only allowed when the expression is a mutable literal.
    {
        var day : Day = 0;
        printf("test(0) = %d\n", test(0));
    }
    

    //printf("Whatever = %d\n", Day.Whatever); // This should trigger an error.
    {
        var day : Day = .Monday;
        test(.Tuesday);
        if day == .Wednesday {
            day = .Sunday;
        }
        if .Wednesday > day {
            // test(1 + .Tuesday); // This produces an error, as expected.        
        }
        
    }
    
}

Additionally, support for enum bit-flags is also available:

#import "LibC";
#import "Basic";

enum @flags Entity_Flags {
    Invisible;
    Solid;
    Cast_Shadows;
}

func main () {
    printf("%d\n", Entity_Flags.Invisible | .Solid | .Cast_Shadows);
}

• Tuples are now part of the language:

#import "LibC";

func test() -> (a: int, b: float) {
    // arguments in tuple-expressions are automatically inferred
    // according to the fields of the tuple being assigned, returned, or passed to.
    return (1, 4);
}

func test3() -> (c: int, d: float) {
    return (1, 2);
}

func test2(a: (a: int, b: float)) {
    printf("T2: %d\n", a.a);
    printf("T2: %f\n", a.b);
}

func main() {
    var t = test();
    var t3 = test3();

    // t and t3 are the same type as far as the type system is concerned,
    // their tuples will share the same type information in the RTTI system,
    // and internally, they will have the same type table index,
    // however, since the semantic analysis system assigns types in-place,
    // t and t3 have distinct fields.

    // t.c = t3.a; // this fails

    printf("%d\n", t.a);
    printf("%f\n", t.b);

    printf("%d\n", t3.c);
    printf("%f\n", t3.d);

    t = t3;

    t = (2, 4);

    printf("%d\n", t.a);
    printf("%f\n", t.b);

    test2(t);
    test2((6, 9));

    var n: int;
    var m: float;

    // In the event that a tuple-expression is on the left-hand-side
    // of an assignment, the type of the tuple is directly determined by
    // the types of its arguments, regardless of the type of the tuple
    // on the right-hand-side.
    // Each field of the tuple on the right-hand-side is unpacked into
    // the individual arguments of the tuple-expression on the left-hand-
    // side.
    (n, m) = test();

    // Like other types of assignments, you cannot try to use a literal
    // in a tuple expression on the left-hand-side of an assignment.
    // (n, 1) = test(); // does not work.

    printf("n: %d\n", n);
    printf("m: %f\n", m);

    // These do not work yet, but are planned to be available eventually.
    /*
    var (i, j) = test();
    var (i: int, j: float) = test();
    */
}

Additional changes include improved support for C interfaces on various platforms, improved support for trivial-constant-folding, and various fixes.

I want to again thank the contributors for picking up this project and helping to improve it!

The code for the compiler can be found at: jiyu. Pull requests, feature requests, and issue reports are all welcome.

• There is now a #clang_import directive. This allows importing declarations from C header files directly into Jiyu scopes. Example:

 #clang_import "#include <stdio.h>";
 
 func main() {
     printf("Hello, Moonman!\n");
 }

The string following the directive can contain arbitrary C code. This works for a number of C declaration types, including structs, unions, enums, functions, and typedefs. Do note however, that only importing bindings for declarations is supported, there is no support for compiling C function bodies, and there are no plans for Jiyu to replace a C compiler. I have been using this for awhile on Windows, and have even gotten the nuklear imgui library working without needing to manually write bindings, but this may not work well on Mac/Linux yet if you need to import system headers.

• There is now a mechanism to enable member-function-call syntax for arrays. I have documented this in this demo below:

#import "LibC";
#import "Array";

// Array module defines:
// __array_add<T>(arr: *[..] T, item: T)
// __array_reset<T>(arr: *[..] T)
// __array_reserve<T>(arr: *[..] T)
// __array_resize<T>(array: *[..] T, _amount: int)
// __array_pop<T>(array: *[..] T) -> T
// __array_contains<T>(array: *[..] T, item: T) -> bool
// __array_add_if_unique<T>(array: *[..] T, item: T) -> bool

func main() {
    // When the compiler sees <array>.func(...), it will attempt to transform this sequence into
    // __array_func(<array>, ...) or __array_func(*<array>, ...).
    var arr: [..] int64;
    arr.add(1);
    arr.add(2);
    arr.add(3);
    arr.add_if_unique(3);
    arr.add_if_unique(4);

    for arr printf("Value: %lld\n", it);

    printf("Popped value: %lld\n", arr.pop());

    for arr printf("Remaining Value: %lld\n", it);

    arr.reset();
    for arr printf("Remaining Value ????: %lld\n", it); // This should not output anything!

    locally_scoped_functions();
}

func locally_scoped_functions() {
    // Since this special syntax turns into a scope-lookup,
    // we can override the defaults defined in the Array module.
    func __array_add<T>(array: *[..] T, item: T) {
        printf("Hello, Sailor!");
    }

    var arr: [..] int64;
    arr.add(1);
    for arr printf("Remaining Value ????: %lld\n", it); // should not print anything!
}

• There is now a -emit-llvm flag to dump generated LLVM IR to a file.
• Unions are now available, simply use the union keyword in place of struct.
• break and continue now work for while-loops.
• os() now checks against LLVM’s support operating system types and the target triple of the compilation. See Triple::getOSTypeName for a list of valid inputs. The input is case-insensitive.
• Raspberry Pi 4 is now officially supported. Relevant tweet.
• Diagnostic-esque information is no longer printed to the terminal by default. There is now a -v flag to print out this information for debugging. There’s also a verbose_diagnostics Build_Option available to get this information out of a compilation through a metaprogram.
• while-loop conditions now coerce-to-bool.
• There are now built-in typealiases to reference the types for <string>.length and <array>.count fields. They are __builtin_string_length_type and __builtin_array_count_type.
• Global variable structs are now properly default-initialized based on the default initializations of their fields.
• castano has added support for referencing let-constants declared within structs.
• castano has added a Sublime Text highlighting extension.
• castano has added support for multiline string literals similar to how they are implemented in Swift:

 
#import "Basic";
#import "LibC";

let singleLineString = "These are the same.";

let multilineString = """
These are the same.
""";

let multilineString2 = """These are the same.""";

let multilineString3 = """
    These are the same.
    """;

func main() {
    // Multi-line string equivalency.
    printf("%.*s\n", singleLineString.length, singleLineString.data);
    printf("%.*s\n", multilineString.length, multilineString.data);
    printf("%.*s\n", multilineString2.length, multilineString2.data);
    printf("%.*s\n", multilineString3.length, multilineString3.data);
    assert(singleLineString == multilineString);
    assert(singleLineString == multilineString2);
    assert(singleLineString == multilineString3);

    let indentation = """
        This text has indentation.
        And that is fine.
        """;

    printf("%.*s\n", indentation.length, indentation.data);

}

• Support for passing and returning structs to/from C has been improved.
• There are now strideof() and alignof() operators. These return the stride and alignment of any input type. Unlike C, the the size of a struct is the amount of space required to store the struct, the stride is the amount of space needed to store the struct with padding to its alignment.
• There is now support for unary ! and unary ~.
• There is now support for a rudimentary level of struct inheritance:

 func test_inheritance() {
    struct Node {
        var i = 10;

        func do_a_thing() {
            printf("Node do_a_thing()\n");
        }

        func do_a_thing2(n: *Node) {
            printf("Node.i: %d\n", n.i);
        }
    }

    struct Node_Child : Node {
        var b = "test";

        func do_a_thing3(n: *Node_Child) {
            printf("Node_Child.i: %d\n", n.i);
            printf("Node_Child.b: %.*s\n", n.b.length, n.b.data);
        }
    }

    var child: Node_Child;
    child.do_a_thing2();
    child.do_a_thing3();

    Node_Child.do_a_thing();
}

I want to again thank the contributors for picking up this project and helping to improve it!

The code for the compiler can be found at: jiyu. Pull requests, feature requests, and issue reports are all welcome.

• There is now a psuedo-test-suite program located at <jiyu>/tests.jyu. This automatically compiles the test programs in <jiyu>/tests. This is a way to quickly verify that changes to the compiler don’t break some basic features.
• String-literals can be implicitly cast to *uint8 and *int8. This is a convinience for passing string-literals to C functions.
• Added compiler_run_metaprogram(). This function allows metaprograms to compile and run other programs at compile-time.
• There’s now support for loading libraries that the user has specified with the library keyword into metaprograms. This is a seemless operation, so there is no extra work required of the user. This may not entirely work with user-installed libraries on Linux/Mac at the moment, but this will be improved over time.
• The compiler now accepts the “-c” switch to end compilation at generating an object file.
• The compiler now accepts the “-o” switch to specify the target file/executable name to output to.
• The switch “-triple” is now “-target”.
• ahmadrosid has added some missing functionality on Linux and the compiler should work on Linux now.
• Control-flow statements break and continue are now available and working.
• There is now a version of for that gives a pointer to array elements instead of a const-copy of the elements. The syntax is “for * <array_typed_expression> { … }”.
• castano has added an open-half version of for, which excludes the value of the upper-bound expression of the for range. Example: “for 0 ..< 2 { … }”, would be equivalent of “for { int it = 0; it < 2; it += 1) { … }” in C.
• if-conditions now have coerce-to-bool. Integer, float, and pointer-types are now automatically compared to literal-0, string is automatically compared to empty-string.
• There is now syntax for calling member-functions of structs with more ease:

struct My_Struct {
	var i: int;
	// You can overload between a reference and a pointer.
	func do_something(data: My_Struct) { .... }
	func do_something(data: *My_Struct) { ... }
	
	func do_something2(data: *My_Struct) { ... }
	
}

func my_func() {
	var data: My_Struct;
	data.do_something(); // _data_ is implicitly passed to My_Struct.do_something(data: My_Struct);
	var data_ptr = *data;
	data_ptr.do_something(); // _data_ptr_ is implicitly passed to My_Struct.do_something(data: *My_Struct);
	data.do_something2(); // A pointer is taken of _data_ and passed to My_Struct.do_something2();
	
	// You can still call these functions the traditional way, too
	My_Struct.do_something(data);
	My_Struct.do_something(*data);
}

• castano has added compiler_load_string(), exposing previously internal functionality to directly compile code from strings, in metaprograms.
• Cross-compilation support has been improved. I have demonstrated running Jiyu code on an iOS device, and running Jiyu code bare-metal under Qemu.

Additionally, there have been a number of fixes and changes to improve the compiler, by myself and others.

There’s currently work being done to import C-headers directly into Jiyu code without any external tools. I will write another post documenting that once it is working well.

I want to again thank the contributors for picking up this project and helping to improve it!

The code for the compiler can be found at: jiyu. Pull requests, feature requests, and issue reports are all welcome.

Libraries

Libraries and macOS frameworks can now be specified using the declarators library and framework, respectively.

#if os(Windows) {
	library "msvcrt";
	library "opengl32";
}

#if os(MacOSX) {
	library "c";
	framework "OpenGL";
}

These inform the built-in linker command on what libraries to link, and will eventually inform the LLVM JIT on what libraries to load.

Modules

The compiler now looks for module-libraries in the jiyu/modules folder. The folder is located by searching up the directory tree from the executable’s path for a folder named jiyu, then modules is appended to the path. This will likely change as the compiler matures, but it currently makes working with modules easy. There is a new #import directive to reference a module from the scope that the directive is declared in. Since modules are intended to be referenced from many parts of a program, they exist in a scope-bubble of their own and cannot see declarations in the program’s global scope. Due to this, modules must also import other modules that they depend on.

#import "Array";

func main() {
	var array: [..] uint8;
	array_add(*array, 1);
}

Compiler

There is now a Build_Options struct for specifying compilation options. Currently, only the output executable name and target-triple can be specified. Consequently, create_compiler_instance() -> *Compiler now accepts a *Builder_Options parameter. Build options for a compiler instance can not be changed after create_compiler_instance() is called.

struct Build_Options {
	var executable_name: string;
	var target_triple: string;
}

// ...

#import "Compiler";

func @metaprogram main() {
	var options: Build_Options;
	options.executable_name = "my_cool_program";
	options.target_triple   = "arm-none-elf-eabi";
	
	var compiler = create_compiler_instance(*options);
	
	// ...
}

LLVM

• Jiyu programs can now be debugged in your favorite debugger!

• Aggregate types are now passed via an invisible reference.
• As mentioned previously, one can now specify a target triple to the compiler. LLVM will honor this, enabling cross-compilation.

Conclusion

With these few changes, the compiler and language should be a bit more practical to use. I plan to post updates on a monthly basis, granted there’s enough changes, or a big enough change, that’s worth talking about.

The code for the compiler can be found at: jiyu. Pull requests, feature requests, and issue reports are all welcome.

Back in 2015, I had started a compiler project, called htn, with just the goal of being able to compile code, interact with C-code, and output to a handful of assembly-language targets, including 6502, 32-bit ARM, and 32-bit x86. I’ve since dropped the project due to other projects and my day-job at-the-time consuming much more of my free-time. I had been thinking about this project often over the years as I had enjoyed the work I was doing on it. Unfortunately, the code isn’t maintainable for me anymore, I have grown considerably as a programmer since and am unwilling to change substantial portions of the code just to get it to a point of maintainability. Doing so would require nearly starting over again; so I started over, again.

Jiyu

This time around, I’ve built an LLVM-based compiler with an out-of-order compilation model not unlike any modern language created in the last 5 or so years. My initial goals are quite simple: implement an appealing syntax that reduces friction and enables ease of refactoring, have the ability to just drop the compiler and its dependencies into existing toolchains for cross-compiling, and be able to manipulate memory unencumbered. Fortunately for me, the former and latter-most issues have largely been solved by other projects in this space. I really, really, enjoy Swift’s syntax, but I despise its type system as the design of it makes writing code, that needs to read and write unmanaged memory, difficult. I also really like Jai’s type system; it is reminiscent of C’s, but with built-in array, and string, types, and less confusion. I’ve opted to adopt both these things, let’s look at some code:

func @c_function printf(fmt: *uint8, temporary_c_vararg) -> int;

func main() {
	var b = cool_function(2);
	printf(“Value of b: %f\n”.data, b);
}

func cool_function(a: float) -> float {
	return a * 2;
}

Most of the syntax here should be familiar to anyone who’s done even a light amount of Swift programming, with a few notable exceptions. The tag, @c_function, informs the compiler that we’re declaring a function that is external C-code, and to take care in calling this function correctly. temporary_c_vararg is fancy placeholder for the ... syntax for a C-variable-argument function. Since strings are represented by a small struct containing the length of the string and a point to its data, we must dereference the data pointer before we can pass it to printf.

User-types

For user-types, the language currently supports type-aliasing and structs:

typealias c_string = *uint8;

func @c_function printf(fmt: c_string, temporary_c_vararg);

typealias FILE = void;
func @c_function fopen(path: c_string, mode: c_string) -> *FILE;
func @c_function fclose(file: *FILE) -> int32;

struct Matrix {
	var data: [4][4] float;
}

func identity_matrix() -> Matrix {
	var m: Matrix;
	
	m.data[0][0] = 1;
	m.data[1][1] = 1;
	m.data[2][2] = 1;
	m.data[3][3] = 1;
	
	return m
}

Variables are zero-initialized by default in this language. There’s currently no way to declare an uninitialized variable, but that is on the list of things to do. Structs may be populated by any other type of declaration, including functions. There’s currently no notion of member-functions, however, that’s another thing to come in the future.

struct Matrix {
	var data: [4][4] float;
	
	let SIZE = 16;
	
	func ortho(left: float, right: float, top: float, bottom: float, near: float, far: float) -> Matrix {
		// ...
	}
	
	func identity() -> Matrix {
		// ...
	}
}

func main() {
	var matrix = Matrix.identity();
	// ...
	var size = Matrix.SIZE;
}

Static-if

Another borrowed feature from Jai, is a scope-aware static-if:

func main() {
	#if os(MacOSX) {
		var b = 1;
	}
	#if os(Windows) {
		var b = 2;
	}
	b = 3;
}

The usefulness of this is that whenever code in a disabled static-if block is changed, it is still being parsed, enabling syntax-specific errors to still fire. os() is a special construct that checks the input for a known operating-system name, then resolves to a Boolean literal if that operating system happens to be the target platform. This is not too dissimilar to how sizeof() works, except, in this case, “MacOSX” and “Windows” are special identifiers that os() recognizes. Due to the complexity of allowing arbitrary expressions in the #if condition, I’ve specifically disallowed arbitrary expressions. Only expressions that specifically fold or resolves to literal values may appear here; otherwise, even just allowing let constants in the condition causes a number of dependency issues relating to when that let exists in a scope that the #if can see. We cannot have #if depend on information that only exists after another #if or #load has resolved in order to maintain simplicity in the out-of-order compilation system.

Templates

There’s currently limited support for function-templates. Struct-templates will will be on their way, eventually. Template declaration and resolution rules are similar to Swift’s: template argument names are declared between < and >, the type of the template arguments are deduced by arguments passed to the template function.

let NEW_MEM_CHUNK_ELEMENT_COUNT = 16;

func array_reserve<T>(array: *[..] T, _amount: int) {
    var amount = _amount;
    if amount <= 0
        amount = NEW_MEM_CHUNK_ELEMENT_COUNT;

    var new_mem = cast(*T) malloc(amount * sizeof(T));

    if array.data != null {
        memcpy(new_mem, array.data, cast(size_t) (array.count * sizeof(T)) );
        free(array.data);
    }

    array.data = new_mem;
    array.allocated = amount;
}

func array_resize<T>(array: *[..] T, _amount: int) {
    array_reserve(array, _amount);
    array.count = _amount;
}

func array_add<T>(array: *[..] T, item: T) {
    if (array.count+1 >= array.allocated) array_reserve(array, cast(int) 
    
    array.allocated + NEW_MEM_CHUNK_ELEMENT_COUNT);

    array.count = array.count + 1;
    (<<array)[array.count-1] = item;
}

func array_reset<T>(array: *[..] T) {
    array.count = 0;
    array.allocated = 0;

    if array.data != null free(array.data);
    array.data = null;
}

func main() {
	var data: [..] float;
	
	array_add(*data, 1.0);
	array_add(*data, 2.0);
}

Compile-Time Function Execution

There’s rudimentary support for CTFE. One may run any program at compile-time by invoking the -meta flag on the command line: jiyu -meta build.jyu. Additionally, the main function may be marked by the tag @metaprogram to force the compiler to always run that program at compile-time. Just a small change to our first code example will cause the program to always be executed at compile-time:

func @c_function printf(fmt: *uint8, temporary_c_vararg) -> int;

func @metaprogram main() {
	var b = cool_function(2);
	printf(“Value of b: %f\n”.data, b);
}

func cool_function(a: float) -> float {
	return a * 2;
}

Compile-time code may also harness the compiler’s main driver functions to compile, or run, other code. I won’t delve too deeply into that, but here is a code sample:

#load "Compiler.jyu";

func @metaprogram main() {
    var compiler = create_compiler_instance();
    compiler.executable_name = "run_tree/game";
    
    if compiler_load_file(compiler, "src/main.jyu") != true return;
    if compiler_typecheck_program(compiler) != true return;
    if compiler_generate_llvm_module(compiler) != true return;
    if compiler_emit_object_file(compiler) != true return;
    if compiler_run_default_link_command(compiler) != true return;
}

Conclusion

This has been a brief look at the language. There are still a number of small issues to tackle (like specifying libraries the code depended on; currently the compiler has the linker command baked-in), but I plan to fix these within the coming weeks. Please see the jiyu_game project for more in-depth use of the language.

The code for the compiler can be found at: jiyu. Pull requests, feature requests, and issue reports are all welcome.

Over the past few years I’ve been increasingly losing interest in interacting with major operating systems for my day-to-day needs. Major operating systems have become largely bloatware that is more intrusive to the user’s experience than it is useful. I will try to outline the surface features at this time. Features that are non-user facing (within the kernel, programming interface, etc..) will be outlined at a later time.

These are the basic philosophies for designing this new OS:

1. Operating systems should be designed to help the user get from point A to point B or perform task XYZ with as little friction as possible.
2. A design choice should not compromise philosophy(1), or philosophy(2).
3. The OS should be stripped of all bloatware.
4. The OS should be as unintrusive to the user as possible.
5. Upgrading the OS should improve the user’s experience, not degrade it.
6. The OS should be as easy and as accessible as possible; it should support multiple points of interaction by default. Whether the user wants to use a keyboard and mouse, a touch screen, or even a gamepad, the OS should be agnostic to these common points of interaction and should provide a pleasant experience.

The Terminal

Currently, command terminals support basic IO, command execution, and simple text rendering (coloring, italics, bold face, etc..). These aren’t bad features at all, this small feature set is what makes interacting with consoles still a pretty good experience. However, this limiting feature set induces poor implementation choices of software that uses the console as a primary user interface. A prime example of a such program is the GNU debugger. GDB is an otherwise good debugger, but it’s console-based user interface makes even the most trivial of debugging tasks impossible to efficiently get done. We could do better.

Bitmap rendering

I first heard about this idea from Gary Bernhardt of Destroy All Software in his talk A Whole New World. Although Gary mentions a few interesting uses for terminal raster graphics, there’s much wider implications of what this feature would be capable of. This enables full-featured UI’s from within a terminal. A terminal app could render a webpage, draw game graphics, or display a powerpoint. Typically, a terminal app like (for example) gource would have to open a separate window to draw its output. The user then has to exit the newly opened window the click back into the terminal. This is unneeded friction that does nothing but waste the user’s time.

Mouse/agnostic pointer input

Since we have to support philosophy(6). The terminal should be able to do more with a pointer device than just copy-paste. As a natural, harmonious progression with bitmap rendering, a terminal app should be able to capture basic pointer input like position and clicking/tapping.

Scripting language

The terminal should support a robust scripting language to allow the user to automate tasks or interact with the system. Although this is a current feature of most terminals, many terminal languages are designed to do most of their work in the form of executing other terminal programs. Scripts in this language should be native to the system; they should be able to work like any other program without having to invoke other terminal programs to perform a task.

Windowing system

Although the features proposed in The Terminal would make windowing obsolete, there’s still a major use for apps exclusively designed around a window system. The terminal is typically thought of as a power-user-y interface; it inherently requires a keyboard and a bit of learning to effectively use it. A windowing system still provides a smaller amount of friction than the terminal.

iOS and Window’s MetroUI are forward-thinking systems. They maximize simplicity while minimizing (or trying to) friction. A simple tile-based UI would be an ideal solution to support philosophy(6). It would allow the user to use a keyboard and mouse, a touch screen, a gamepad, or any other abstract input devices without any compromises. This could be further extended to allow navigating the entirety of the filesystem from this interface.

If well-executed, we could do away with desktop environments. Although they’re highly preferred now on desktop PC’s, I believe power-users will become accustomed to these low-friction systems.

Developer Tools

Developer tools for many OS’ tend to be bloated and not very user friendly. Although we can’t strictly avoid legacy toolchains, we can at least provide a better user-experience for developers as part of the OS ecosystem.

Debugger

One of the things Windows does right that Unix-based OS’ do wrong is that Windows provides debugging utilities as part of the kernel. Anyone wishing to implement a custom debugger can simply link against the kernel’s public interface library and can get language-agnostic debugging callbacks that any user program can easily interpret and manipulate. We can take this core idea of the debugger being a part of the OS a little further by having executable object parsing, op-code disassembling, and symbol information loading being shipped as part of the debugging API. User-built debuggers don’t have to rely on these OS provided utilities, but the advantage to having a standard API for these is that anyone trying to implement a reasonable debugger without large dependencies, can just depend on the OS provided libraries and they should just work.

API deprecation

Some API designers tend to believe that deprecation is “mark for removal”. I don’t agree with this idea. A deprecated API should remain available as long as there is software to depend on it. If the API in question breaks one of the core philosophies, it may be justifiable to remove the API from the core OS, but it must still be accessible to any user who wishes to use it through some other means (such as a compatibility package).

Adoption

This will probably be the trickiest part to execute. If we look at Steam’s hardware survey alone, Windows dominates the market at 95% of users. Although this is just based on gamers, I’m sure these numbers tip even more in favor of Windows for casual PC users and business users. I don’t believe Mac OSX is a real competitor in this space since the target for this OS would be PC users (where Mac OSX is solely designed for Mac devices). Although Linux is somewhat popular in the open source space, I don’t think it is a direct competitor. Adoption for Linux is still very small and still suffers from some of the problems I’ll outline below.

GPU drivers and games

This will be one of the most important aspects of the OS to get the gaming space to adopt. Unfortunately, the main issue here is a chicken and egg problem. In order to get GPU vendors to implement drivers for our OS we need to have a significant standing in the gaming space by having games and users. In order to have games and users, we need to have good driver support to get developers to even consider porting their games. Fortunately, we can still get the process started by implementing our own drivers for a subset of hardware (Intel’s integrated graphics for example have open specifications. Although this is less than ideal, this would be a step in the right direction.

Pre-installed on devices

This would be a goal to reach much farther in development, but it is very important to get the OS established as software that is pre-installed on machines. This is one of the main aspects of Linux that causes its low adoption rate (although Ubuntu tried this for awhile and failed).

I’ll explain the importance of this with an analogy. When users purchase an Android device for example, most do not buy the device with the intention of side-loading a different flavor of Android. Most users purchase an Android phone and use the pre-installed version of Android because it is just there and there’s no friction involved in having to set it up. Likewise, if Android phones only shipped with vanilla versions of Android with the option of installing manufacturer versions, the install base for manufacturer-based versions (i.e., Samsung’s TouchWiz or HTC’s Sense) would be much lower, perhaps even near non-existent. To further expand on this thought, CyanogenMod (arguably the largest non-OEM Android version, which isn’t typically pre-installed to devices) claimed in August 2015 to have 50M+ users. According to The Verge, Android phone sales had hit ~271M units in just the quarter leading up to August 2015. The number of Android devices with pre-installed versions far, far out weigh the number with custom OS versions installed.