The Time I Tried Making My Own Programming Language

Date: 2023-09-14 23:34

Sometimes, you'll be writing code and decide you want a challenge. I'm the kind of person who will try something I can't do, fail, try some more, and somehow manage it. This strategy doesn't always work, as I proved with the W++ project.

Back in $MONTH, I decided to try my hand at making a programming language. I immediately threw any ideas about writing a compiler into the bin, because writing a compiler is insanely difficult.

Compilers Are Hard

Compilers are made by very smart people with a deep understanding of the language they're writing for, how computers work, and how to handle errors and edge cases. Most compilers also provide build-time optimisations that make the program run faster, which are even harder to implement, as they can introduce instability if done incorrectly. Compiler devs are unilaterally smarter than amateurs like me. Most would say, "Cute. You're trying to write a programming language. You're just an idiot who hasn't experienced the real world. Get back to reality."

Design

First order of business for any language is to design it. I wrote a specification document, which was basically a source code file. You can download the specification here, if you so wish.

This bit was easy, and I ended up making something that looks like Python and C#. Cool.

How To Write a Programming Language

Generally, there are 3 steps to writing a programming language:

Lexer

Short for lexical analyser. This converts your source code into "tokens", which are objects that define a type and value.
You can have multiple types of tokens for keywords, expressions, values, comments, punctuation, and so on.
This is trivially easy, in fact, lexical analyser generators exist that can generate a lexer for you.

Parser

This takes your tokens and parses them to create an abstract syntax tree. This takes your list of tokens and works out what to do with them.
This is initially simple, but can get really complex really quickly.

Execution (Pick one option)

Compiler

A compiler takes your parsed code and converts it into assembly.
This code is then thrown into a separate assembler and linker, which build an executable file for you to run. Building an assembler and linker isn't necessary, as those are simple and generic to many languages.
This is notoriously difficult (see my section on that).

Transpiler

This program converts your tokens into a compiled language, such as C or Java, and throws them at a compiler.
This isn't as hard as compiling, but can be complex if your language is different from the one you're compiling to.

You're also just throwing your code into a black box, which isn't a great idea.
You also need to be familiar enough with the target language to automatically convert code over to it.

Interpreter

This program simply opens a file, goes through it line by line (or in this case, parsed token by parsed tokens), and runs it.
This is SLOW.
It's also the easiest option.

I know that there are more ways to do the Final Step, but that's not important.

Writing The Lexer

This was trivially easy. I managed to get an artificial intelligence to do it. Crazy what regular expressions can do.

The Parser

Writing a basic parser is easy. Say you want to define a string:

str x = "3";

There are 4 elements:

The str keyword;
The variable name x;
The equals expression
The value ("3")

What I did was write a Parser script that was fed a series of tokens and checks tokens and moves on to the next one. It meant that if it saw a str token I could make a parseString() method that parses the string. All you have to do is expect a specific series of tokens, and throw an error if you don't get what you expected. A simple expect()method does the trick. Jump to the token you are looking for, collect all the info you need, create some kind of string definition, be it a dictionary or a class. Pretty simple stuff.

Initially, writing the parser was quite simple and I was getting the hang of it. I saw how flexible the system I had set up was, which took a LOT of trial and error to do, and was about to start writing code for parsing functions when I suddenly hit an unexpected but pretty massive snag.

The Issue

Examine this code:

def function() {

int x = 4;

str y = "hello";

}

From the perspective of a lexer/parser combo, how do you tell what code is inside the function and what isn't? It's more difficult than you think.

Traditionally, programming languages defined an "end" line or similar that would tell the programming language that the function has ended. However, I didn't want to add this, as it was a pretty janky solution that makes for needlessly complex and ugly code.

As such, the only way to tell if a function has ended is with closing curly braces, which are super ambiguous.

While I was fixing this issue, which I never ended up fixing, I realised an even bigger issue.

The Bigger Issue

As I tried to fix my function parsing, I began to think. I quickly realised that the language was a lot more complex than I thought. For example, it had to handle function definitions with both parameters and no parameters, statements with either explicit logic or variable references, proper error handling, some way of getting static typing working when I was using Python, a dynamically typed programming language, evaluating variable names, logic, bitwise operations, lists and dictionaries. I also wanted to implement classes which would be a nightmare to sort out, as they had constructors, methods, attributes, subclasses, instance checking, and a lot more. Suddenly, I realised I had bitten off more than I could chew.

And that's why W++ failed. I tried to do too much.

Conclusion

I learnt a lot when working on W++. It taught me a valuable lesson: Be appreciative of existing programming languages and don't try to enter the market for no reason. It's disrespectful to the people who dedicate their time to making existing languages work. There's nothing wrong with them.

Unless it's JavaScript.

Fuck you, Netscape. You did this to us. It's all your fault.