Lukas’ weblog

Some ways to approach focus in life

2025-12-31T15:00:00+01:00

Motivation

It is a repeating pattern for me, I am craving for: In some quiet minutes after work days, I am sufficiently good at getting projects done. Once leisure time (holidays / vacation) reaches me, I am craving for the moment to sit down, revisit the current state of affairs and plan ahead what the next steps are. I like that moment. It needs a lot of mental freedom, but I like it so much, that I make sure to get such a moment from time to time.

The most high-level question one can answer in this moment is “What do I want to focus on in my life?”. I am lucky enough, that I am not in existential mode all the time. At Aikido in Graz recently, I met a person coming from Odesa to Graz one month prior. We had a longer chat and that person obviously only had one focus in mind: ‘SURVIVE’. I am fortunate enough to be able to focus on other topics as well like digital typesetting or languages.

In this post, I want to present three frameworks which can help to identify what one wants to focus on.

Framework 1: Maslows’s pyramid of needs

The categories are taken over from the famous pyramid whereas the notes next to it, are my personal ones. It is my personal goal to have some projects in each category. If I have many projects in the category ‘Esteem’, but few projects in ‘Physiological Needs’, I am lowering the priority of ‘Esteem’ projects and increasing the priority of ‘Physiological Needs’ projects.

Self-Actualisation (most sophisticated need): get bored to initiate creativity intentionally; implement new approaches/ideas; defining goals for your next 1/5/10 years in life; spread profound knowledge you gained by long-term efforts
Esteem: interact/discuss matters with loved ones to reinforce ideas/concepts and dismantle prejudices; do something consciously which was once difficult in your life, but is now trivial; give talks; engage in a podcast episode; implement a difficult project having a good strategy at your avail
Love & Belonging: plan ahead how/when you can meet friends/family to prevent neglect; strengthen your romantic relationships; organize communal events; invite someone over; talk to the neighbors; give tiny gifts or small-talk to people you know but you are not familiar with
Safety needs: physical exercise; think through fallback options, if personal belongings get lost; discuss insurance options; declare assuring thresholds for income versus expenses limits; invest into physical or digital security
Physiological needs: plan ahead food supplies; have emergency food in your storage; take care of your household and workplace; buy new clothes before the old ones are falling apart; get enough sleep

I am pretty sure my notes are a bit of a stretch. Some might be miscategorized w.r.t Maslow and many may not apply to your idea of ‘projects’. But it should give you an idea of my approach: look at each category, write down notes, and attach my (mostly existing) projects to the notes. After this process, I might recognize that my current projects are egocentric. Or that I could share my new knowledge in a certain talk.

Framework 2: time dimension

Another approach can be to have projects in all of these three categories:

past: finish an old project which still makes sense, but had too little value to be finished; revisit old responsibilities which are usually self-initiated but should be done (→ volunteering work); renew friendships; summarize past experiences
present: finish work-in-progress projects; answer inquiries; review pull requests
future: plan ahead schedules; think through what would improve your life; consider in which way this new technology could improve your life

Framework 3: four necessities

This is a list of categories which I came up with personally. It is a list which makes it easier for me to achieve a uniform distribution of projects (e.g. 10 current projects in each category).

mental: walks in nature; listening to thoughts of other people (face-to-face or in podcasts); implement creativity projects; expand knowledge; learn new skills
social: host friends; engage in local clubs & events; discuss life with people from other cultures; understand how society works on a local/regional/national/international level; build relationships & friendships
physical: stretching exercises; exercises for strength; running; bicycling; ball games; martial arts
representational: report project status; publish your knowledge; micro-blog; write blog posts

Conclusion

I am fully aware that my blog post has many assumptions. I have a very broad definition of project (“any TODO which takes more than 1 day of work”) and I consider it in a flat structure (umbrella projects and subprojects are treated alike). It does not help many people to ‘even out’ projects in certain categories. But if this blogpost helped you to regain a high-level perspective what you are focusing on in life, this blogpost was worth it writing it down.

U([0, 1)) → U([0, 1])

2025-12-28T10:00:00+01:00

Motivation

Given a sampler for a uniform distribution in [0,1), can one attain a sampler for a uniform distribuition in [0,1] (i.e. the value 1 is possible as well)?

First, a disclaimer: the question itself is ridiculous and a purely mathematical thought experiment. I also don’t have an solution here. But it was fun to think it through. It is ridiculous, because either you need it in practice on a computer. Then you need to consider the realities of IEEE 754 with its incapability to represent a continuous interval. Then you need completely different approaches than represented here. Or you consider the mathematically continuous range [0,1) in ℝ. In this case, measure theory tells you that the measures of [0,1) and [0,1] are equal and thus the question becomes boring in all mathematical contexts.

Ok, so we can sample values from a uniform distribution U([0, 1)) and need an algorithm with returns one value more with the same probability.

First idea

Let \(s ∈ U([0,1))\).

\[ u(s) := \frac{1}{1-s} \]

So with \(f(x) = 1-x\) I can map \(U([0,1))\) to \(U((0,1\))] (sorry, the markup language parser screws up the braces). Function \(f(x) = 1/x\) is fun, because it gets arbitrarily close to 0, but does not reach it. The argument 0 itself is disallowed in ℝ (c.f. partial function). We use this mechanism in the way that 1 is never returned by the sampler \(U([0,1))\). However, the idea fails because it completely distorts the uniform probability and value 0 cannot be attained.

Second idea

Let \(s ∈ U([0,1))\) and \(t ∈ U([0,1))\).

\[ u(s, t) := \frac{s+(1-t)}{2} \]

Now the idea is to sample twice. We use one value in [0,1) (namely \(s\)) and one value in (0,1] (namely \(1-t\)). Since we now apply addition, we don’t distort the probability. Division by 2 just maps the sum from interval (0, 2) to (0, 1). Oh right, we already know the reason why this result is invalid: values 0 and 1 are excluded.

Third idea

Now I started integrating the idea of rejection sampling. Draw a sample and test it. Check it, potentially reject it and do something else.

Let \(s ∈ U([0,1))\) and \(t ∈ U([0,1))\).

\[ u(s, t) := \begin{cases} s & \text{if } s < 0.5 \\ 1-t & \text{else} \end{cases} \]

Due to the case distinction, we either pick a value in [0, 0.5) or (0, 1]. With set union, this makes [0, 1] which is our desired interval. However, did we preserve the properties of a uniform distribution? It immediately looks suspicious if the intervals of the two cases overlap, because this distorts the probability. Value 0.3 might be returned due to case 1 or 2 whereas value 0.7 can only result from case 2. Hence, the values in (0, 0.5) are twice as likely as the values [0.5, 1].

Fourth idea

So let us align the intervals.

Let \(s ∈ U([0,1))\) and \(t ∈ U([0,1))\).

\[ u(s, t) := \begin{cases} s & \text{if } s < 0.5 \\ 1-\frac{t}{2} & \text{else} \end{cases} \]

Now the two cases cover the intervals [0, 0.5) and (0.5, 1]. So we miss value 0.5 … damn.

Fifth idea

Does it help if we make the condition independent of the returned value?

Let \(r, s, t ∈ U([0,1))\).

\[ u(r, s, t) := \begin{cases} s & \text{if } r < 0.5 \\ 1-t & \text{else} \end{cases} \]

The two cases cover the intervals [0, 1) and (0, 1]. Its set union is [0, 1] which covers our desired interval. However, value 0 can only be created if the first case applies whereas 1 can only be reach through the second case. All other values can be returned from either case. So values {0, 1} are less likely than other values.

Fifth idea

Ok, can we introduce a special handling for the one value we want to extend upon?

Let \(s, t ∈ U([0,1))\).

\[ u(s, t) := \begin{cases} \begin{cases} 0 & \text{if } t < 0.5 \\ 1 & \text{else} \end{cases} & \text{if } s = 0 \\ s & \text{else} \end{cases} \]

If we start to create two return values for one value, we obviously split the probability into two. Thus we don’t create a uniform distribution, because values {0, 1} are half as likely.

Sixth idea

Ok, let us get back to previous ideas. If we don’t use less-than but less-than-or-equal in the condition, we can get a closed interval. This way, we can design our desired interval.
Now, I need to use a loop to express the process:

forever
1. Let \(s, t ∈ U([0,1))\).
2. if \(s\) ≤ 0.5, then return \(s\)
3. if \(t\) ≤ 0.5, then return \(1-t\)

So we either return a value from the interval [0, 0.5] or from [0.5, 1]. Because we neglect the cases when the condition is not true, it is difficult for me to quantify the probability distribution in this case. I think the fact that value 0.5 occurs in both cases makes it more likely than other values. However, I think this approach is also bad, because we have no guarantee of termination. We cannot guarantee that we always get a sampled value.

Conclusion

I looked at different ideas to extend U([0,1)) to U([0,1]), but failed at finding a mathematically beautiful solution. I am neither a probability theorist nor did any literature research. I wrote down the question several years ago during my studies as a note and discussed it with work colleagues recently. The brainstorming lead to what I wrote into the disclaimer. Finally, I think combining a function and its inverse (functions \(10^x\) and \(\log_{10}(x)\)) might be a viable approach, but I lacked creativity to combine them usefully. Facts like \(1/x \neq 0\), \(10^x \neq 0\), \(\log_{10}(x) \neq 0\) and \(10^0 = 1\) can include/exclude the missing value. But all of it is just food for thought.

Raporto de la eŭropa tago de lingvoj

2025-10-11T00:00:00+02:00

Ĝisdatigo 2025-10-19: Amiko sendis korektadon al mi pro miaj eraroj. Mi ĝisdatigis ĉi tiun artikolon.

Enkonduko

La Esperanta klubo de Graz partoprenis en la eŭropa tago de lingvoj. Mi ĉeestis kaj helpis ĉe la stando antaŭ la ŝtuparo de Schlossberg. Amiko demandis artikolon pri la sperto. Voilá!

Eŭropa tago

Je la 26-a de septembro, la eŭropa tago de lingvoj okazis je la 25-a fojo. Ĉi tiu tago festas la diversecon de eŭropaj lingvoj kaj konektas homojn kiuj interesiĝas pri aliaj lingvoj. Graz estas la 2-a plej granda urbo de Aŭstrio kaj havas multajn lingvajn influiĝojn. Slovenio estas sufiĉe proksima kaj multaj slavlingvaj parolantoj loĝas en Graz aŭ vizitas la urbon. Lernantoj kutime elektas inter latinidaj lingvoj kiel la itala, la hispana, la franca, aŭ la latina mem. Plie Graz havas multajn universitatojn. Eksterlandaj studentoj studas en Graz por unu aŭ pluraj semestroj kaj oni povas havi bonan sperton lerni pri aliaj kulturoj malproksime de Aŭstrio. Mi mem interagis kun homoj de Kolombio, Sud-afriko, Ĉinio, kaj Japanio. Konklude, Graz havas realan sperton pri la temo kaj la festo ekokazis je la 9-a horo en la urbocentro ĉe Schlossbergplatz.

Dum la tago multaj prezentaĵoj rilate al lingvoj okazis. Je 10:45, Muhammed Dumanli prezentis siajn Poetry-Slam-tekstojn fokusitajn al kulturaj aspektoj. Je 13:15, dancogrupo organizis komunan ‘linian’ dancon. Ĉiu partoprenanto rajtis danci laŭ la instruo de organizantoj kaj la usonana kontrea muziko. Je 14:30, oni povis aŭskulti anglan teatraĵon. Je 15:45, Simon Ošlak-Gerasimov legis sian tekston pri la slovena aspekto de Graz. Memoru ke Graz baziĝas sur la slovena vorto ‘gradec’ esperante ‘malgranda kastelo’.

La organizantoj de la festa tago preparis tri centrojn en la urbo. Unue, la proksima museo de Graz ofertis komunan urbopromenon pri lingvoj. Ili ankaŭ ofertis prezentaĵon pri la 30-jara membreco de Aŭstrio en la Eŭropa Unio. Due, multaj institucioj (p.e. urba biblioteko, pedagogia universitato de Stirio) ofertis specialan programon pri lingvoj en siaj domoj. Trie, oni starigis tri tendojn ĉe Schlossbergplatz. Diversaj grupoj havis standojn en ĉi tiuj tendoj. ‘Institut culturel franco-autrichien’ klopodas pri pli bona rilato inter Francio kaj Aŭstrio. Ili ofertas franclingvajn kursojn. Aparte la ‘Integrationsreferat der Stadt Graz’ (urba institucio por integrado) kaj la Eŭropa Komisiono en Aŭstrio prezentis sin. ‘deutsch in graz’ ofertas germanlingvajn kursojn por eksterlandanoj kaj prezentis ilin. Inter tiuj grupoj, ses esperantistoj subtenis la Esperantan standon. Esperanto havis plenumitan tablon de libroj kaj lingvajk afiŝoj.

La interago kun interesuloj tre plaĉis al mi. Se oni mencias Esperanton, diversaj kategorioj de respondoj ekzistas. Viro ne konis Esperanton. Familioj kutime tuj klopodas instrui infanojn pri la elparolado de bazaj vortoj. Virino tuj respondis ‘estis bona ideo, ĉu ne? Sed ne funkciis, ĉu ne?’. Kiam ni interagis kun homoj de Indonezio, ni serĉis komunajn ecojn de niaj lingvoj.

Konkludo

Estis amuza tago por mi kaj taŭgis ke ni reprezentis Esperanton publike en ĉi tiu evento.

Review: Vienna Coffee Festival 2025

2025-09-13T20:00:00+02:00

Motivation

A friend of mine invited me to attend the Vienna Coffee Festival together. So we did for one day. I wanted to write a summary, what I experienced and learned.

The event

The Vienna Coffee Festival (edition 2025) takes place from Friday (yesterday) to Sunday (tomorrow). The entrance poster mentions “11 years”. So I assume it takes place the eleventh time. It is an event for coffee lovers where coffee brewers, coffee machine producers, cup and mug producers, coffee roasters, coffee baristas, and latte artists come together to share thoughts, ideas, and let people experience different kinds of coffee. We got “Super early bird tickets” in April and paid 13€ whereas the ticket on-site costs 25€. Upon entry, you get a mug for 1€ which is paid back when you leave and bring back the mug. The core idea is that you visit various booths and they brew coffee for you; for free. So you can taste various flavors and roasts. However, there are also booths where people present talks about the economic side of coffee, explain the differences in cultures, or just play music. There was also the Austrian competition for baristas going on during our attendance.

The venue

Karl-Marx hall is used as venue for the festival. In the front, your ticket is checked and you can get your refund mug. Many coffee brewing booths come first and some talk- or music-related booths can be found in between. Not the entire hall is filled. The final third cannot be reached and the rear part is used for food stands. We got some Indian vegan lunch, but they also offer Thai or more Austrian/Czech style cuisine. Water supply and toilets are available in the north and well-maintained.

What I learned

Markus from gota coffee was so kind to present us a lot of details about coffee making. He showed us the ikawa coffee roasting machine. This machine allows you to define a profile, which defines the temperature curve the machine is following during the roasting process. Curves are shared between roasting aficionada on their website. Markus points out that the level of uniformity in the roasting process achieved by the machine is pretty good which makes results more predictable. The issue with coffee roasting is that commonly you get coffee beans in sacks of 5 kg. Thus it is not easy to try many different coffee beans with the same curves on a hobby level.

By the way, MyHomeRoast was also present at the festival. A sales woman introduced us to their approach. It seemed like MyHomeRoast has limited configurability (compared to ikawa), but they also seem to work in different price ranges. Whereas MyHomeRoast machines cost about 700€, the prices of ikawa machines are not even listed on their website.

Katharina from gota coffee made us some cascara tea. This is some herbal tea made from the dried skins and pulp of coffee cherries. She mentioned that one uses about 4g of cascara for 100ml hot water. Cascara was considered a waste product in Europe, but tea was made at plantation sites since the early days. It contains a small amount of coffee and we loved it. Katharina also introduced us to AeroPress, an Espresso equipment which uses air and human strength to build up the pressure.

Competitions

On Saturday, there were three people competing in “Latte Art” in the morning and five people competing in “Barista” in the afternoon. It is not easy to present your coffee (origin, date of roasting, flavor characteristics, …) and brewing approach (temperature, duration) while actually making the coffee in time. Each participant seemed to finish off explaining their signature drink. We observed a recognizable difference between first-time participants and experienced competitors. The jury watched each step, but recognizably only one judge was allowed to be in the vicinity of the examinee. It felt like the style of presentation, hygiene, timing, and taste were taken into account.

Conclusion

What a fun experience. It seems to be a growing community and all people have been kind to us. I am a coffee drinker and made coffee with different equipments, but what I do, is far from the professional level. It was nice to learn some stuff from a different field.

A card game, I learned during high school

2025-08-17T20:00:00+02:00

Motivation

At high school, we got bored at 10-minute breaks between classes. So my friends taught me a card game. I remember it thoroughly, but I cannot tell its origin. Technically, we called it “stress”, but looking at the German Wikipedia article, I only recognize different rules.

The rules

Distribution of cards

Each player gets 4 cards to be placed face-down in front.
Each player gets 4 cards to be placed on top of face-down cards face-up.
Each player gets 4 cards to be held in hands and only the player himself/herself may see the card.
The remaining cards are put into a stack in the center of all players face-down (drawing stack).
One card is removed from the drawing stack and placed face-up. This card initiates the game stack.
The card cementery is the final stack. Once a card is moved there, it cannot be played anymore.

Each player is allowed to exchange cards from one’s hand with the cards placed face-up. The strategic goal is to have your most valuable cards as face-up cards to your avail. What signifies “valuable” follows from the next section.

The value of a card

Players take turns counter-clockwise.

They have to play a card to react to the top card on the game stack. Suits do not matter. In order to react, one must place one or more cards on top of the game stack; commonly a card of same or higher value (the order from lowest to highest is 2 3 4 5 6 7 8 9 10 J Q K A).

If and only if you have several cards of the same value, you may place them on the game stack in one turn.
However, some cards have special semantics:

2

may be played at any point in time, a card of any value may be placed on top.

3

may be played at any point in time, this card is ‘invisible’ and thus the value of the top card below matters (if there is none, it acts like a ‘2’).

8

the next player has to skip.

9

the next card to be played must have a value less-than nine.

10

the entire game stack is moved to the cemetery. The same player starts a new game stack with any card.
If you fail to provide a card, you have to pick up the entire game stack. All those cards now constitute your cards in your hands to react with (recognize: having many cards available, can provide an advantage in your strategy to react as well). The next player continues and begins a new game stack with any card.

Figure 1. A game situation in the card game. I have some cards in my hand, four cards are placed face-up, and face-down each. I have many high cards (Ace) or special cards (8, 10) in my two decks which might make me lucky. But my opponent has some special cards (3, 9) in the face-up deck as well. The game stack is visible in the middle, and the drawing stack is barely visible on the far-right. The cemetery is not visible.

Who wins the game

First, you may only play the cards in your hands. If you have less than 4 cards, you need to draw one card from the drawing stack as long as some are available.
If you have no “cards in your hands” left anymore, you have to play with the four faced-up cards in front of you. Recognize: it is an advantage for other players to know your values now. They can slow down your progress with the last remaining cards.
If you have no “faced-up cards” left anymore, you have to play with the four faced-down cards now. Since no-one can see the value of the cards (not even the selecting player), the player picks an arbitrary one and hopes to provide an admissible card. Even if you fail four times, each turn one faced-down card was removed and thus the selecting player makes progress.

If all cards in one’s hands, faced-up, and faced-down have been placed admissibly on the game stack, the player has finished. The player which finished the fastest, has won.

Guidelines for the design of file formats

2025-08-05T18:00:00+02:00

Update 2025-10-19: I added edn as mentioned file format.

Motivation

I don’t think I am an expert on this topic, but I feel like there are some simple guidelines which don’t get retold often enough. I designed some file formats myself and have some frustrating experiences with recurring file format definition errors. Academically, I think there should be more interest in this topic instead of the common “I parsed PDF files using machine learning” papers. Anyhow, I hope some academics find some more answers to open questions, but for now let us summarize those guidelines.

Prior art

I listed the entries by my own rating of “contributes to the topic at hand” from most to least significant:

[paperDSL] paper “Design Guidelines for Domain Specific Languages” (2014) by Karsai, Krahn, Pinkernell, Rumpe, Schindler, and Völkel
[talk38c3] talk “38C3 - Fearsome File Formats” (2024-12-30) by Ange Albertini
[articleKOMPPA] article “On File Formats” (2025-05-19) by Jari Komppa
[paperLITTLE] paper “Little Languages” (1986) by Jon Bentley

Paper “Design Guidelines for Domain Specific Languages”

This paper from 2014 [paperDSL] lists 26 guidelines in various five categories. Without discussing them in detail, I am going to list their names here:

Language Purpose
- Guideline 1: Identify language uses early
- Guideline 2: Ask questions
- Guideline 3: Make your language consistent
Language Realization
- Guideline 4: Decide carefully whether to use graphical or textual realization
- Guideline 5: Compose existing languages where possible
- Guideline 6: Reuse existing language definitions
- Guideline 7: Reuse existing type systems
Language Content
- Guideline 8: Reflect only the necessary domain concepts
- Guideline 9: Keep it simple
- Guideline 10: Avoid unnecessary generality
- Guideline 11: Limit the number of language elements
- Guideline 12: Avoid conceptual redundancy
- Guideline 13: Avoid inefficient language elements
Concrete Syntax
- Guideline 14: Adopt existing notations domain experts use
- Guideline 15: Use descriptive notations
- Guideline 16: Make elements distinguishable
- Guideline 17: Use syntactic sugar appropriately
- Guideline 18: Permit comments
- Guideline 19: Provide organizational structures for models
- Guideline 20: Balance compactness and comprehensibility
- Guideline 21: Use the same style everywhere
- Guideline 22: Identify usage conventions
Abstract Syntax
- Guideline 23: Align abstract and concrete syntax
- Guideline 24: Prefer layout which does not affect translation from concrete to abstract syntax
- Guideline 25: Enable modularity
- Guideline 26: Introduce interfaces

As can be seen from the list, it considers it from a very abstract level and thus really contributes to the field. It applies generically if you want to design a file format. But it also applies in a specific context; for example if you want to redesign how mathematicians write their formulas. It is the only set of guidelines which also talks about notations.

Talk “38C3 - Fearsome File Formats”

Ange Albertini [talk38c3] has built up a lot of expertise on file formats over the last decade and gives an overview in this talk about how file formats can be abused and repurposed. In this talk, I want to focus only on one slide in particular which lists recommendations how to design a “good file format”:

In text, the ten commandments are:

Magic at offset zero (fast identification, no bypass)
Clear chunk structure (forward compatibility, easy parsing/cleanup)
Version number (forward thinking)
No duplicity (duplicitly → discrepency)
No “constant” variables (ossification → hardcoding)
Up-to-date specs (reflect reality)
Samples set (Theory isn’t enough)
Extensibility (your format will evolve in unknown ways)
Keep the spirit (don’t reuse formats for different intent without trivial distinction)
Perfect is the enemy of good (shortcuts will be taken to avoid over-complexity)

Article “On File Formats”

Rather recently, Jari Komppa wrote an article [articleKOMPPA] on the design of file formats (HackerNews discussion). He lists the following recommendations:

Does a file format exist for this yet?
Does it need to be human readable?
Chunk your binaries.
Allow partial parsing.
Version your formats.
Document your format.
Don’t include fields just in case.
Consider the target hardware.
Compression.
On filename extensions (i.e. consider four letters, but three letters are mostly allocated).

Paper “Little languages”

The paper by Jon Bentley [paperLITTLE] is by far the oldest. I do believe the notion of domain-specific languages was not sufficiently developed at that time and he coined the notion of little languages. The most prominent example from the paper is awk as little language. The idea is that domain-specific languages shall be developed and tools like awk help for the first step. He recites “an old rule of thumb” that “the first 10% of programming effort provide 90% of the functionality”. Only if the language evolves sufficiently, developers should consider using parsing tools like lex and yacc. Regarding the design of such languages, the paper lists the following recommendations:

Orthogonality: keep unrelated features unrelated.
Generality: use an operation for many purposes.
Parsimony: delete unneeded operations.
Completeness: can the language describe all objects of interest?
Similarity: make the language as suggestive as possible.
Extensibility: make sure the language can grow.
Openness: let the user ”escape” to use related tools.

Discussion

I want to start this discussion with a definition. Technically, there is no notion of a “binary file” and a “text file”. In practice, the distinction helps because we can immediately align our expectation whether we need a text editor (text file) or a specialized software (binary file) to handle the file. What is the distinction?

Text files versus binary files

Fundamentally, text means that some serialization format or character set exists to encode text. The famous ASCII (American Standard Code for Information Interchange) standard covers 128 characters and other sets are not considered ASCII these days. The other famous text encoding is Unicode which has taken up the tremendous effort to cover all used writing systems of the world in one encoding. To serialize Unicode scalars into actual bytes, different character sets can be chosen. UCS-2 and UTF-16 are deprecated, but UTF-32 can be still found. Unlike UTF-32, UTF-8 has the advantage of backwards-compatibility to ASCII and even ISO 8859-1. Furthermore it uses less space to serialize text in common languages like English. Since each character uses one up to 4 four bytes, the same text encoded in UTF-32 usually longer, but cannot be indexed by codepoint in constant time. To summarize: there is a long history of character sets including EBCDIC, Mac OS Roman, Windows-1252, Shift-JIS, Cork, and WTF-8. But with a current adoption above 98% within the World Wide Web, UTF-8 is the de-facto standard (backed by manifestos like UTF-8 Everywhere) and commonly picked as character set for new text file formats. And someone how thinks Unicode is unnecessary for English text, must be considered a bit naïve.

Going back to the question of binary versus text, the result is that some file formats declare that only byte sequences according to the declared character set are considered admissible files; namely text files. This is put into contrast to binary files where any sequence of bytes is admissible per default. Recognize that file formats like HTML require you to declare the character set, formats like PDF allow you to switch character encoding in the file as often as desired, and formats like HTTP headers are so complex that custom RFCs were written. One counterexample is the TOML file format which declares that “a TOML file must be a valid UTF-8 encoded Unicode document” and therefore is a genuine text file format.

What a text encoding contributes

One notorious problem with file format definitions is that people think that terms like “whitespace”, “hyphen”, or “line break” are universal, unambiguous names for characters. No, no, and no. Instead the notion of whitespace (more specifically Unicode scalars with Whitespace property), hyphen (more specifically U+002D - HYPHEN-MINUS), and line break (more specifically Mandatory break according to UAX#14) specifically come from text encodings like Unicode.

If you don’t specify the text encoding, I don’t know what those words mean. For Unicode encodings like UTF-8 or UTF-16, I clarified the meaning. If you use ASCII instead of Unicode, everyone understands that “whitespace” means 0x20, the space character as only representative of this group. If you mention hyphen, it is even more unambiguous than the Unicode case, because the less common U+00AD SOFT HYPHEN exists (among others). In the case of ASCII, “hyphen” means 0x2D unambiguously. But in ASCII there is no line break definition at all. Is 0x0A (“line feed”) a line break? Is 0x0D (“carriage return”) a line break? Both? Conventionally, 0x0A is a line break on Linux machines and the sequence 0x0A & 0x0D is a line break on Windows machines. But why is 0x0C (“page break”) not a line break? If you define a page break, you necessarily contribute a line break?! We are never going to know this, because ASCII does not define what a line break is.

If you actually decide to use a well-thought through standard like Unicode, you can answer difficult questions quickly in a standardized way as well: Is Unicode normalization semantically meaningful?

If you don’t specify the text encoding, I literally know nothing about the content. I don’t know what a whitespace character is. I don’t even know what a character is, because I don’t know how many bytes constitute a character.

Regarding syntax escaping

Some binary file formats and most text file formats have some requirement like “arbitrary user content follows”. In this setting, you really don’t know when the user content is finished. As a result, you are going to need some byte sequence which tells “user content finishes here” which is not interpreted as user content itself. You need to escape the “user content syntax”.

I wrote an article about syntax escaping some time ago, but the gist is this: syntax escaping can be avoided by a length specifier which is only practical for binary files (never let a user count bytes or Unicode codepoints). Otherwise, you can decide to declare one byte sequence to be “escaping”. If you repeat this byte sequence, it regains its original meaning, but otherwise some escaping sequence is started which might signify something like “user content stops here”.

The worst thing, you can do is to ignore the problem. If you allow arbitrary user content, but don’t declare an escaping mechanism, you either open up yourself to ambiguities or violate the requirement “arbitrary user content”. My personal opinion is that XML’s escaping mechanism is simple and extensible compared to other approaches.

Regarding file extensions

File extensions give the operating system a clue which application might be capable of interpreting a file. For historic reasons, they tend to be short (2 to 4) sequences of Latin characters. They are an incomplete concept leading to unintended collisions. For example all kinds of markup syntaxes are declared as .md file these days. Historically .txt used to be full of collisions. But it still makes sense to align all users upon one file extension.

What I would like to stress here as well is the MIME type. It is equally helpful to align all users upon one MIME type. The x- prefix opens up MIME types to custom standards. So text/x-foobar would be a valid choice.

Regarding magic numbers

One might think that magic numbers are unnecessary boilerplate. If the specified structure is unique for your file format anyhow, why should a magic number be necessary? The answer is simple: Not all tools want to look at the entire document structure to determine whether a file follows a certain file format. If it only has to read some leading bytes (namely the so-called magic number), they are much quicker to determine whether the file is interesting ^[1]

Regarding version numbers

The simple argument pro version numbers is “implementors can easily dispatch interpretation”. If your document follows the specification 1.0, the source code for 1.0 interprets your file. If your document follows the specification 2.0, the source code for 2.0 interprets your file. This way you can easily introduce backwards-incompatible version changes.

Design requires re-re-re-iteration

File format design is design. Every design needs iteration for perfection and consistency. Please finish the draft version in a straight-forwards, usecase-centered manner. But be open to improve upon your design in subsequent versions. Iterate and iterate and iterate. And re-iterate again. And ask your target audience about their opinion. Then you mastered the art.

About the general approach to design

One thing, I would like to point out which comes from programming language design is that design should go from specific cases to generality. What is meant that one can specify very extensible elements in your syntax. But you should define those elements for their specific cases and disallow others. In subsequent versions, you might understand which other cases exist and which cases make sense. Under these circumstances, you might open up that element for more (or more general) cases.

Let me illustrate this with a trivial example: You might have 10 specific cases to distinguish, but you have to use one byte as discriminant. Therefore 256 cases can be distinguished, but you only need 10 cases. Now the general approach to design can be done in the following wrong way:

Specify the 10 cases
Declare 246 cases to be “implementor-defined”

Instead the following correct way can be taken:

Specify the 10 cases
Disallow 246 cases

The point of the latter approach is not that “implementor-defined is always stupid”. If you are certain please specify (for example) 10 cases for the desired cases, 20 cases for “implementor-defined” usecases, but be aware that disallowing some values opens up extensibility in future versions. You should restrict your design tightly. Once you gained experience and feedback, you should open up to other cases. Being restricted in the beginning enables the necessary extensibility for later.

A generic approach for defining binary file formats

There is a simple design which can model any binary data model unambiguously. It is called TLV (Type-Length-Value).

The file format has to follow the general recommendations first. Introduce a magic number. Introduce a version number. Put your metadata in a header. And then let us write down the data in the body of the file.

The body is a sequence of entries. Every entry consists of a type, a length, and a value.

A type is a discriminator (thus, of fixed width) telling what kind of data you are supposed to expect in the value. For example, byte 0x13 might identify value to an unsigned integer in big endian. 0x13 is one instance of this type.
A length specifies how many bytes the value constitutes of. It needs to be of fixed width as well and common choices include 16 or 32 bits. For example, bytes 0x00 0x04 might identify length 4 and thus our example value is expected to be an unsigned integer in big endian of 4 bytes.
The value is a sequence of bytes. Because of the previous values, you know exactly how many bytes you are supposed to read to understand the value. Furthermore, we attached some semantics through the type discriminator. The specification is now supposed to specify how to interpret those bytes of this type.

You have to encode a boolean value? Designate a type, length one should always be required, and specify which two values are admissible. Done.
You have to encode Unicode text? Designate a type, the length can be adjusted to the actual byte length, and specify which encoding you require in the value. Done.

This design is very simple and very generic. Does it solve all problems and do all binary files become instances of the TLV design? No. TLV is generic and a very good guideline. However, it applies only to binary files (no human wants to think through a three-step process all the time and particularly count bytes) and binary files mainly exist because they optimize some requirements over text files. Binary files might optimize parsing performance or space usage. As a result, designers start to skip fields. For example, if an entry of type 0x42 is always preceeded by an entry of type 0x41, space-optimizing designers might claim that the type byte 0x42 must be left out. Indeed, position-defined entries do not need a type if it can be derived from the position index. But in this very moment, the TLV design principle is violated and TLV remains only as “general rule of thumb”.

Everyone should be familiar with TLV and follow it for binary files, if ambiguity-freedom cannot be guaranteed for the entire design. A similar design where values usually represent chunks is the Interchange File Format.

Target hardware and endianness

When it comes to binary files, endianness is necessary to be specified. Usually deciding upon this value directly leads to the question of target hardware platforms. Endianness can be defined arbitrarily. Big endian or little endian? It is trivial. Just pick one. But the only meaningful way to pick the best value is thinking about the target platform. Does it target Intel machines? Then little endian makes more sense. Are humans going to look at the values from time to time? Big endian might be more convenient, but less optimized for desktop computer hardware.

You should know your target domain, your target audience, and common hardware platforms. But don’t optimize prematurely.

If you start cramming all data into bitvectors to save a few bytes to optimize space, you neglect that machines are optimized to operate on bytes and cache lines. Extracting individual bits is a time-consuming operation. You might be better off adding a few unused bits exchanging space for time.

In the end, benchmarking your prototype parsing implementation reveals the actually interesting parts to optimize. This becomes especially important, if you plan to apply compression on parts of your data.

Syntax and semantics

My final point would be that syntax and semantics are two different concepts. You need to be aware of it. You may be able to use the syntax of an existing standard and define custom semantics on top of it. One example would be XML (syntax) and InkML (semantics). You may also define a new syntax like Simple Outline XML for existing semantics like XML.

I gave examples for text files here, but this also applies to binary files. The only problem is that binary files usually have a very specific data model which differs to other formats. But generic binary file standards include ASN.1, edn, and postcard (introductory talk on youtube).

If you are able to split syntax and semantics, you end up in one of two scenarios:

you can use an established, proven-in-practice standard for one of two components. The necessary tools do not need to be written again.
you get the possibility to remove one component and exchange it for something else, if you recognize a mistake. Programming languages like Dylan just removed their LISP-style syntax and introduced something new.

Not too bad, right?

Decision list

Finally, I want to contribute a decision list where items to consider are listed:

Are you sure the effort of defining a new file format is worth it? If no, stop. If yes, proceed.
Split syntax and semantics. Can you reuse syntax (e.g. XML, S-expressions, JSON, YAML, …) or semantics (eg. JSON, YAML, ASN.1, postcard, …) of existing ones?
The following holds true for text and binary files:
- No duplicity (duplicitly → discrepency)
- Add a version number. No exceptions. Consider versioning schemes to communicate users which expectations regarding compatibility / upgrade necessity are given.
- Ask for feedback regarding syntax (text files) and data model (text files and binary files)
- Avoid unnecessary generality. It is easier to permit features later on than standardizing elements later when they are in use already
- Ask a domain expert for feedback and iterate.
- Increment the version number and release.
- Which interfaces to embed content for other file formats do you provide?
- Revise which elements provide modularity and extensibility in your file format.
Is it going to be a text file?
- Declare the character set (UTF-8 is recommended)
- Unicode is utilized by the U-notation: U+002C COMMA. Other character sets commonly use plain hexadecimal notation like 0x2C. If you refer to a character, get used to this notation and use it exclusively.
- Depending on your character set, you may use words like ‘whitespace’ now to describe your file format
- Do you want to base your syntax on existing concepts and notation? Remember that any existing technology exists, because it has some valuable benefits. But you are designing something new, because it does satisfy your requirements. Comprehend the benefits, integrate them into your design, and reiterate multiple times to achieve consistency.
- Define the syntax escaping mechanism if arbitrary user content is allowed
- Discuss punctuation versus keywords. Punctuation is a small set of characters and thus brief. But only programmers are used to use them in various contexts. Keywords are longer and extensible, but you have to discuss questions like casing and singular versus plural (depending on the writing system and language).
- Discuss whether you want to include comments (elements which carry no semantics, but provide an opportunity for documentation to the author)
- Discuss whether you want to allow trailing separators (e.g. ["item1", "item2",] if comma is your separator)
Is it going to be a binary file?
- Add a magic number at offset zero
- Declare: are multi-byte values encoded in little endian or big endian?
- Declare and illustrate the big picture structure of your file format (e.g. header/body/footer). It is easy to get lost in details (or bore the hell out of the reader) when describing binary file formats.
- Enable parsers to skip structural parts of your file format (e.g. the entire body, because its length is declared)
- Follow the TLV design. Declare the semantics of values in the file.
- If you don’t follow the TLV design, define the escaping mechanism (length declaration is recommended)
Publication:
- Provide example files.
- Provide a specification document. Specify where people can direct their feedback to. Specify the version this document documents.
- Develop tools to read, write, analyze, and fix files in this format.
- Suggest a file extension for files. Suggest a MIME type for files.

Conclusion

File format definition is a difficult art. And hopefully I summarized some guidelines for you. If you succeed, people are going to enjoy writing parsers for it. If you fail, your file format is going to suffer from fragmentation and limited adoption. Good luck!

1. For UNIX users, a simple scenario is grep which has to decide whether a file is binary (to be ignored) or text (to be searched in).

Revjuo: Aŭstria Esperanto-Kongreso 2025

2025-05-18T10:00:00+02:00

Enkonduko

Ekde la 2-a ĝis la 4-a de Maio 2025, la unua Aŭstria Esperanto Kongreso okazis ĉe Graz. Graz estas la dua plej granda urbo de Aŭstrio kaj mia eksloĝurbo. Lastatempe Ewald iĝis estro de la Aŭstria Esperanto klub (post sia tempo kiel Stiria estro) kaj li revis pri ĉi tiu evento. Afablaj Esperantistoj de pluraj landoj helpis realigi ĝin. Persone, mi ne kontribuis ĉar mi translokiĝis dum la organiztempo kaj mi nur prelegis.

Evento

Dum tri tagoj, pluraj prelegoj okazis. Inter la prelegoj estis sufiĉe da tempo ĉar entute nur 14 prelegoj okazis. Nan ankaŭ kontribuis du sesionojn kun ekzercado de ĉigongo. Vendrede kaj sabate vespere okazis muzikeventon. Ĝenerale la prelegoj temis pri kulturoj kaj lingvoj. La kongresejo estis la partidomon de la urbestrino.

Mi mencios nur kelkajn prelegojn:

En “Sopiro je Senco”, Norina kaj Philipp klarigis kiel ili retrovas logoterapion en la Esperanto komunumo.
En “Unuiĝintaj Nacioj, kiel la reprezentantoj de UEA laboras tie”, Mireille raportis ŝian sperton kun reprezento de Esperanto en la UN.
La prelego “Gravaj sciencistoj de Graz kaj Stirio” estis mojoza ĉar mi certas ke ĉiu aŭskultinto lernis ion pri sciencistoj de Graz. Ŝi prezentis biografion de sciencistoj kaj iliaj gravaj rezultoj.
Mi lernis ke la preleganto Viŝnja nur lernis Esperanton por unu jaro nun kaj ŝi jam perfekte prezentis la temon “La vera genio: Spomenka Štimec”. Estis tre interesa prelego ĉar mi lernis pri la literaturo de multaj aŭtoroj kaj ŝi ankaŭ menciis la rilaton inter Esperanto monumento ĉe Graz kaj Zagreb. Mi tute ne sciis; mi pardonpetas.
En la prelego “Esperanto kaj Rumantsch Grischun”, mi lernis pri la Rumantsch Grischun lingvo parolante en Svislando. Mi tute ne sciis pri la lingvo.
Uli komparis diverslingvajn tradukojn de “La eta princo” de Antoine de Saint-Exupéry Statistika en sia prelego. Mi ne diras ke ĉiu statistiko estis interesa (ofte nur estis rezulto de la skribsistemo) sed li bone klopodis kolekti statistikajn rezultojn kaj plaĉis al mi.
Amuze en la tria prelego, Mireille prezentis fotojn de diversaj landoj montre banoj, duŝoj, vestolavadoj, kaj necesejoj.

Mia propra prelego funkciis akceptable. Mi prelegis pri alireblo en cifereca kompostado. Mi volis paroli pri etiketataj PDF dosieroj, sed mi eksciis ke la temo estus tro teknika. Do mi plilongigis la enkondukon pri ciferecaj programoj. Antaŭ la prelego mi komprenis ke la kvar programoj ne taŭgas por 30 minutoj. Mi forigis miajn lumbildoj de du programoj kaj ekprezentis. Mi tre ŝatis miajn finajn lumbildojn sed ankoraŭ trovis akuzativan eraron dum la prelego. Finfine mi perfekte trafis la 30 minutojn sed mia parolo estis malbone. Mi konis la vortojn sed mi ne sufiĉe flue parolis pri la temo. Klopodu, Luko!

Mi ne parolis pri la diversaj interagoj de homoj. Mi parolis kun diversaj afablaj homoj. Tre plaĉis al mi ke diversaj Esperantistoj organizis libroservon, muzikon, sesion kun korpa ekzercado, kaj helpis realigi la eventon.

Konkludo

Estis malgranda evento de proksime 70 partoprenintoj. La organizo ne estis tiel bone, sed bone taŭgis por la partoprenaro. Dankon por la kontribuoj, Esperantistoj!

Review: RustWeek 2025

2025-05-17T09:00:00+02:00

Motivation

My company was generous and sent me to RustWeek 2025 organized by RustNL in Utrecht, Netherlands. RustWeek is a conference for rust developers and especially many core contributors can be found there. So this was a nice opportunity to get some educational input for the programming language. I visited the conference together with a work colleague.

Travelling

We left Vienna by night train towards Amersfoort on Sunday. Arriving late on Monday, we went to our hotel. Unpacking our stuff, we got our conference badges, headed for lunch, and started our day of work very late. Getting the first impressions of Utrecht, I concluded this is a very nice place for me. The people are nice and expectedly I love the bicycling infrastructure. The canal Merwedekanaal benoorden de Lek gave me sea-style vibes (sure, as Austrian my understanding of seas is very limited).

Figure 1. The Netherlands are famous for their windmills

Figure 2. A common street situation in Utrecht (pavement made of bricks and many green trees)

Conference venue

The conference took place at Kinepolis Jaarbeurs (cinema of the conference center). Mara turned this into a wonderful experience by subtly replacing the cinema promotional posters by rust-tailored ones. Splendid!

" instead" width="100%">

Figure 3. Subtile rust cinema posters

Even though the conference essentially took place the entire week, we only had tickets for the two conference days with talks.

Figure 4. My conference badge

Conference

On 2025-05-13, the talks started. The intro was very popular. It showed a giant ferris orbitting the earth with the final title “RustWeek” in the style of the Universal Studios intro. They mentioned it took them three days to create it including one day for rendering. Its duration is less than 30 seconds, if I remember correctly.

Figure 5. Rendered intro

There were some very interesting talks for me and I am not finished with watching all recordings yet. But I am going to lose some words about the following talks:

Alex Crichton on the question “10 Years of Rust: Why?” gave a really suitable keynote for the event. He gave a rationale for the steps taken and tried to analyze what was necessary to reach 10 years of developing the language.
Due to my personal interest in digital typesetting, I am very grateful to Raph Levien and his work. Raph spoke about “Faster, easier 2D vector rendering” and spoke about one algorithm to improve rendering of fonts. He tried to reason about the use of GPUs for font rendering and explained which rust crates come into play.
I once wrote a blog article about icu4x 1.0 when I started using it. I am using icu4x in the opstr project. Shane spoke about Beyond ICU4X 2.0 and future goals. I promised him to provide a review of icu4x 2.0 as well. Stay tuned at this blog!

There are so many more talks to talk about. And simultaneously it is important to talk to the people. So many nice people with aspirations and dreams for the programming language. And I did not mention how much I enjoyed to resolve some rust issues in my head during the conference. A lot of kudos to my work colleague who is following core development efforts and helped me out several times. By the way, I have identified 4 Austrians in total at the conference.

On the day after the talks, rust 1.87 was released live from Utrecht. As I mentioned, we only participated on the two days and I left by night train towards Linz on Thursday evening.

Figure 6. Conference venue hall

Figure 7. Final photo by the RustWeek organizers

Conclusion

I absolutely enjoyed Utrecht. I absolutely enjoyed the conference. I was astonished how well-organized the conference was by the RustNL community. So I want to thank them a lot and hope for a wonderful conference for them in 2026 as well!

Aikidovereine in Graz und Salzburg

2025-04-13T00:00:01+02:00

Motivation

Ich bin neulich von Graz nach Salzburg gesiedelt. Ich wollte meine Sammlung an Aikidovereinen teilen, die Trainingseinheiten in Graz & Salzburg anbieten. Ich liste nur Trainingseinheiten, die …

auf der Webseite
für einen Standort der jeweiligen Stadt ein Training anbieten
und die sich (aus reinem Eigeninteresse) an Erwachsene richten.

Graz

Verein	Aktuelle Leitung	Einheiten
Aikido PRO	Valentin Lasnik	Di 16:30, Fr 18:00
Aikikai Graz	Peter Poltsch	Di 18:30, Mi 17:00, Do 20:00, Fr 18:00
ASKÖ Aikido Graz	Günther Steger	Di 19:00, Do 19:30, Fr 17:00
Aikido Union Graz	Frank Koren	Mo 19:00, Mi 18:30
Sobukan Union Graz	Andreas Schoch	Do 19:00

Verein

Aktuelle Leitung

Einheiten

Aikido PRO

Valentin Lasnik

Di 16:30, Fr 18:00

Aikikai Graz

Peter Poltsch

Di 18:30, Mi 17:00, Do 20:00, Fr 18:00

ASKÖ Aikido Graz

Günther Steger

Di 19:00, Do 19:30, Fr 17:00

Aikido Union Graz

Frank Koren

Mo 19:00, Mi 18:30

Sobukan Union Graz

Andreas Schoch

Do 19:00

Salzburg

Verein	Aktuelle Leitung	Einheiten
Ko Jun Dojo - Aikido Union Salzburg	Bruno Wintersteller	Mo 19:30, Do 19:00, Sa 10:00
Enshiro Dojo Salzburg	Harald Paßrucker	Mo 19:30, (Fr 20:00)
Tanden Dojo Salzburg	Alexander Ermakov	Mo 20:15, Mi 20:15
Kenbukai Salzburg	Ute Schwarzmayr	Di 19:00, Fr 08:00

Verein

Aktuelle Leitung

Einheiten

Ko Jun Dojo - Aikido Union Salzburg

Bruno Wintersteller

Mo 19:30, Do 19:00, Sa 10:00

Enshiro Dojo Salzburg

Harald Paßrucker

Mo 19:30, (Fr 20:00)

Tanden Dojo Salzburg

Alexander Ermakov

Mo 20:15, Mi 20:15

Kenbukai Salzburg

Ute Schwarzmayr

Di 19:00, Fr 08:00

Zusammenfassung

Wenn ich externen Leuten von Aikidō berichte und es als Nischensport oder -kunst beschreibe, sind sie immer überrascht wieviele Vereine es trotzdem gibt. Die Liste oben habe ich nur durch Internetrecherche gefunden und vielleicht habe ich sogar welche übersehen. In diesem Sinne ist Aikidō wirklich lebhaft.

syntok release

2024-12-19T00:01:00+01:00

Motivation

Assume you have a program to understand desired markup languages. Assume you have a typesetting engine. Assume you have components to generate desired output formats like PDF, EPUB, HTML5, and so on. By combining these tools, you get a program for your digital typesetting needs, right?

No, you will soon recognize that syntax highlighting is a crucial element of such systems. Programmers want their generated documents to feature syntax highlighting. Indeed, source code is often terrible to read without syntax highlighting. Can one easily distinguish types from identifiers? Can one identify substructures if the syntax does not require an off-side rule? One could certainly pull in one of the many syntax highlighting library efforts, but isn’t this overkill and a too strong dependency?

As a result, I thought about a building block. A serialization format which encodes how syntax shall be serialized.

Syntok

Let me introduce syntok: serialized tokenization of syntax.

Consider the following example C++ program:

#include 

int main() {
	std::cout << "hello " << ([](void){ return "world!"; })() << std::endl;
	return 0;
}

In my weblog, this source code appears colorized (also called “syntax highlighted”) to make it more readable. This is possible because a ruby binding to pygments is utilized to generate a colorized HTML version from the code snippet. But wait… isn’t this annoying? I need a ruby binding to run some python software to generate HTML output?! Or in the case of tree-sitter, I need C and JavaScript to generate HTML or XML output. Would it not be nice to just take a file which encodes the individual tokens and the software can decide the remaining colorization parts? And if pygments and tree-sitter can emit these tokens, we can use them interchangably.

Thus, instead of one tool covering the entire pipeline of reading some syntax and generating some specific output format, I want to split the pipeline up. One tool reads syntax and generates a syntok file. One tool reads the syntok file and generates the output format.

For the example above, the following file can be the corresponding syntok file:


 xmlns="https://spec.typho.org/syntok/1.0/xml-schema">
   category="preprocessor-instruction" start="0" end="8">#include 
   category="system-library-ref" start="9" end="18"><iostream>
   category="whitespace" start="19" end="20">


   category="type" start="21" end="23">int
   category="whitespace" start="24" end="24"> 
   category="identifier" start="25" end="28">main
   category="parameter-list" start="29" end="30">()
   category="operator" start="31" end="34"> {
        
   category="namespace" start="35" end="37">std
   category="operator" start="38" end="39">::
   category="identifier" start="40" end="43">cout
   category="operator" start="44" end="48"> << "
   category="string" start="49" end="54">hello 
   category="operator" start="55" end="63">" << ([](
   category="type" start="64" end="67">void
   category="operator" start="68" end="70">){
   category="keyword" start="71" end="77"> return 
   category="string" start="78" end="85">"world!"
   category="operator" start="86" end="95">; })() << 
   category="namespace" start="96" end="98">std
   category="operator" start="99" end="100">::
   category="identifier" start="101" end="104">endl
   category="operator" start="105" end="105">;
   category="keyword" start="106" end="114">
        return 
   category="integer" start="115" end="155">0
   category="operator" start="116" end="118">;
}

syntok is an XML file (file extension .synt) which has a root element syntok and contains item elements for the individual tokens. start and end document the byte offsets and crucially category associates a category to this token. Now, I would like to point out two obvious points:

The set of categories (“data model”) can be selected by the tokenizer itself. For a short period of time in the beginning, I assumed I can contribute a data model for tokenization. For example, remember that many syntaxes do not have a “namespace” category but introduce arbitrary other synactic elements (e.g. python). It is impossible to contribute such a generic taxonomy. Instead I have to rely upon a folksonomy.
In general, the category-to-syntax-highlighting-color association needs to be contributed externally. But of course, one can trivially just hash the category name and pick a color based on the hash (this is what I did in my example programs … and certainly it does not always lead to beautiful colorization!).
The quality of tokenization is allowed to vary. What about the final operator-categorized item? Why is ';' and '}' not split up with a whitespace-categorized item? Simply put, because for 99% of applications, whitespace won’t have a special style (e.g. different background color). So the given quality suffices. And indeed, a better tokenizer hopefully splits them up to satisfy even more applications.

One of the immediate advantages is that a tool can generate the syntok file, but now the user can intervene and adjust the tokenization (or add additional markup) before it gets represented. This solves a common difficulty I experienced from the LaTeχ package world. If my source code has slight adjustments (often happens with ASM, happened when python highlighting did not yet have python3 support, SQL versus PL/SQL, …), the software will irrevocably represent erroneous syntax.

One of the requirements is that the entire file is tokenized. So the start and end attribute provide a partition^[1]. Recognize that the syntax is linear and flat. It does not represent hierarchical structure often found in markup languages and programming languages.

The specification

The specification document was written in AsciiDoc and is readable in this git repository:

syntok

Furthermore, it comes with a bunch of tools, I used while using the standard in production. Most importantly:

A python script taking a .synt file to generate colorized CLI output
A JavaScript-powered webpage taking a .synt file to generate colorized output on an HTML page

Furthermore:

An XSD file to verify some properties of a syntok file
A python script to verify remaining properties of a syntok file
A python script to generate syntok template by Unicode categories
A python script taking a tree-sitter dump and the original file to generate the syntok file

F.A.Q.

Why XML?: I think JSON and XML have the broadest support as data serialization formats to be written and read. YAML never gained sufficient traction (without reciting the reasons here). I lookad at XML and JSON and recognized that writing XML is much simpler because of simple escaping rules. Recognize that the user-provided content can be arbitrary (even binary) and in these cases, I would not dare to write my own JSON writer in C or assembly, but I would to do so for XML (in fact, I did back at university).
Why didn’t you allow both formats?: There was one unpublished version specifying JSON as well as XML serialization. In the end, I felt like this fragments the topic unnecessarily and makes it difficult for tooling providers.
Why document start/end?: Since people often ignore that XML is whitespace-sensitive, I think it can easily happen that someone introduces content accidentally. Then the original content gets lost. When someone wants to debug this situation, having the start/end attributes, helps a lot. I admit, it makes it more difficult to adjust syntok file manually.

Conclusion

I hope this standard contributes to ease syntax highlighting in digital typesetting. Of course, it needs proper support by syntax highlighting libraries or, even better, by parser authors.

Let syntax be tokenizable!

1. In the mathematical sense. Colloquially this would be named “complete”.