Iro

The simplest kind of context item is the pattern matcher.

The pattern matcher simply matches from the current cursor position (to the end of the line), and if matched, assigns one or more styles to the matched text. The pattern matcher does not alter the context stack in any way.

Multiple capture styles may captured in a single regular expression, but never ever overlap groups or omit characters from a match. See : non-overlapping groups.

Please see the algorithm section for how these match patterns are utlized in relation to the current line cursor.

When Iro is performing a debug render (as opposed to translating into alternative grammar formats), the match that closest to the line cursor is the match that will be selected. Upon matching, if the first match is not at offset 0 from the current line and column cursor position, non matched characters are emitted to the renderer without a style.

The matched text alongside corresponding styles will then be sent to a renderer, the line cursor will be moved forward, and the remainder of the line (or the start of the next line if zero characters remain on line) will be tested against context matchers in order in the same context again.

If the regular expression cannot be matched, then the next command is evaluated. If all regular expressions do not match then the remainder of the line is emitted with no associated style and the line cursor movest to the beginning of the next line.

A pop operation pops the current context off the stack. Iro maintains a stack of contexts, typically traversed through pushes and pops. It is important to make sure that pushes and pops align up carefully.

A matched pop operation will consume the matched text, emit the matched style or styles, pop the current context from the stack. And start scanning in the new context (if the context changed).

An EOL pop is a pop (remove the current context from the stack) that occurs when the end of the current line is reached. As such, there are no required attributes.

Inline push and pop are useful for lexing langauges that are stateful. Most languages enter into different lexical states. The most commmon type of lexical state might be quoted text.

An inline push pushes its own internal inline context on a stack of contexts maintained by Iro (or any of the supported exporters).

There are two modes in which an inline push operates:

There follows two examples of an inline push, for handling text within quotes.

Push is the same type of behaviour as : inline_push except instead of entering an inline context, it enters an explicit new context.

In textmate compatibility mode (the default mode of operation), only one context may be pushed onto the stack at one time. If this compatibility mode is disabled, then it is permitted to push more than one context onto the stack upon a match occurring.

The 'include' context item allows the contents of other contexts to be referenced and imported as if they were part of the current context.

That is, if we reference a context with ten context items, those ten context items will be copied and pasted into the current context.

Recursive includes loops are flagged by the compiler.

In Iro, any attribute that starts with two underscores is deemed to be a constant.

Constants may be referred to using the $${} syntax to reference constant value attributes. Any number of constants may be utilized as long as they start with two underscore characters.

Constants can be useful for adhering to (DRY) principles.

It is possible to push the same context on the stack multiple times in a row. This technique can be used to highlight unmatched brackets.

For example, consider the following grammar:

Sample code to implement bracket matching in grammar.

The result of the above grammar.

It is important to note that the prior JSON sample is not attempting to enforce the rules of the langauge through highlighting, beyond matched bracket highlighting (which is trivial to achieve).

By way of example, the following snippet would be syntax highlighted without error, even though it is invalid JSON:

The JSON grammar described earlier is an approximation and not intended to model the nuances of what is and is not possible in JSON. It is common practise for syntax highlighters to have undefined behaviour when faced with improper grammar.

By all means, build your model as good as you possibly can, but be aware. Perfect parsing of a language is an order of magnitude may be much more complex, for very little benefit. It is the job of the parser to highlight errors in the source, not the job of the syntax highlighter.

The snippet shown above shows an invalid regular expression. The '\s*' part of the regular expression does not belong to a match group (a bracketed region). This means that when emitting tokens that Iro (and other engines) would not emit characters that match this part of the expression meaning that the ungrouped characters would be lost.

The compiler will detect non matched characters.

The snippet shown above shows that three capture groups are defined in the 'match' regular expression(correctly), but only two styles are defined.

The compiler will flag this.

In the snippet shown below, 4 groups are defined. 3 outer groups, and one inner group. The problem here is that 'ou' is grouped twice, therefore will be emitted twice.

Although the compiler does perform a check that the number of match groups in the regular expression corresponds to the number of supplied styles, the compiler does not detect overlapping matches in regular expressions, so please be aware of the overlapping match scenario.

See Use of non-capture groups for information on how to resolve.

A common requirement is to use an inline push (or push) to deal with quoted text.

When dealing with literal text, there is invariable some kind of escape character handing that must occur. When dealing with escape characters, it is important to be aware of greedy regular expression that may steal the escape character.

In the below example, we enter into a literal text state upon encountering a " character. We flag that upon encountering another " character we should leave this lexcical state (aka context).

But we also wish to be able to place quotes inside the quoted block without it being a match that triggers a pop. In most languages, the backslash \ character is employed as a prefix to escape the quote so that the parser (and the syntax highlighter) knows to treat the \= pair as an escaped pair of characters.

A naive way to implement this logic is to define a rule for handling the \= pair, and define a rule that handles everything else.

The problem here is that the ([^"]+) expression will consume trailing backslashes. Meaning that this text "one\"" will be tokenized as follows:

Failed.

So, let’s stop the bottom pattern from picking up the backslash character:

Passed.

Seems good, but what if we want to be able to place a literal backslash in the text too. We’d need to escape the backslash with a backslash. Let’s try "o\\ne\""

Failed.

So this time, let’s support 3 items:

Passed.

Passed.

The following scope report corresponds to the JSON Sample shown earlier except we deleted the textmate scope from the '.escape_char' style.

Here is the modified style:

Here is the resultant report:

The following scope report corresponds to the JSON Sample shown earlier except we deleted the textmate scope from the '.escape_char' and 'punctuation' styles.

Here are the modified styles:

Here is the resultant report:

The following scope report corresponds to the JSON Sample shown earlier except we modified 'punctuation' style to have the same textmate scope as the '.escape_char' style.

Here are the modified styles:

Here is the resultant report:

First we should identify the initial state and the items we would like to syntax highlight in in the initial state.

In this first example, there is only one state (possibly two if we view quoted text as its own state).

Here are the things we wish to syntax highlight in the initial state. This does not necessarily need to be a complete list. This process is iterative.

So, let’s start with:

We start with a small amount of boilerplate, which can be copied and pasted from this section of the document.

It incorporates the 3 significant sections - header, styles, contexts.

From the 3 categories, plan to create 3 styles. Remember, style names start with period/dot (.) character by convention. This was a questionable design decision by yours truly, and it keeps me awake at night.

Iro currently only supports the dark theme for debugging rendering (render against a dark backdrop).

Now we create an inline push rule to handle quoted text. We enter an inline lexical state upon encountering the " character. We only pop out of the state when we encounter another " character. All text handled before the closing " character is assigned the default style of ".quoted_text".

This is a fairly simple rule. It looks for // and then consumes everthing after this. Remember, we are using a line based matcher, so .* only takes us to the end of the line.

The debug render of the document contains a lot of useful debugging information. Every single piece of text and whitespace has a hiver which displays the current style and stack for the current location.

In this below diagram, it displays which rules have been executed in which order, using line numbers; very useful in larger models.

In order to make stack traces easier to read, match items are able to assign unique identifiers to context items. Contexts already have ids.

UIDs must be unique across all declared UIDs in the .iro file and must start with 'uid_'. This is to make it clear when a stack trace element is a context, or when it is a context (as contexts are not permitted to start with the 'uid_' prefix).

Adding in a textmate scope and a pygments scope is required to obtain the greenlight (at present). Ace inherits from textmate, Rouge inherits from Pygments.

And here is the report:

We assign textmate and pyments scopes to the final two styles. Now we have full scope coverage across all supported exporters.

And here is the report:

If running from the command line, the created syntax highlighting definition files are available in the 'iro-gen' folder (they are generated at the same time as debug builds).

If running from the web UI, select the exporter specific tab to see the appropriate exporters. The download button may be used to download all generated exporters in a zipfile.

It is not in the scope of this document to describe how to import these files into various editors/IDEs/Frameworks.

When the scanner skips characters (like the ' x' and ' ' and ' documen') in the walkthrough. These characters are emitted with no style associated with them.

All matches emit events, and these events are handled by the highlighter.

Iro renders text for the purposes of debugging, but its algorithm simulates the textmate and Ace editor algorithm (derived from observation).

Here are a list of common Textmate scopes. This list is advisory. Other scopes may be supplied. The Textmate website contains some additional information on how to assign these scopes.

Ace editor generally mirrors the scopes used by Textmate in order to be semi-compatible using an adapted version of the Textmate lexing algorithm.

If assigning a Textmate scope to a style, then the Ace scope inherits the same scope. If an Ace scope is explicitly set, then this overrides the inheritance.

Here are the pygments/rouge scopes that are supported by Iro. This list is advisory. Other scopes may be supplied

Colors within styles can use either a hex value (in the #000-#fff format) or a color alias, as described in the following table.

Do not worry about "hardcoding" colors into your definition file. These colors are only to be used for debugging purposes as when emitting grammar definitions for external syntax highlighters, colors defined within a style will be completely ignored.

A blank document will be parsed into containing a single root object with no contents. Every valid item parsed within a document is added to the root document. Within the root object and within any object, an object can contain attributes, objects and collections. All items within an object must be identified.

In Rion, all objects must have a type but id is optional.

In this snippet, we are defining an unidentified object of type 'pattern'. Objects without type are usually order sensitive (order is maintained).

Basic attributes are simple keys and values.

Rion supports a native Regular Expression syntax such that no escaping is performed.

Some attributes may involve multiple values.

Collections are containers for objects. If the order of objects in a sequence is important then that is a reason to use a collection.

In the above example, the collection is 'contexts [] { …​ }' and it contains a single typed and unidentified object (of type patten).