Parnassus

Part 2 of a series on how to write an IDE plugin that can paint on the IDE code editor line-by-line along with the code.

Introduction

The first part of this series examined an area not exposed by the Open Tools API (Delphi’s plugin API): integrating with the code editor. Yet several plugins, such as CnPack, Castalia, and two of my own soon-to-be-released plugins (Bookmarks and a Mystery Plugin), integrate into the code editor painting along with the code. How do they do it? The answer is by hooking a number of IDE methods at runtime, and the first article showed how to do this with one method, TCustomEditControl.PaintLine, and ended painting some text on the top left of the editor.

This article covers several topics:

• Integrating with line numbers, folded code, and paint coordinates for a line, in order to know exactly what is onscreen in the code editor and where it is, and accurately paint your own additions to it
• How to coexist with other plugins that also paint in the code editor

This articles assumes you have read the first; are familiar with the concept of code hooking; and are familiar with function pointers to OO methods and can convert between OO and non-OO prototypes of the same method (ie with an explicit Self parameter you pass in order to call an OO method, procedural-style.)

Code editor integration

Once you have a plugin that can paint on the code editor – see Part 1 – you need a number of other things:

• To know which file the code editor is displaying
• To know the line number that is being painted in your hooked paint event
• To account for folded code in the above, ie to know which lines are visible onscreen and which aren’t, and if a line is visible as normal, is hidden, or is a folded line (has the + symbol indicating it can be expanded)
• Miscellaneous things, like getting a point onscreen in pixel coordinates for a line / column pair

Which file is being displayed?

The TCustomEditControl.PaintLine method has a number of parameters, including Self (the edit control) and others. Yet none of these directly indicate which file is in the editor being painted.

To do this we need to hook another TCustomEditControl method: SetEditView.

A brief overview of the edit control’s architecture: the IDE has (usually) only one edit control visible onscreen. It is docked in the center of the IDE. Others occasionally appear – you can open a file in a second window and in the Options dialog one can be hosted – but that’s outside the scope of normal IDE usage for this article. The tabs at the top of the editor change the file that’s being displayed. The tabs do not change which TCustomEditControl is visible onscreen. There is only one, and instead the contents of what it displays are changed, via a view.

A view represents the file, its contents, and metadata about it required for display.  Much of this same information is available in the OTAPI via OTAEditView, which represents a view, and which we’ll get to in a moment.

In order to track which file is being displayed onscreen, we need to hook SetEditView. The code will then remember which view was the last set for any specific edit control. Then, when painting a control, you can look up information about the view, such as the filename of what’s being displayed inside the view.

I’ll assume you know how to hook a method, given the mangled method name and a method prototype. (See the first article for details.) Please note that at the end of this article I discuss the best hooking method to use – it’s not the one used in the first article – and some caveats about how to use it. Please make sure you read that section.

SetEditView’s mangled name and decrypted prototype is:

“Self” is the edit control the method is being called on; EditView is the IDE’s internal view.

However, the IDE’s internal view is an opaque object; I don’t know what it contains or how to use it. So, how do we get useful information like the filename? The answer is by hooking another method, this time on the edit view itself – the EditView parameter above – and get back a ToolsAPI OTAEditView interface. This one is documented and publicly useable and you’ll find it in ToolsAPI.pas, and it includes a Buffer property which has a FileName.

Mangled name and prototype:

We don’t need to hook this method, just call it! “Self” is the IDE’s EditView object we just got in the above hooked method.

Putting it all together – pairing edit controls and filenames

A quick example of how to use these together: hook SetEditView. In your hooked method, call EditView.GetOTAEditView. Using the resulting OTAEditView, do whatever you want, such as get the filename.

There are several caveats about what might be nil in certain places, some of which can cause crashes.

EditView can be nil. The resulting OTAEditView’s Buffer can be nil for some views. Check everything else as well. (My code doesn’t actually assert; it checks. Maybe it’s overcautious, but remember, if this code crashes, it will crash the IDE.)

Note that in these code examples I switch between naming the object the hooked method is called on “Self” and “Sender”. “Self” is because, if it was implemented using OO syntax, the object would be accessible via Self. In my code I often dispatch these hooked methods to another implementation, which is in a class, and so there Self is my class instantiation not the IDE object. At that point I refer to it as Sender, similar to events.

Through this, you can keep a map of edit control to OTAEditViews and associated data. This means, when painting a specific edit control, you now know which view is being painted, and that means which file.

Line numbers and folded / elided code

PaintLine

In the first article, I didn’t explain all the parameters of the TCustomEditControl.PaintLine method.  Here’s a better prototype:

A and B are still opaque and I’m not sure what they represent. LineNum and LogicalLineNum on the other hand are very useful. But what’s the difference? The answer lies in code folding and what the IDE calls “elided lines”.

Code folding is when a block of code in the editor, such as a method or region, is collapsed.

LineNum is always continuous onscreen: 1, 2, 3…

LogicalLineNum jumps if one or more lines are collapsed: 1, 2, 3, 7, 8…

You are probably interested in knowing which line of code is being painted, especially if you want to annotate the code (for example a spell checker might underline some words, but it needs to know which line of code it is in order to look up via the line number the spellcheck results which it calculated elsewhere.) The field of interest is LogicalLineNum, since it represents not the line onscreen but the code line number. Ie, it is the number you would see painted in the gutter or in a compiler error message.

Calculating more data about elided lines

The data in PaintLine is great simply for painting simple cases, but it’s not enough for more complex situations. Each line is painted standalone: there is no information about the lines above or below. Consider the above screenshot: the method TForm1.FormCreate is folded. When painting the line “procedure TForm1.FormCreate…” LogicalLineNum will be the line of code. There is no way to know from the parameters in PaintLine (that is, the parameters that have been decoded so far; the IDE knows!) if this line is actually a folded line – that is, if the line after it is not LogicalLineNum+1 but something else entirely, and when painted it has the + symbol to the left and it the ellipsis (…) on the right.

However, through another IDE method you can build this information for the view.  The method in question is TCustomEditControl.LineIsElided, and its mangled name and prototype is:

This allows you to ask the edit control – which is showing the contents of a view – if a particular line is “elided”, that is, is hidden.  LineNum is a real (logical) line number.

Combine this with OTAEditView’s TopRow and BottomRow properties, which tell you the first and last (logical / real) lines onscreen. With this as a valid range, you can use the above method to build up information about if the line is visible, if it is hidden, and most importantly if it is a line the line after which is hidden, ie it’s a line that shows the collapse/expand + symbol and ellipsis.

Miscellaneous useful data

So far so good for lines of code onscreen, ie vertical placement. However, the code editor can be scrolled horizontally too; while that doesn’t affect which line is drawn, it does affect how much of the contents of that line are visible. Consider the example of the code spellchecker above: if there is a spelling error at the token between characters 7 and 12, how does it know where onscreen characters 7 and 12 are?

The answer lies in yet another IDE method, TCustomEditControl.PointFromEdPos.

This method takes a TOTAEditPos, a ToolsAPI type which specifies a line number and column (character) number, and returns a TPoint.  If a character takes up a rectangle onscreen (since it is a monospaced font), this point is the pixel coordinate of one corner of the character. To get the enclosing rectangle, the top left point is that given by calling PointFromEdPos with (column, line-1) and bottom right point is that given by calling it with (column+1, line) minus 1px.

Again, mangled name and prototype:

The meaning of the two boolean parameters is unknown to me, but they don’t seem to be necessary for the purposes given here.  Pass true for both.

And an example of calling it:

This can be useful for a number of things: not only knowing onscreen where a specific token is (if you know that token’s position in the line of code being drawn), but also, say, the leftmost edge of the code area when called with column OTAEditView.LeftColumn, and so forth. This lets you account for scrolling as well.

What can you achieve?

With the combination of the first article and the above, you can now:

• Paint on the code editor in the editor’s PaintLine method
• Know which view is being painted
• Know which file is being painted
• Know which line is being painted
• Know which lines of code are visible onscreen and which are folded / invisible, and which lines onscreen are the folding line
• Get the pixel position of text onscreen from the character and line

If you combine this with code to get the text of the file in the view, you could, for example, write a plugin that spellchecked code and drew a wiggly line underneath errors. If you combined it with a Delphi parser, you could write a better Error Insight, or you could completely take over code drawing, never let the IDE’s default painting occur at all, and paint the code absolutely differently. The possibilities are endless: the code editor is one of the things in Delphi that is ripe for having much more functionality added. There aren’t enough of us writing code editor plugins (me, Castalia, CnPack, ModelMaker, …?) and it’d be great to see more people in that list. Just pick something unique to do!

Coexisting

Code hooking

In the first version of this article I used some quite simple hooking code.  It works, but there are problems, and instead I recommend you take a different approach.

The reason is to do with what code hooking does.  It rewrites memory in the process, in executable code, to jump to your code instead of executing the original.  For all of the hooked IDE methods, though, you need to call the original – you need to let the IDE paint, even though you paint yourself, for example.

Most hooking code solves this problem by writing the jump, and then in the method you provide that is jumped to, you call code to patch the memory back to how it was, call the original method (which is now no longer patched so won’t jump back to yours but will execute as it originally was meant to), and then patch it again afterwards so that next time it is called it jumps into your method again. This is slightly convoluted but allows you to both hook the method (and have your own executed in its place) and to call the original.

The problem lies in all this patching. It’s fine if your code is the only code that patches the IDE’s memory (although it’s a bit of a performance hit to patch so often.) But what happens when there are two pieces of code both trying to patch the same method – two plugins both hooking TCustomEditControl.PaintLine, for example? Basically, it’s a mess.  There are several different methods of patching, and some overwrite others. (For example, one simple method calls the original code by patching back, and calling the method pointer again, then re-patching.) It’s very easy – and in fact with some code unavoidable – to patch in a way that breaks other plugin’s patches, so that your code is called but other plugins’ code is no longer called at all.  That means that your plugin can break other plugins, not a good thing. Worst of all, it makes a plugin appear to be broken when it isn’t (it’s someone else’s code that breaks it), and this behaviour can change depending on the order of which plugin is loaded first, which usually depends on which was installed first, which is effectively random.

While investigating some bugs I built a table of different patch code out there in the wild, applied in various orders (simulating one plugin being loaded before another, then reversing that), and built a table of the results when a hooked method was called – what worked and what didn’t. It’s large and potentially confusing to include it here, but the results are that there are definite best practices for which library to use and when and how to patch.

The guideline for this is:

• Patch using Delphi Detours. It’s an excellent patch library, written by Mahdi Safsafi. Specifically, one great thing it does is allow you to call the original method via a “trampoline”. You do not need to unpatch and repatch just to call the original. (As it happens, there are performance benefits here too.) This also means it supports “chaining” patches, when you follow some restrictions.
• Patch each method you are interested in once. Then, do not unpatch it until your plugin is unloaded as the IDE shuts down. If your code stops being interested in the method, don’t unpatch it; instead leave the patch there but have your code’s logic be to just call the trampoline.
  The reason for this is that patches are applied in one specific order (the order plugins are loaded); are never modified – ie accidentally overwritten – during normal runtime; and are unpatched in reverse order when the IDE unloads its plugins.

My plugins, CnPack, and Castalia can all be loaded into the IDE at once, and all work despite the fact we all patch the same methods. That’s quite a feat. I am very grateful to the maintainers of CnPack for accepting a patch I wrote to change the hooking method that project used, and I am grateful also to Jacob Thurman of Castalia for writing his code the same way. Thanks guys!

If you write an IDE plugin, please follow the same techniques.

Other things to be aware of

A few other miscellaneous notes:

• The code editor is quite performance-conscious in how it paints. It paints each line only when it needs to; if you scroll, it will only paint the lines that become visible, not all lines onscreen.
  • Because of this, be aware that if you paint something position-sensitive (let’s say, text saying “this is 100 pixels from the top of the screen”) it won’t be repainted when the user scrolls and will be out of date unless you specifically invalidate it.
  • This goes for horizontal scrolling too.
• Painting can be expensive. Do not invalidate the entire code editor unless you absolutely have to. Instead, invalidate the smallest possible rectangle – the line, or even better, part of the line. This is normal best practice for other programs too but often isn’t adhered to – it’s easy to get in the habit of calling TWinControl.Invalidate. Don’t do that here, and definitely don’t use Repaint. Use InvalidateRect instead, with the smallest rectangle possible.
  If you ignore this, two things will happen:
  • You may drastically slow down the IDE. Scrolling may be slow. Painting of anything above the code editor will be slow, which means something like moving a form over the editor might be slow even on modern windows with Aero and the DWM. This includes things like pop-up code insight. Even typing might be slow.
  • You will see a lot of flicker in the code editor. Flicker is a good sign you’re invalidating too much.  The code editor doesn’t seem to be double-buffered (sadly) but usually it doesn’t matter and you rarely see painting flicker in it. If you do see flicker, it’s a good sign you’re invalidating too often and too much each time.
• The problem is not that painting is slow (although it can be depending on what needs to be painted) but how often it is painted. In my testing, it is okay to have code that is slower to paint each time, or that spends a few extra cycles calculating the minimum area to invalidate, but does so rarely. Make sure it’s fast enough the user can scroll without noticing a difference, and then don’t worry: within reason, only worry about the number of times you paint and how much of the screen you cause to be repainted, not how long it takes each time. An expensive operation done rarely is better than a slightly faster operation done way too often and too much.
• The TPaintContext structure. Unofficially, the contents of it have been decoded. My own code does not use it, because it’s unnecessary. Using a TControlCanvas to paint (as in the first article) is not significantly slower, and means the canvas is your own object – no need to worry about saving and restoring state, etc. The other fields are mostly also things you can figure out either through the View or by reading the IDE options. Since it’s an undocumented internal structure and could change at any moment, it’s also not as safe to use. (The method prototypes could also change, but they’re unavoidable, and also significantly easier to figure out should they change. The fewer dependencies on IDE internals, the better.)

Final words

I’ve discovered all this through the process of writing two IDE plugins of my own.  The first, Parnassus Bookmarks, is free and is almost ready to be released.

These are two screenshots of the Bookmarks plugin. As you can see in the top screenshot, it makes use of knowing about folded code in order to paint when there are bookmarks inside a folded area. (The large 5 is just an animation caught halfway.) The screenshot on the right shows even more: the small caret symbol is a temporary bookmark (one you can drop and then navigate back to with a single keypress) and it draws at the character location where it was placed.

This plugin is free and will be available within about a week, maybe two. (Currently ironing out a codegen bug in XE and XE2 with generics and interfaces.)

The second plugin is a secret project, and is commercial. Expect to hear more over the coming months.

For those who are interested, I have developed a couple of support libraries while building these plugins. One of them is the ‘Parnassus Core Editor’ library, a metaplugin useable by other plugins for all the bookkeeping required when managing editors and painting: not just which editors and views are related, to get the editor state, line states, manage code folding and elided lines, file open/close notifications, getting file contents, handling changes in files and specifying what those changes were (“two lines inserted at line 5”) and responding such that data can “stick” to a line as it is mutated through the IDE, mouse events to (for example) catch clicking in an editor and create your own buttons / hotspots, keyboard events, etc. There’s a lot there; about nine months of work, all in neat interfaces and collections and subscribable events. I’m considering opening this up to others. I’m uncertain about how (license? Open source? Mixed, licensed for commercial use?) but if you’re interested in licensing it, feel free to get in touch.

Finally, I hope you’ve found this series useful. I would like to promote more people writing Delphi plugins – sometimes it seems an arcane art, yet it’s an area where so much could be done and it would be great to see a larger ecosystem grow – and I hope that sharing the information I have helps that. Further, I hope that you can take what’s written here and write something amazing.

David runs Parnassus, a small Delphi consulting company. As well as hacking the IDE and writing plugins, you can also hire him to write code for you. Good code! Read more here.