Mysteries of IDE plugins: Painting in the code editor (Part 1) 16


Part 1 of a series on how to write an IDE plugin that can paint on the IDE code editor line-by-line along with the code.

Introduction: IDE plugins

IDE plugins are a mysterious topic.  With little official documentation, most material is scattered across a variety of blogs and websites, often out of date and referring to pre-Galileo IDEs.  But the documentation is improving and there are a number of great websites (try David G Hoyle’s – the best website on the the ‘Open Tools API’ I have seen) delving into the interfaces that OTAPI makes available.

If you want to write a normal IDE plugin, such as a dockable window containing your own material, something that interacts with the projects or project groups, adds or removes text to source code in the editor, implements a debug visualizer, etc, then the above collection of links plus some Googling should get you going.

But there are some things that the IDE’s API simply does not expose, things that advanced plugins need to do – like painting on the code editor.  There is no painting access or interface in the published API.  So how do open-source (like GExperts or CnPack) or commercial (like Castalia) plugins do it?

I can’t answer how commercial plugins work, since I don’t know, though I suspect very similarly to the technique presented here.  But I can tell you a technique I figured out with the help of looking at how CnPack achieved it to solve a few issues.  It works well, and seems stable, reliable, and fast.

This is the first of a series of posts on advanced Delphi IDE plugin functionality. There are several goals: now I’ve found out how to do some of these trickier things, I want to share the information.  Second, I am writing a family of IDE plugins – things to fix parts of the IDE that after many years of usage I wish worked differently, or to add functionality that doesn’t exist.  And while I want to sell these plugins for a (small to trivial!) cost – I hope you, Delphi readers, will find them as useful as I will – I don’t want to hide the information I learned about how to build them.  Quite the opposite: I’d love to see a healthy plugin ecosystem develop around Delphi and Appmethod, rather than the small number of open source and commercial plugins that currently exist.

The following assumes you have a rough idea of how to create an IDE plugin – ie how to create a BPL that it loaded by the IDE.  But even if you don’t, you can read on anyway :)

Contents

  • Introduction, and a few quick notes on IDE plugins (above)
  • What is TEditControl?
  • Painting unsuccessfully: by hooking windows and messages
  • Painting successfully: patching IDE methods at runtime

By the end of this article, if you already know how to write an IDE plugin, you should be able to make your plugin paint on the code editor in an event that occurs each time a line is painted.  Completing the functionality, such as drawing in the right spot for each line, handling the gutter area, handling folded code etc, will be discussed in future articles in this series.

What is TEditControl?

Delphi/C++Builder is written mostly in Delphi, and so uses VCL menus, actions, and controls, an Application object, etc.  You can see this when writing standard plugins, where you use a descendant of TForm (eg a TDockableForm) to make a dockable window in the IDE, add images to the IDE’s image list, add menu items, etc.

This extends to the code editor itself, an instance of TEditControl, an internal Embarcadero-only descendant of TWinControl.  Here you can see it in Spy++:

An instance of TEditControl seen inside the IDE.

There is a single instance of the edit control shared by all source code tabs in the main IDE window – the tabs are more like a TTabControl than a TPageControl.  However, you can have multiple edit windows, each with an edit control, even showing the same source code.

The relationship between TEditControls and the documented OTAPI interfaces, such as IOTAEditView, can get complicated and will be addressed in another article.

But how do you find this control and paint on it?

Painting unsuccessfully: by hooking messages

I tried a number of techniques that didn’t work and I won’t delve into great detail.  My rough approach was to hook the window messages or events related to painting, and also to scrolling or other events in order to update internal information in my own plugins.

For example, I tried setting the control’s WindowProc to my own, in order to catch WM_PAINT messages – none were caught.  You could also try finding an OnPaint event or similar.  I also tried catching WM_HSCROLL and WM_VSCROLL messages via WindowProc, installing an application OnMessage handler (note if you try this approach that it uses a multicast event dispatcher, so don’t assign your OnMessage event to Application.OnMessage directly), and installing a message hook, and none of those methods caught those messages either.  I don’t know why: it’s possible that I was hooking the wrong control (despite it being the edit control) or that the messages are handled somewhere else.  I don’t know the internal structure of how the IDE interacts with the edit controls.

I would recommend you do not go down this road, since I had no success with this or several variations of it.  (That’s one short sentence that describes quite a bit of spent time.)

To do any of these, you also need to find instances of the edit control itself, wherever it is (or they are) in the IDE, another minor complication.  But it turns out that purely for painting, you don’t need to know about any specific instance of the edit control.  Instead…

Painting successfully: patching IDE methods at runtime

One method of implementing painting I had read about online was hooking or patching methods of the IDE classes.  This was referred to obliquely, as ‘you can patch the methods’, with no details about how.  So figuring out how was my next approach.  There are two steps: how to patch any arbitrary method, and then finding the right method(s) to patch.

How to patch an arbitrary method at runtime

A method has an address: a location in memory at which it starts.  Virtual methods or thunked (eg most Windows API) methods jump to this from another location in a variety of ways.  To patch a specific method, you need to find its true location (following the virtual call or the thunk) and then overwrite the first few bytes of the method itself with a call to jump to your replacement.  If your replacement does everything you want, that’s all you need to do, but if you want to call the original implementation as well then you need to replace the bytes you overwrote with their original contents and then call the original method by its original address.

Luckily there are several libraries out there for doing this at runtime in Delphi.

Important note: Please see an update to this section in Part 2 of this series. The below library is the wrong library to use to patch at runtime – it works, but it is incompatible with other plugins also trying to hook the same methods. Instead, follow the method described in Part 2.

The patching code I use is a variant of Chau Chee Yang’s code.  (You could probably also use another, such as the Delphi Detours Library.)  Using it, you can patch a method with code similar to:

 FOriginal := TCodeRedirect.GetActualAddr(…address of a method…);
 FHooked := TCodeRedirect.Create(@FOriginal, @ReplacementMethod);

GetActualAddr is a method that checks for an indirect jump.  I have not investigated the details of how well this handles both virtual methods and WinAPI-style thunked methods – in fact while I know the theory of how both are implemented I’m hazy on the exact details since I’ve never had to examine them closely.  This code works as-is for the purposes of the patching done here.

My version of the above code includes a bugfix for the Disable method: the code on the linked page attempts to write to memory to patch back the method, but it fails because it does not change the protection of the memory first allowing it to be written to. It also does not re-protect it once written, or flush the instruction cache afterwards.  A simple modification based on Enable gives:

procedure TCodeRedirect.Disable;
var
 OldProtect: Cardinal;
 P: pointer;
 n: NativeUInt;
begin
 if FInjectRec.Jump <> 0 then begin
 // David M – based on Enable()
 P := GetActualAddr(FSourceProc);
 if VirtualProtect(P, SizeOf(TInjectRec), PAGE_EXECUTE_READWRITE, OldProtect) then begin
 WriteProcessMemory(GetCurrentProcess, GetActualAddr(FSourceProc), @FInjectRec, SizeOf(FInjectRec), n);
 VirtualProtect(P, SizeOf(TInjectRec), OldProtect, @OldProtect);
 FlushInstructionCache(GetCurrentProcess, P, SizeOf(TInjectRec));
 end;
 end;
end;

Finding the right method to patch

You can now patch a method on a Delphi class live at runtime.  But which method?

At this point, I started searching through the methods in coreideX.bpl looking for appropriate ones to patch.  It looked like being a long search with many false starts and crashes based on incorrect method prototypes.  But I realised that there are open-source projects which implement editor painting already and they may well use this or another method, and so I looked at one to see how they achieved it: CnPack, which I had already downloaded based on David Heffernan’s suggestion for how to keep track of code folding – a topic I will return to in a later article.

Note: I have to thank the authors of CnPack for their code, which helped me solve some problems with my own.  It took me quite some time to figure this out, and I gratefully used their code for reference and help.

CnPack does indeed patch IDE methods – in fact it patches a lot of IDE methods, many more than just this one.  They patch one in particular, though: not a generic paint method for the control, but a method PaintLine – which looked perfect, since the purpose of painting on the edit control is (probably) to paint on specific lines of code, not to paint ‘in general’.

Remember, patching the TEditControl.PaintLine method means that whenever any instance of TEditControl tries to paint a line it calls your implementation instead of its own.  You don’t need to know about specific instantiations, just the class, because you are writing a replacement method for the class.  Your method will then call into the original patched-out code, in order to draw normally, as well as performing whatever painting it wants to do.

Note: this is completely unsupported by Embarcadero (or me.)  Their internal code can change at any time, even in the one version of the IDE.  Nevertheless, this specific method seems to be stable since at least Delphi 2010.  CnPack has a number of different method prototypes for different versions of the IDE, so this has changed in the past and may in the future.

The method in question is Editorcontrol.TCustomEditControl.PaintLine.  It resides in coreide*.bpl, such as coreide160.bpl, which is always loaded into the IDE.

PaintLine seen in Dependency Viewer

In the above screenshot, you can see PaintLine in Dependency Viewer with a “mangled” name. Once demangled via tdump, it is (given in C++ format):

__fastcall Editorcontrol::TCustomEditControl::PaintLine(Ek::TPaintContext&, int, int, int)

We don’t know the structure of TPaintContext (which would be very useful) nor the meanings of those three integer parameters (yet; actually CnPack has deciphered two of them.)  Also not shown is the implicit Self parameter.  But this is enough to create a stub.

Hooking PaintLine and using it to paint

We already have a prototype for the method, but in the form of an object method not a standalone procedure.  To hook, we need to define a compatible method with the extra Self parameter:

TPaintLineProc = function(Self: TWinControl; A: Pointer; B, C, D: Integer): Integer; register;
A is a pointer / reference to TPaintContext, which we don’t have the definition for so cannot use. B, C and D are the three int params.  However, the first parameter is the normally-hidden Self parameter; that is, the object on which this method is called.  Note the register calling convention, which so far as I can tell is fastcall by another name.  This allows you to implement an object-oriented method – one called on an object, that is, with a Self parameter – in procedural style.
Create a stub method using this prototype:
function HookedIDEPaintLine(Self: TWinControl; A: Pointer; B, C, D: Integer): Integer;
begin
 // …
end;

Note: ‘Self‘ here is a TWinControl, since we know that the edit control is a TWinControl descendant, and we can assume that the patch works and will only be called with a TEditControlSelf‘ parameter.  You could also make it a TObject and check its type at runtime.  I did this in my first implementation.

To be completely clear, this is not an object method but a plain, non-OO procedure.

And to get it to be called, hook the IDE’s method:

var
 FOriginalPaintLine: TPaintLineProc;
 FPaintLineHook : TCodeRedirect;
 
 …
 
 FOriginalPaintLine := TCodeRedirect.GetActualAddr(GetProcAddress(FCoreIDEHandle, ‘@Editorcontrol@TCustomEditControl@PaintLine$qqrr16Ek@TPaintContextiii‘));
 // Patch the code editor’s PaintLine to use our method
 FPaintLineHook := TCodeRedirect.Create(@FOriginalPaintLine, @HookedIDEPaintLine);

Note: FCoreIDEHandle is the module handle of coreide*.bpl, loaded into the current process.  Finding this is left as an exercise for the reader, and there are variety of ways. This method may get you started.

 
Now, fill in the stub method to call the original method:
 
function HookedIDEPaintLine(Self: TWinControl; A: Pointer; B, C, D: Integer): Integer;
begin
 FPaintLineHook.Disable;
 try
 Result := FOriginalPaintLine(Self, A, B, C, D);
 finally
 FPaintLineHook.Enable;
 end;
end;
You need to disable the hook before calling through the pointer to the method because otherwise you will recursively call back into your own hooked implementation, because you are calling the address of the patched code that jumps to your method (because that is the address of the real original method, the first few bytes of which were overwritten.)  Unpatch it back to what it was before calling it.

Note: Another exercise for the reader: call back into your own painting class to call your own code, rather than remain in procedural code only.

At this point you have a lot of work complete, but no visible result at all because your hooked method still doesn’t do any painting and just calls the original.  To paint, you need a canvas, and luckily we are dealing with VCL controls and know that the edit control is a descendant of TWinControl.

Note: Since Delphi is written in Delphi, you can find out quite a lot of information when debugging the IDE in a second instance of the IDE – one example is inspecting the Self object (the edit control) above if you break in your method.

There is a handy VCL class to represent a canvas on a TWinControl: TControlCanvas.  Create and use one:

 Canvas := TControlCanvas.Create;
 Canvas.Control := Self; // The TEditControl
 Canvas.TextOut(0, 0, ‘Hello world’);

The results

Voila!  You now have text on the code editor:
Hello world!

Next steps

There are a number of important missing elements that need to be addressed in order to turn this proof-of-concept code into useful code:
  • It always draws at (0, 0), not at the line that is supposedly being painted.  Since this is called for every line, the above ‘Hello world’ text is actually drawn over itself in the same palce many times, once for each line the code editor paints.  We need to extract the line information and paint at the correct offset down for each line, and correct offset right to handle the code gutter
  • It doesn’t have any idea what unit or text it is actually drawing over
  • It doesn’t have any idea what lines of text are visible
    • …and when it does, that is going to have to include handling code folding
Luckily, these are also solvable and the next article will address these problems too. Until then, consider we have made great progress: we can now paint on the code editor from an IDE plugin!

Why am I investigating these topics?

I’m glad you asked :)

I’ve used Delphi since it was Turbo Pascal, first as a teenager/student and then over the past decade in action as my day-to-day IDE – including C++ Builder.  There are things about the IDE I wish worked differently: mistakes I keep making because I expect certain behaviour, even though I should know by now it works subtly differently.  There are also features I wish it had, and feature tweaks that would just make it a bit ‘slicker’, something it needs in order to compare to IDEs like Visual Studio which have a very polished UI.  Polish matters in an application you use all day, both visually and in behaviour.

I run a small Delphi consulting company called Parnassus, through which I solve problems and write good code.  I’m expanding and am writing a family of small IDE plugins, each of which improves the IDE by changing current behaviour or adding new features.  The first of these, the one through which I have learned about the problems and techniques documented in this blog series, is small – it was my learning project – but useful: a new implementation of bookmarks, one much better than the bookmarks in the IDE or GExperts.

Preview of an alpha, pre-release version of Parnassus Bookmarks

It will be available soon, and if you are interested in beta-testing please contact me by email or by commenting below.

Discussions about this page on Google+


16 thoughts on “Mysteries of IDE plugins: Painting in the code editor (Part 1)

    • Dave

      Great :) Out of interest, what is “inexpensive” as a reasonable price for a plugin?

      As for useful – I hope all will be useful. Are there any specific things you would like to see implemented, though? I have a text file on my Desktop with just under forty different ideas at the moment, all of which I’d like to do… an impossible task without having it as fulltime income, I think. I do hope to build some of the top / best ones over the next few months.

    • IlyaS

      $5 for every feature ;)

      Bookmarks are looking useful already. Autosaving IDE layout, opening unit into code (not to form designer). This doesn’t sound like painting in the code window, right?

    • Dave

      You&#39;re welcome, glad you found it useful.

      I do try to reference things in my posts – IMO a good informative post will tend to have lots of other material “behind” it, the items/knowledge on which it is based. Linking to those makes the post like a Wikipedia article, higher value than the text alone.

    • Dave

      Jeroen: Commercial. It’s a side project beyond the consulting I’m doing at the moment. My plan is to make several small, low-cost plugins, each doing one thing, rather than one single big plugin with many features. I hope that will let people pick and choose what they want added to the their IDE, and make it more affordable as well.

      If it doesn’t work out – and we’re talking the timescale of a year or more to see how it goes – then sure, I would happily donate the code. (I’m not sure how easy it would be to integrate, because I’m building my own base framework that’s quite different in design to what I’ve seen of the open-source ones.) But let’s see how it goes first :) I do like being fairly open about code and IP, and that’s one reason I’m writing this series on the techniques. Even if the articles don’t show the actual production-level plugin source code, they show the basis and may help others write more plugins too.

  • Remy

    Interesting reading indeed. I am a little puzzled that you would use a patching approach that does not utilize a trampoline to avoid re-writing the same memory multiple times on every function call, especially for something called as often as a paint routine. With a trampoline, you can call the original code without having to unpatch and repatch it each time. Maybe that can be an optimization for your next article.

  • Jacob Thurman

    Hi Dave, great work figuring this out. This is indeed one of the techniques I use for things like Castalia’s advanced syntax highlighting (http://twodesk.com/castalia/smart_syntax_highlighting.html).

    I know how big of an undertaking this is. Can I take the liberty of filling in a few blanks?

    The integer parameters of PaintLine are a bit different depending on which version of the IDE you’re in. In pre-Galileo (Delphi 1-7) IDEs, the parameter you call ‘C’ is the line being painted. In Galileo (Delphi 8-XE6, and presumably beyond), the parameter you call ‘B’ is the line being painted. So you probably want something like this:

    {$IFDEF GALILEO}
    LineToPaint := B;
    {$ELSE}
    LineToPaint := C;
    {$ENDIF}

    You also mentioned some difficulty figuring out where to actually paint the line in question. You can derive this from info in the ToolsAPI (Using IOTAEditView.TopRow and IOTAEditView.BottomRow) and the editor control’s ClientHeight to get the height of a line:

    LineHeight := EditControl.ClientHeight div (EditView.BottomRow – EditView.TopRow);

    Or, since you’re looking at using “internal” methods of TEditControl, there’s one called PointFromEdPos that helpful too, if you translate the line being painted (and the column you want to paint on) into a TOTAEditPos.

    Finally, you mentioned code folding. TEditControl has an IsElideLine method that will probably help you here as well. (Obviously, this doesn’t exist in pre-galileo IDEs, so you’ll want to stub it out if you’re supporting those).

    Thanks for the great writeup, and good luck with your IDE plugins!

    • Dave

      Hi Jacob,

      Thanks for the kind words!

      I have actually figured out some of that already – much of it through looking at CnPack, which gave definitions for two of the parameters (line and logical line numbers.) Ditto for IsLineElided, the use of which you can see in the screenshot at the bottom of the blog post. I didn’t want to confuse the first article by getting into positioning, code folding, or what the parameters actually were, since the article was about getting to the first stage of painting “something” which is quite a big step by itself.

      I’m planning to write this up (getting the right positions to paint in, and handling code folding, etc) in Part 2, to come… sometime :)