winforms - C# RichTextBox Remove Custom SelectionBackColor

Question

After having done some research, I believe I am asking the same question as Remove richtextbox SelectionBackColor. I have encountered the same issue, but I believe the answers in that thread were insufficient as the question was not clearly explained. Please see below:

In a RichTextBox, how do I remove a custom BackColor from some, but not all, of the text (SelectionBackColor) so that it assumes the BackColor of the control even if that BackColor changes in the future?

I have a method that highlights some text and changes its BackColor using SelectionBackColor. I have another method that changes the BackColor of the entire control. These events can happen independently.

If I want to "remove" some SelectionBackColor, I can try to set the SelectionBackColor to Color.Transparent, but it ends up being White. That is fine, temporarily, if my RichTextBox's current BackColor is White. If I set SelectionBackColor to the current BackColor, it is fine temporarily, until that BackColor changes from another method.

After the RichTextBox.BackColor has changed, any places that were previously highlighted use White or the previous BackColor, instead of assuming the new color like text that had not previously highlighted.

I have tried deleting and replacing the text, but that negates the ability to retain any other custom formatting of that text, to my knowledge. Setting SelectionBackColor to null does not work.

One can easily see what I am talking about using the code below:

protected override void OnLostFocus(EventArgs e)
{
    base.OnLostFocus(e);
    this.BackColor = Color.Gray;
    if (SelectionLength > 0)
    {
        SelectionBackColor = Color.Yellow;
    }
}

protected override void OnGotFocus(EventArgs e)
{
    base.OnGotFocus(e);
    this.ResetBackColor();
    if (SelectionLength > 0)
    {
        // The goal of this line is to "remove" the yellow.
        // By assigning it any value, it seems to have lost
        // the ability to use the control's BackColor normally.
        SelectionBackColor = this.BackColor;// or Color.Transparent
    }
}

Type some text into a custom RichTextBox object with the code above, highlight a small portion of it, then make the box lose focus. You will see the highlighted text in yellow. Then, make the box gain focus. The yellow background will go away, as expected. However, if you move your caret elsewhere in the text and make the control lose focus again, you will see the previously-highlighted text does not assume the gray background color.

Answer 1

This is interesting. It appears that (on Windows 7/.Net 3.5 where I tested and perhaps elsewhere) System.Windows.Forms.RichTextBox.SelectionBackColor may have a bug clearing the selection back color. The source code does:

    public Color SelectionBackColor {
        set
        {
            //Note: don't compare the value to the old value here: it's possible that 
            //you have a different range selected.
            selectionBackColorToSetOnHandleCreated = value;
            if (IsHandleCreated)
            {
                NativeMethods.CHARFORMAT2A cf2 = new NativeMethods.CHARFORMAT2A();
                if (value == Color.Empty)
                {
                    cf2.dwEffects = RichTextBoxConstants.CFE_AUTOBACKCOLOR;
                }
                else
                {
                    cf2.dwMask = RichTextBoxConstants.CFM_BACKCOLOR;
                    cf2.crBackColor = ColorTranslator.ToWin32(value);
                }

                UnsafeNativeMethods.SendMessage(new HandleRef(this, Handle), RichTextBoxConstants.EM_SETCHARFORMAT, RichTextBoxConstants.SCF_SELECTION, cf2);
            }
        }
    }

This does not, as you have noticed, actually clear the back color when set to richTextBox.BackColor. Nor does it clear the color when set to RichTextBox.DefaultBackColor, which simply sets the selection back color to the default grey control color. It looks as though the source code is trying to clear the selection back color when set to Color.Empty -- but on my machine at least, it does nothing.

But if I create an extension method that also sets cf2.dwMask = RichTextBoxConstants.CFM_BACKCOLOR; for an empty color before sending the message, then SetSelectionBackColor(Color.Empty) now works!

    public static void SetSelectionBackColor(this RichTextBox richTextBox, Color value)
    {
        if (richTextBox.IsHandleCreated && value == Color.Empty)
        {
            var cf2 = new CHARFORMAT2();

            cf2.dwEffects = RichTextBoxConstants.CFE_AUTOBACKCOLOR;
            cf2.dwMask = RichTextBoxConstants.CFM_BACKCOLOR;
            cf2.crBackColor = ColorTranslator.ToWin32(value);

            UnsafeNativeMethods.SendMessage(new HandleRef(richTextBox, richTextBox.Handle), RichTextBoxConstants.EM_SETCHARFORMAT, RichTextBoxConstants.SCF_SELECTION, cf2);
        }
        else
        {
            richTextBox.SelectionBackColor = value;
        }
    }

Full method, with constants and classes adapted from here and here and here and here and here:

public static class RichTextBoxConstants
{
    // http://referencesource.microsoft.com/#System.Windows.Forms/winforms/Managed/System/WinForms/RichTextBoxConstants.cs,31b52ac41e96a888
    /* EM_SETCHARFORMAT wparam masks */
    internal const int SCF_SELECTION = 0x0001;

    internal const int EM_SETCHARFORMAT = (NativeMethods.WM_USER + 68);

    internal const int CFM_BACKCOLOR = 0x04000000;

    /* NOTE: CFE_AUTOCOLOR and CFE_AUTOBACKCOLOR correspond to CFM_COLOR and
       CFM_BACKCOLOR, respectively, which control them */
    internal const int CFE_AUTOBACKCOLOR = CFM_BACKCOLOR;
}

[StructLayout(LayoutKind.Sequential, Pack = 4)]
public class CHARFORMAT2
{
    // http://referencesource.microsoft.com/#System.Windows.Forms/winforms/Managed/System/WinForms/NativeMethods.cs,acde044a28b57a48
    // http://pinvoke.net/default.aspx/Structures/CHARFORMAT2.html

    public int cbSize = Marshal.SizeOf(typeof(CHARFORMAT2));
    public int dwMask;
    public int dwEffects;
    public int yHeight;
    public int yOffset;
    public int crTextColor;
    public byte bCharSet;
    public byte bPitchAndFamily;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 32)]
    public string szFaceName;
    public short wWeight;
    public short sSpacing;
    public int crBackColor;
    public int lcid;
    public int dwReserved;
    public short sStyle;
    public short wKerning;
    public byte bUnderlineType;
    public byte bAnimation;
    public byte bRevAuthor;
    public byte bReserved1;
}

public static class NativeMethods
{
    // http://referencesource.microsoft.com/#System.Windows.Forms/winforms/Managed/System/WinForms/NativeMethods.cs,e75041b5218ff60b

    public const int WM_USER = 0x0400;

    public static void SetSelectionBackColor(this RichTextBox richTextBox, Color value)
    {
        if (richTextBox.IsHandleCreated && value == Color.Empty)
        {
            var cf2 = new CHARFORMAT2();

            cf2.dwEffects = RichTextBoxConstants.CFE_AUTOBACKCOLOR;
            cf2.dwMask = RichTextBoxConstants.CFM_BACKCOLOR;
            cf2.crBackColor = ColorTranslator.ToWin32(value);

            UnsafeNativeMethods.SendMessage(new HandleRef(richTextBox, richTextBox.Handle), RichTextBoxConstants.EM_SETCHARFORMAT, RichTextBoxConstants.SCF_SELECTION, cf2);
        }
        else
        {
            richTextBox.SelectionBackColor = value;
        }
    }
}

public static class UnsafeNativeMethods
{
    // http://referencesource.microsoft.com/#System.Windows.Forms/winforms/Managed/System/WinForms/UnsafeNativeMethods.cs,0d546f58103867e3
    // For RichTextBox
    //
    [DllImport("user32.dll", CharSet = CharSet.Auto)]
    public static extern IntPtr SendMessage(HandleRef hWnd, int msg, int wParam, [In, Out, MarshalAs(UnmanagedType.LPStruct)] CHARFORMAT2 lParam);
}