How can I effectively work with UTF-8 strings in ImHex?

[methapi]

I need some useful advices for editing binary files containing UTF-8 encoded values. Currently I have these issues with that while tinkering with ELF:

1. As many users of ImHex know from i.e. from this issue, custom encodings that interpret multi-byte expression into a single symbol (like in UTF-8 I'm talking about) renders this type of encoding table very lazy. So, I can't see UTF-8 decoded strings to see what I"m actually editing.

2. OK. I can work without that. So I found some string via Search and moved to its address. As WerWolv suggested in the issue I've mentioned above, I used Decode with encoding feature to at least see the found text. Look here.

That's a useful thing, but not at all in editing. I need:

• To find out which symbol I need to edit (its position);
• To edit it (obviously), BUT in the way of inputting UTF-8 symbols directly from my keyboard, NOT trying to guess which combination of bytes I need to enter manually to get an actual unicode symbol.
  At this step the editing process becoming, to put it middly, dreadful. Hex editor doesn't allow to put more than 1 byte a time even in ASCII column. I'd edit the data with Data inspector, but while working with .so, I have to save the length of data region, which is ignored by String option of Data inspector as it doesn't show my changes live.

What should I do avoiding to switch into some other HEX editor which may be more stable with editing multi=byte values?

[paxcut]

Have you tried using the pattern editor? It supports utf-8 so you should be able to type the chars directly. select them, copy them and paste them to the hex editor as a string should enter all the bytes that encode the chars, at least it is doing that for me but my keyboard doesn't create utf-8 unless i press some keys that allow me to type the codes so i can't test that part.

[scottanderson]

Utf-* decoding must still be aligned to the beginning of a codepoint's cell .

[paxcut]

like I said above for utf-8 assuming you are somewhere within a valid sequence a maximum of 3 (4?) look-backs guarantees you'll find a starting byte. if not found then you can start moving forward to look for a starting byte. So alignment necessary? yes, but does it require starting a the beginning of the document? No. You can start at an arbitrary offset and find all valid sequences in the visible segment.

[scottanderson]

I think you're responding to something I did not say. It might help if we use the terminology from the spec here, since this distinction seems to be causing recurring confusion. Decoding is a specific operation defined by the Unicode spec. If the sequence beginning at the supplied offset is ill-formed, the decoder must treat that as an error. That's a fact defined by the spec, not something to debate.

Finding the codepoint boundary from an arbitrary offset is a separate operation from decoding. I was not contradicting anything you said about finding codepoint boundaries.

[scottanderson]

“When a process interprets a code unit sequence which purports to be in a Unicode character encoding form, it shall treat ill-formed code unit sequences as an error condition and shall not interpret such sequences as characters.” https://www.unicode.org/versions/Unicode17.0.0/UnicodeStandard-17.0.pdf

[paxcut]

You posted that utf must "still" be aligned (quotes for emphasis). I assumed you were responding to my post on decoding from arbitrary offsets. There are no discussions on how to treat ill formed sequences that i know of and nobody is suggesting we accept them, so what are you responding to?