Skip to content

x86: fix decoding of mandatory prefixes#2856

Draft
jxors wants to merge 4 commits intocapstone-engine:nextfrom
jxors:mandatory-prefix-fix-v2
Draft

x86: fix decoding of mandatory prefixes#2856
jxors wants to merge 4 commits intocapstone-engine:nextfrom
jxors:mandatory-prefix-fix-v2

Conversation

@jxors
Copy link
Contributor

@jxors jxors commented Feb 6, 2026

Your checklist for this pull request

  • I've documented or updated the documentation of every API function and struct this PR changes.
  • I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

As mentioned in #2853, the current decoding of mandatory prefixes is not correct. It expects a hard-coded ordering of prefixes that is not required by instruction reference manuals, and also does not handle repeated 66 prefixes.

Initially, I tried to modify the existing function with additional checks. However, it is not possible in general to determine whether a prefix should function as a mandatory prefix without inspecting the opcode, so these checks need to occur later in the decoding process.

To fix this, I have removed the mandatoryPrefix field and replaced it with a function that resolves conflicting mandatory prefixes in the getID function.

Test plan

I have added 23 27 64 tests, comprised of instructions that were originally decoded incorrectly by capstone as well as unexpected edge-cases. For example, instructions with opcode 0FB8..0FBF treat only the REPE/REPNE prefixes as mandatory, and still expect the data size override to function normally.

In general, this patch should only affect instructions in the secondary opcode map with both a REP (F2/F3) and a data size override (66) prefix.

Closing issues

closes #2853

@jxors jxors marked this pull request as draft February 6, 2026 15:06
@jxors
Copy link
Contributor Author

jxors commented Feb 6, 2026

I've marked this as a draft, because 664ef3670f38f62427 is still not handled correctly with this patch.

It used to decode as adcx esp, dword ptr [edi] (which is incorrect). It now fails to decode.

I will fix this issue and do more testing to make sure everything works as intended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

x86: decoding of mandatory prefixes is too strict

1 participant