typelocked undefined4 variable

[Wall-AF]

How can a variable that is of datatype undefined4 fail to change type and complain that it is typelocked?

I added some WARNING output to the decompiler code in Funcdata::attemptDynamicMmppingLate(...) (funcdata_varnode.cc), but I don't know where to go from there! Any ideas???

[gitsult4n]

The lock is on the Symbol, not on the data-type, and undefined4 does not clear it. It only downgrades what the lock means.

undefined4 is a core type with metatype TYPE_UNKNOWN, and Symbol::checkSizeTypeLock turns that combination into a size lock while leaving the type lock set, database.cc:276:

void Symbol::checkSizeTypeLock(void)
{
  dispflags &= ~((uint4)size_typelock);
  if (isTypeLocked() && (type->getMetatype() == TYPE_UNKNOWN))
    dispflags |= size_typelock;
}

size_typelock is documented as "Only the size of the symbol is typelocked" (database.hh:264). So isTypeLocked() stays true, and everywhere in the decompiler that wants the type to float has to ask the second question, isSizeTypeLocked(). Most places do: ActionNameVars::linkSymbols (coreaction.cc:3115), the PTRSUB path (funcdata.cc:415), syncVarnodesWithSymbols (funcdata_varnode.cc:991), and Varnode::updateType, which refuses to lock a varnode to an unknown type at all (varnode.cc:501, "Unknown data type is ALWAYS unlocked").

The function you instrumented is the one that does not ask, funcdata_varnode.cc:1444:

  vn->setSymbolEntry(entry);
  if (!sym->isTypeLocked()) {
    localmap->retypeSymbol(sym,vn->getType());
  }
  else if (sym->getType() != vn->getType()) {
    ostringstream s;
    s << "Unable to use type for symbol " << sym->getName();
    warningHeader(s.str());
    localmap->retypeSymbol(sym,vn->getType());
  }

A symbol locked to undefined4 takes the second arm, the pointer comparison against the propagated type fails, you get the warning, and line 1451 then retypes the symbol anyway. In that path the warning is cosmetic. The other one in the same function, "Unable to use symbol ... Size does not match variable it labels" at :1420, is the one that actually refuses to attach the symbol.

Now the useful part, where a typelocked undefined4 even comes from. A plain local never gets one, LocalSymbolMap.java:167:

boolean istypelock = true;
if (Undefined.isUndefined(dt)) {
    istypelock = false;
}

so the lock arrived from one of these instead:

• A parameter. Locking is decided by the function's signature source, with no undefined check, LocalSymbolMap.java:207 boolean lock = (dbFunction.getSignatureSource() != SourceType.DEFAULT); and :245 paramSymbol.setTypeLock(lock);. The Decompiler Parameter ID analyzer commits with ANALYSIS, so every parameter of that function is locked, undefined4 ones included. Set the signature source back to DEFAULT (Edit Function, or getFunctionAt(addr).setSignatureSource(SourceType.DEFAULT)) and they all unlock.
• Listing data. HighCodeSymbol locks unconditionally, HighCodeSymbol.java:45 and :92. Clear or retype the datum in the Listing.
• A variable made with Split Out As New Variable. Isolation forces the lock, database.cc:310 and :477. Retype it back to undefinedN or delete the local.
• A typedef whose base is undefined: Undefined.isUndefined does not strip typedefs, but the metatype sent down is still unknown, so you get lock plus unknown.

For a hash-storage local specifically, Commit Local Names is worth running, it calls clearObsoleteDynamicLocalsFromDatabase (HighFunctionDBUtil.java:301) which deletes dynamic locals whose hash no longer attaches to anything, and a stale one of those is a common cause of a retype that appears to do nothing.

To pin it down rather than guess, dump what the GUI actually sends: Debug Function Decompilation in the decompiler window (DebugDecompilerAction.java), then look for the <symbol ... typelock="true"> element for that variable and whether its entry is a <hash> or an <addr>. And in your own instrumentation print sym->isSizeTypeLocked() next to isTypeLocked(), sym->getType()->getName() and vn->getType()->getName(). If the size-lock flag is set, nothing is stuck, you are looking at a spurious warning.

Which is arguably worth an upstream issue: guarding funcdata_varnode.cc:1447 with && !sym->isSizeTypeLocked() would match what every other consumer does and drop the warning without changing behaviour.

Read off master today (application.version 12.2). Source only, I did not build the decompiler.

[gitsult4n]

Three corrections to my post above, one of which matters because it would have made things worse.

First, the patch I suggested is wrong. funcdata_varnode.cc:1447 is the else if, so adding && !sym->isSizeTypeLocked() there makes both arms fall through for a size-locked symbol and localmap->retypeSymbol(sym,vn->getType()) never runs, leaving the symbol on undefined4. That is the very symptom you are chasing. If anything is worth proposing upstream it is the guard on the first branch instead:

if (!sym->isTypeLocked() || sym->isSizeTypeLocked()) {

Second, line numbers. I read master, which is 12.2-DEV, and those offsets do not resolve in a released build. On Ghidra_12.1.3_build the same function is at funcdata_varnode.cc:1378, the type-lock branch at :1420, the warning string at :1425, and the size-mismatch warning around :1394. Roughly 25 lines lower than what I quoted.

Third, and most useful to you: that function only ever sees dynamic locals. ActionDynamicSymbols::apply walks localmap->beginDynamic() to endDynamic(), so only hash-storage symbols of the local scope reach it. Parameters are built with newMappedSymbol (LocalSymbolMap.java:238) and never become dynamic entries, and Listing globals are not in the local scope at all, so of the origins I listed, the parameter and the global one cannot be what you are looking at through that function. Same for "Split Out As New Variable": IsolateVariableTask.java:100 substitutes an unsigned integer first, with the comment "An undefined datatype will not be considered typelocked", so isolation yields a locked uint, not a locked undefined4. What is left for a hash-storage local is a typedef whose base is undefined, or a symbol that carried a lock in from a previous commit of locals.

One more thing worth knowing before you keep instrumenting. There is an early return above your code, :1413 on master and :1389 on 12.1.3:

if (vn->getSymbolEntry() != (SymbolEntry *)0) return false; // Symbol already applied

The early pass, attemptDynamicMapping from ActionDynamicMapping, normally attaches a locked symbol first through Varnode::setSymbolProperties (varnode.cc:437, which tests isTypeLocked() and is true for a size-locked symbol), so the late pass bails there. If your instrumentation is firing at all, the early attach did not happen for that varnode, and the interesting question is why. Printing entry->getSize(), vn->getSize() and whether attemptDynamicMapping matched the same hash will tell you more than the warning does.