The following code suddenly caused the error above:

SET OF UNSIGNED4 computeGroups(INTEGER rid, INTEGER modulo) := BEGINC++
...
ENDC++;

Layout_RecordToReplicate := RECORD
SET OF INTEGER groups;
Layout_Record rec;
END;

Layout_RecordToReplicate computeR(Layout_Record d) := TRANSFORM
SELF.groups := computeGroups(d.RecordId, modulo);
SELF.rec := d;
END;

r := PROJECT(INPUTDS, computeR(LEFT));

Layout_Record normFct(Layout_RecordToReplicate r, INTEGER c) := TRANSFORM
SELF.GroupId := r.groups[c]; // this index access seemed to cause the error
SELF.RecordId := r.rec.RecordId;
SELF.TokenCnt := r.rec.TokenCnt;
SELF.TokenIdSet := r.rec.TokenIdSet;
END;

rNorm := NORMALIZE(r, COUNT(LEFT.groups), normFct(LEFT, COUNTER)); // here the error occured


The issue was that the data type of the groups in Layout_RecordToReplicate didn't match the return type of the C++ function (SET OF INTEGER vs. SET OF UNSIGNED4). ECL didn't complain about this syntactic issue, so the error occured at a later stage when actually accessing the set.

UPDATE: When returning a SET from a C++ function it is important to set __isAllResult = false;