I solve such szenarios by getting a copy of the first collection and then remove alle elements in the copied colleciton from the first.
btw when using weakrefs (I assume you are doing in this sample) . It's important to not rely on the IsAlive Property.
Because it can change right between the check and the next lien of code.
It's a much better pattern to obtain a reference to a local variable and then check if it is null .
As long as the local variable exists the object behind the won't get lost
var myRef = buttons[buttonIndex];
if(myRef != null)