Figured it out. When the scene buffer is cleared values of 0 are treated as unused and if the next program uses something that had a zero, it adds it to the current scene. Just raising all values first and then setting them back to 0 and setting your scene keeps this from happening.