Magical damage is both a direct form of damage and a modifier that adjusts all damage based on the target's magical resistance.
When attacking solely with magical damage, the total damage is adjusted based on the target's magical resistance.
When attacking with multiple attack types, the damage is first calculated on the type of attack (magical or physical) and then adjusted based the target's magical resistance.
The monster has 50% fire resistance, 0% physical resistance and 50% magical resistance.