The following is an excerpt from page 14 of the Warhammer Annual 2002 release with an expanded explanation by Gav Thorpe regarding Combat Resolution and Challenges.
So, how do you work out the Combat Resolution for challenges when you've got Overkill, multiple wound magic weapons and warriors with Killing Blow? Actually, it's not as complicated as it seems, as long as you follow the To Wound procedure methodically. First of all, let's remind ourselves that Combat Resolution is equal to the total number of wounds inflicted on the opposing character, capped at five more than they had before the attacks were made. For example, if a character had only a single Wound remaining and failed to save against 8 wounds (perhaps he's fighting a Bloodthirster...), this stops at the maximum of +6 for Combat Resolution.
The same applies to multiple wound attacks. Make all the rolls to wound, make armour saves and Ward saves, and then work out how many wounds have been inflicted. Again the limit is the number of Wounds the character had and up to five more. For instance, a character with 3 Wounds is wounded four times by a weapon that does D3 wounds. He makes his armour saves and still suffers 2 wounds. This becomes 2D3 wounds because of the special attack. Let's say the attacker rolls a total of 5 wounds. This knocks off the character's remaining Wounds and has two left for Overkill, giving +5 Combat Resolution.
Killing Blows and other 'instant death' attacks are worked out as follows. If a character falls to a Killing Blow, this scores the number of wounds they had remaining, just as with ordinary troops. In addition, the attacker scores an Overkill bonus equal to the number of other wounding hits, regardless of whether they were Killing Blows. Saved wounds are disregarded. For example, a character with Killing Blow in a challenge hits his opponent three times. The enemy has 2 Wounds left. When rolling to wound the attacker scores one ordinary wound and two Killing Blows. His enemy has no Ward save and so is dead. This therefore counts as 2 Wounds (enemy's remaining Wounds), with +1 for the wounding hit caused by the second Killing Blow. In addition, his opponent must save the third inflicted wound or this too will be added to the Overkill total.