0.8 Work to the following types of dwellings does not need to comply fully with the energy efficiency requirements where to do so would unacceptably alter the dwelling’s character or appearance.
a. Those listed in accordance with section 1 of the Planning (Listed Buildings and Conservation Areas) Act 1990.
b. Those in a conservation area designated in accordance with section 69 of the Planning (Listed Buildings and Conservation Areas) Act 1990.
c. Those included in the schedule of monuments maintained under section 1 of the Ancient Monuments and Archaeological Areas Act 1979.
0.9 Work to a dwelling in paragraph 0.8 must comply with the energy efficiency requirements where this would not unacceptably alter the dwelling’s character or appearance. The work should comply with standards in this approved document to the extent that it is reasonably practicable. Historic and traditional dwellings
0.10 The energy efficiency of historic and traditional dwellings should be improved only if doing so will not cause long-term deterioration of the building’s fabric or fittings. In particular, this applies to historic and traditional buildings with a vapour permeable construction that both absorbs moisture and readily allows moisture to evaporate. Examples include those built with wattle and daub, cob or stone and constructions using lime render or mortar.
0.11 New extensions to historic and traditional dwellings should comply fully with the energy efficiency standards in this approved document unless there is a need to match the external appearance or character of the extension to that of the host building. The work should comply with standards in this approved document to the extent that it is reasonably practicable.
0.12 In determining whether full energy efficiency improvements should be made, the building control body should consider the advice of the local authority’s conservation officer.
0.13 Additional guidance is available in Historic England’s Energy Efficiency and Historic Buildings: Application of Part L of the Building Regulations to Historic and Traditionally Constructed Buildings.
So we have the power to avoid harm to character and appearance within a conservation area and if anyone gives you that ‘comply with building regs’ line be sure to consider the potential harm
Historic Englands website has a useful page detailing the interaction between requirements of building regs and the historic environment: