In a large heavy saucepan, bring milk and cream to a simmer. In a bowl, whisk together egg yolks, sugar, cornstarch and salt. Whisk a small amount of hot milk mixture into egg yolk mixture; return all to pan, whisking constantly.
Bring to a gentle boil; cook and stir 2 minutes. Remove from heat. Stir in butter and vanilla. Cool 15 minutes, stirring occasionally.
Line bottom of a 3-qt. trifle or other glass bowl with half of the vanilla wafers; cover with half the banana slices followed by half the pudding. Repeat layers. Press plastic wrap onto surface of pudding; refrigerate until cold. Just before serving, remove plastic and top with whipped topping. If desired, serve with caramel syrup.