In a large bowl, whisk milk and pudding mix for 2 minutes. Let stand for 2 minutes or until soft-set. Combine the pumpkin, cinnamon, ginger and allspice; fold into pudding. Fold in 1/2 cup whipped topping.
Transfer to individual serving dishes. Refrigerate until serving. Garnish with remaining whipped topping. Sprinkle with coarse sugar if desired.