In a large bowl, beat cream until stiff peaks form; set aside. In another bowl, beat the cream cheese, pumpkin, brown sugar, 1 teaspoon pie spice and 1 teaspoon vanilla until blended. Fold in whipped cream.
In a small bowl, combine coffee and remaining pie spice and vanilla; brush over ladyfingers. In a 3-qt. trifle dish, layer a fourth of the ladyfingers, angling some up the sides of the dish if desired. Top with a fourth of the pumpkin mixture and whipped topping. Repeat layers 3 times. Sprinkle with additional pie spice.
Cover and refrigerate until chilled, about 4 hours.