In a saucepan, soften gelatin in milk for 1 minute. Stir in sugar and salt. Cook and stir over medium-low heat for 5 minutes or until gelatin is dissolved. Remove from the heat.
Stir a small amount of hot mixture into egg yolks. Return all to the pan. Cook and stir over low heat until the mixture thickens slightly and coats the back of a metal spoon or reaches 160° (do not boil). Remove from the heat. Add candy and food coloring; stir until candy is dissolved.
Refrigerate, stirring occasionally, until mixture begins to thicken, about 30 minutes. Fold in whipped cream. Place ladyfinger halves around a greased 9-in. springform pan. Pour mixture into center of pan. Cover and chill for 4 hours or overnight. Just before serving, run a knife around edge of pan to loosen; remove sides.