Place a third of the crackers (about 25) in the bottom of an ungreased 13-in. x 9-in. pan.
In a small saucepan over medium-high heat, melt butter. Add the milk, graham cracker crumbs and sugars; bring to a boil. Cook and stir for 5 minutes. Pour half of the mixture over crackers, carefully spreading to cover. Place half of the remaining crackers (about 25) on top. Spread with remaining sugar mixture. Top with remaining crackers.
In a small saucepan over low heat, stir peanut butter and chips until melted and smooth. Spread over crackers. Chill until firm, about 1 hour. Cut into small squares.