In the Wee Small Hours is indeed wonderful, and some say it's his best, but for me Songs for Swinging Lovers takes it by a nose. They're very different records. ITWSH is lush and lovelorn, with Frank slipping effortlessly into the role of heartsick paramour. SFSL on the other hand is loud, raucous and stuffed with the joy of life. You Make Me Feel So Young, It Happened in Monterey, Pennies From Heaven, I've Got You Under My Skin (that demented trombone!), Making Whoopee, Anything Goes. Plus it is the best singalong record ever: see if you can match his phrasing! If you've ever wondered why Sinatra is described as genius, SFSL reveals why.
Whatever, Sideshowbob's right. Stick with the Nelson Riddle collaborations as a starting point and you really can't go wrong.