Stand inside the cellar with the doors closed and the lights off during the daytime. Locate any cracks where the sunlight shines through to determine where cold air is getting in.
Spray expanding foam insulation between any cracks between the door frame and the surrounding walls. Allow the insulation to cure for 24 hours and then inspect the area for any remaining cracks. Apply more insulation as needed.
Stick adhesive foam insulation tape around the inside door frame to fill the gap between the door and the frame. Use a foam tape with a thickness slightly greater than the size of the gaps.
Reinforce the tape by stapling the ends and corners in place with a staple gun. The adhesive may fail in the cold weather experienced by an exterior cellar door.
Cover ground-based hatch-style cellar doors with a thick 6-inch layer of straw during the winter months to insulate them from the cold outdoor air. If winter access is necessary, place the straw in a bag and cover the door. Simply remove the bag when accessing the cellar, and replace it afterward.