I recommend you do the following:
After following those steps you wont have the same doubts and the experiment would have worked better.
Important notes:
Make sure that the temperature in both the fridge and outside will not change. If the temperature changes it might affect the experiment.
When opening a fridge it causes a temperature loss in the fridge, and that is not something we want. So, instead of putting one in the fridge, you can find another place in the house that is cooler than the place where the other banana is set.
Hope it helped,
BiologiaMagister