No, they totally grew enough calories for themselves. My grandparents lived like that. They farmed around 15 hectares, which was actually quite a lot. You can easily grew enough calories for your family on 5 hectares, or even less if you have access to modern cultivars and artificial fertilizer. It’s just even poor people like variety, and will trade some of their crops for stuff they cannot make at home efficiently, like sugar, fish, or candy.