1970s children's book

I'm trying to remember the title of a book I read as a kid. It was about a boy who finds another family living in his house. The family makes cinnamon toast. There is some question that the boy may have imagined the other family as a way of coping, possibly with grief?

I don't specifically remember the cinnamon toast, but possibly The Borrowers? I seem to remember the framing on that one was that they boy was the only one who knew about them. Or alternately it could be The Littles.