I recently told someone I felt guilty spending so much time knitting and crocheting, and she challenged me to think about this in terms of purpose.
What I realized is that most of the things I make are donated, so someone somewhere is (hopefully) keeping a bit warmer and more comfortable because of a blanket I made.
If I make something that I keep, such as a pillow cover, and I enjoy looking at it, and feel proud, then that was well worth my time and effort.
If I make a gift, that is something that makes the recipient happy, so definitely a good feeling all around..
Sure, I could be doing housework or plenty of other things, but knitting or crocheting for a purpose is nothing to feel guilty about.
As long as other important things get done, i.e. food shopping, laundry, and paying the bills, then why not feel good about spending time with yarn?