does white hat seo matter for a small site?

Agreed. And recovering from a manual action on a small site is slow even when you clean everything up - you file a reconsideration request and wait, sometimes weeks. Meanwhile there's no traffic. That alone makes black hat a bad bet for anyone without a second income stream.

Yes, arguably more than for a big site. White hat just means doing it within Google's guidelines: useful content, a site that works, links people give you because they want to. A big brand can sometimes ride out a penalty. A small site that gets hit for bought links or spammy tactics can lose most of its traffic and never really recover. When you don't have a cushion, the slow way is the safe way.