Those people represent ordinary work not being done, families not being made or being made and shattered, books not written, classes not taught, things not learned, scientific discoveries not made, songs not composed. All of that is stuff we lose collectively. Every person comes into this world with stuff to do, and some of that stuff is really important. When it doesn't get done, we all pay the price.
That does damage. It will continue doing damage for decades, because many of those people will never get back their personal stability even if the economy does somehow recover. This is bad.