Of course not. Even if we are holding our breath for such a long time until we want or need to breathe again, we still have such a high percentage of oxygen in our blood that we are far from any health drawbacks.
You, of course, can force yourself to stop breathing until you faint. And it seems likely that if fainting is repeated constantly, this may be unhealthy. But this has nothing to do with freediving.