Here is the video showing this annoying issue.
(Content removed from quote.)

Last time I somehow fixed it by messing with the skybox. But I did not touched this sicne the last time. I just created outdoor area and brushes/models started to disappear again.

i too have had this problem and i may have figured it out, Q1 what skybox are you using? because when this happened to me I didnt use the zm_factory skybox. But when i made a new map and used the factory box for a night time scene i never ran into it again

I was using zm_factory as well. But still I don't think this is specific skybox issue.

are you caulking the sides of the brush you will never see?

Some of them are caulked yes, But I wanted to caulk everything closer to the release of final and playable version of the map. Do you think this might be the issue?
If I remember correctly I had this issue when my map was just 1 room, I caulked it and it does not helped.

i think its a bug with caulking honestly but i dont know what to tell you other than start from scratch until we find a fix

Well, yes. Anyway thank you. I will try to do smth with the skybox. Or will try to copy everything on the new map and create new skybox there. Last time it somehow helped. Hopefully we will find better solution in the future.

Idk what the cause of the problem is, but it's definitely not a caulking problem.

Happened to me when I had missing materials on patches that were on doorways

So, setting umbra_target_only 1 KVP helped for models. And make umbra target worked for brushes as well. BUT now I simply can't see outdoor area through the window, but can see it when I use noclip and go through window. Guess I need to wait till Treyarch will fix it. This is beta after all.

I resolved it!
So for brushes you need to 1. Right click on it. 2.Choose Make Umbra Target.
For Models: Add KVP umbra_target_only to 1.

Thanks to Cornrow Wallace from mappersunited.com.