Solved by CSS Scroll State Queries
Unlike the scrollable scroll-state queries, scrolled remembers the last direction you scrolled into, which you can use to build “hidey bars”: when scrolling down (or having scrolled down), the hidey bar hides itself. When then scrolling back up, the hidey bar reveals itself.