We address the resource allocation challenge in NextG cellular radio access networks (RAN), where resources must be shared among diverse user applications, each requiring guarantees on throughput and service regularity. Due to factors such as fluctuating channel quality, user mobility, and scalability, this problem quickly becomes combinatorial and complex. We begin by analytically modeling it as a constrained Markov decision process, drawing from control theory. By leveraging the structural properties of this model, we decompose the problem into a more manageable form, allowing for the independent computation of relative priorities for each user. By allocating resources in descending order of these priorities, we reduce the combinatorial complexity of resource allocation to a linear one. We introduce Windex, a lightweight method for training neural networks to compute these priorities, taking into account service guarantees, channel quality, and system load. Implemented on a real-time RAN Intelligent Controller (RIC), Windex enables resource allocation decisions in under 20μs per user, efficiently managing resources within each 1 ms scheduling time slot. Our approach, evaluated across standardized 3GPP service classes, demonstrates the least service violations compared to state-of-the-art systems. These results have been validated through simulations and real-world emulations using over-the-air channel traces on a 5G testbed.