The dual group of a spherical variety

Let $X$ be a spherical variety for a connected reductive group $G$. Work of Gaitsgory–Nadler strongly suggests that the Langlands dual group $G^\vee$ of $G$ has a subgroup whose Weyl group is the little Weyl group of $X$. Sakellaridis–Venkatesh defined a refined dual group $G^\vee_X$ and verified in many cases that there exists an isogeny $\varphi$ from $G^\vee_X$ to $G^\vee$. In this paper, we establish the existence of $\varphi$ in full generality. Our approach is purely combinatorial and works (despite the title) for arbitrary $G$-varieties.