Upon entry into Symex mode, point is positioned at the start of the currently selected symex. If you’d like to preserve point at its original location, instead, so that entry into Symex mode is “idempotent,” then add this to the :custom section (not the :config section) of your use-package form:
(symex-preserve-point-on-entry-p t)