LAUSR

This paper is dedicated to studying the following elliptic system of Hamiltonian type: $$\begin{cases}-\varepsilon^2\Delta{u}+u+V(x)v=Q(x)F_v{(u,v)}, & x \in \mathbb{R}^N,\\-\varepsilon^2\Delta{v}+v+V(x)u=Q(x)F_u{(u,v)}, & x \in \mathbb{R}^N,\\|u(x)|+|v(x)|\rightarrow0, & as |x|\rightarrow \infty\end{cases}$$ { − ε 2 Δ u + u + V ( x ) v = Q ( x ) F v ( u , v ) , x ∈ R N , − ε 2 Δ v + v + V ( x ) u = Q ( x ) F u ( u , v ) , x ∈ R N , | u ( x ) | + | v ( x ) | → 0 , a s | x | → ∞ where N ⩾ 3, V , $$Q\in\mathcal{C}(\mathbb{R}^N,\mathbb{R})$$ Q ∈ C ( R N , R ) , V ( x ) is allowed to be sign-changing and inf Q > 0, and $$F\in\mathcal{C}^1(\mathbb{R}^2,\mathbb{R})$$ F ∈ C 1 ( R 2 , R ) is superquadratic at both 0 and infinity but subcritical. Instead of the reduction approach used in Ding et al. (2014), we develop a more direct approach—non-Nehari manifold approach to obtain stronger conclusions but under weaker assumptions than those in Ding et al. (2014). We can find an ε 0 > 0 which is determined by terms of N , V , Q and F , and then we prove the existence of a ground state solution of Nehari-Pankov type to the coupled system for all ε ∈ (0, ε 0 ].