Tuffy Leemans.
Here is Leeman's bio from Giants.com.
Leemans played for the Giants from 1936-43. He was a second-round draft choice in the first-ever NFL draft in 1936 out of George Washington. Leemans was personally scouted by Giants co-owner Wellington Mara. Mara sent a telegram setting up a meeting with Leemans to convince him to accept a career in the NFL with the Giants.
Leemans was the 1936 College Football All-Star game MVP. He led NFL rushers as a rookie in 1936 with 830 yards on 206 attempts. Leemans was named All-NFL in 1936 and 1939. An aggressive, dedicated do-everything team leader, he totaled 3,142 yards rushing, 2,324 yards passing, 442 yards receiving and threw 16 touchdown passes during his career.
The No. 4 has been retired by the Giants. I can't find reference to another player who has worn the number for the Giants -- before or after Leemans.