Having bowled out West Indies for 205, New Zealand posted 278 before being all out in the final session on day two at Basin Reserve.
West Indies wobbled in their second innings as well. Brandon King was unbeaten on 15 at stumps with Kavem Hodge on three at the other end with the tourists still 41 behind.
New Zealand fast bowler Blair Tickner did not bat in the first innings, and will not bowl or field in the match having dislocated his shoulder while diving to stop a boundary on Wednesday.
"I think there might be a couple of guys that are gutted to get the start and not quite kick on, but I think 80-90 lead and we got them two down already, so I think I'll be happy with that," New Zealand bowler Jacob Duffy said.
Earlier, New Zealand captain Tom Latham lost his off stump to Kemar Roach after the hosts resumed on 22 but Conway (60) and Kane Williamson steadied the ship with a 67-run stand.
Anderson Phillip (3-70) clean bowled Williamson for 37 and Roach removed Rachin Ravindra for five to put the pressure back on the hosts.
Hay, however, looked keen to capitalise on his Test debut replacing Tom Blundell, who suffered a hamstring injury in the drawn series opener.
Soon after bringing up his 50, Hay hit Ojay Shields for back-to-back fours before pulling him straight to Roach at square leg. Hay's 61 included nine fours and a six.
West Indies got a double-reprieve early in their second innings when opener Brandon King drove Duffy, who spilled the return catch and the deflected ball narrowly missed the stumps at the non-striker's end with John Campbell well out of the crease.
A much-relieved Campbell (14) celebrated by hitting Foulkes for three fours in the next over before shouldering arms to a delivery from debutant Michael Rae that crashed onto his stumps.
Anderson Phillip walked in as the nightwatchman but returned after a three-ball duck, pinged lbw by Duffy, who convinced his skipper Latham to review the original not-out decision.
