In this song, we learn more about Alexander's personality and get a first glimpse of his son.
In the context of this song, Hamilton is still battling congress to have his financial plan approved. His son, Philip, greatly admires him and is openly proud to call Hamilton his dad, which will (spoiler alert) lead to his demise later on. Eliza, on the other hand, is currently not very fond of Hamilton as he is too obsessed with work. As she tries to convince Hamilton to take a vacation upstate, Hamilton refuses by saying he has to get his plan to congress, even though Angelica would also be joining them all the way from London.
Take a Break in the musical Hamilton
Hamilton definitely wanted to join his family upstate, but as he says in the song, he would lose his job if he didn't get his job to congress, which was true. If his plan doesn't go through, he would lose power in the government for being someone who proposes unachievable things, and lose trust as well. This might not lead to his immediately layoff, but cause a series of downturns for him that would lead to him losing his job.