Make a button move with a cell in Excel

I am trying to create a spreadsheet in which I need a button in each row to stamp the time in the cell next to it and then sort the rows in order of time. My problem is that the buttons do not move. E.g. Button in cell B1 changes time in cell A1 and button in cell B2 changes time in cell A2 and for this example lets say A2 has a lower time than A1 so when sorted A1 and A2 effectively swap data. Now the button in B2 changes the time in cell A1.

Been trying to figure this out for hours, any help would be greatly appreciated.