Yes, both sides were chunkloaded. I tried to unload and reload but the problem persisted.
EDIT:
Ok i think I got it to work know now. As you can see the thermometers read the temp correctly and only one 1 redston signal is able to go through. I will let you know if I get some strange behaviour again
EDIT 2 : No, there is some bug here for sure. My setup is randomly stopping working WITHOUT touching it. It happens after a relog and randomly start to work again if you relog again few times.
Could it be related to the first time the portal loads?
One strange thing is that the thermomethesr show the temp slowly going down and not going directly to ambient temp when it lost the connection to the other portal 
.