Using agar agar can be tricky. If you are not confident that it has melted well and there may be bites, just sieve the mixture. Alternatively, you could pour the mixture into a blender and just pulse it 2-3 times. But it you like the bites, then leave as it is!!!
I didn't add sugar, because the condensed milk had it all. However, if it is less sugary, you can add sugar to it. Similar with the coffee taste, you want it more - simple, add more coffee!!! :)