Sleeping with an ex- help

I was with my ex for two years, we adored eachother but we split when i had a miscarriage and i met someone else. We started talking again in march when i rung him. I was really upset and he came straight over nothing happened that night he just cuddled me all night but eventually we started sleeping together and it fizzled out after a couyple of months, since then i cant stop thinking about him. We met recently and we slept together. it was a really nice night we spoke about our relationship and he said if we ever got back together he wouldnt trust me at first and his barriers would be up. I told him i completely understood this. He suggested meeting up again and i have spoken to him since but he has mentioned nothing about meeting again. I dont know where to go from here i think the world of him and im scared to which makes me back away.

Should i leave it completely?