I love that flower patch!
If you go to the Ruins (the old, falling down church), there should be a statue of a fat little lady (there are three, each a different size in a semi-circle of stones). If you find her, and walk straight in the direction she is looking, you'll find your flower patch again!