You can delete budgets in the details view of a budget.

For iOS:

• open the budget's details and scrol down > then click on 'Remove from list'
• or you just do a left-swipe in the budget's overview on the concerned budget 

For macOS: 

• Click on the concernedbudget to open the details > chosse 'delete'