This is a question that I am often asked by clients.
The decision to come off medication should be your decision and you should speak to the person who prescribed it to you.
Medication for depression/anxiety can be addictive and should not be stopped immediately; often medical professionals will suggest that you taper off your medication gradually under their supervision.
If you have tried to come off medication before but have found that you have become more anxious or depressed after reducing the dose, then do not let that stop you trying a second time
Certain medications can be addictive and may require a longer tapering off period.
From a psychotherapists point of view, I would suggest that you must be in a good place mentally before reducing your medication. If you are going through any significant change or period of stress, it may be wise to wait until you are through the difficult time before stopping medication, unless you have a specific reason to do so.