Firstly consider
Choosing defences to 1.e4 and 1.d4 that complement each other. Example: if you play the French against 1.e4 then you might consider answering 1.d4 with 1... e6. This way you retain the right to play any of the Queen's Gambit, the Nimzo/Queen's Indian, Classical or Stonewall Dutch, or even Modern Benoni while offering white the option to transpose back to the French with 2. e4.
But
Be aware that certain defences, like the French Winawer, the Alekhine, or lines of the Closed Spanish (where, as per the Larsen witticism, both sides stand worse) can produce such idiosyncratic positions as to be excellent personal specialties.
Recognise that
Fashion is a factor in chess. Opening choices with black have often reflected the preferences of the world champion or dominant player of the day. When that was Bobby Fischer, many players started out with the Najdorf Sicilian and the King's Indian Defence. When it was Garry Kasparov, many players likewise started out with the Najdorf Sicilian and the King's Indian defence. Only to ultimately decide those openings were too complicated for their relatively low level of play. By contrast, Magnus Carlsen has shown that, in the age of the chess engine, many openings considered second or third rate are nevertheless playable. Do not be put off playing a defence as lethal at the amateur level as the Benko Gambit on the grounds than someone rated 2700 considers it refuted.
But nevertheless
Some defences are better than others. If you have any ambition, resist the temptation to play rubbish gambits. Spend time learning how to distinguish between good, bad, and all points in between. Learn why the Hennig-Schara gambit is good and try it. Learn why the Englund gambit is bad and how white refutes it.
You probably already know that
Generally, players tend to have a favourite defence to 1. e4 but struggle to identify a defence against 1. d4 they are similarly enthusiastic about. Investigate the various defences to 1.d4 until you have a favourite
Then
strive to know as much about it as possible.
Relax,
Contra received wisdom, certain universal systems like the Stonewall Dutch or the Tarrasch defence, which can be played in some form against 1.d4, 1.c4, or 1.Nf3, are viable choices. Not only do they reduce your overall workload, in the case of the Tarrasch, for example, it teaches you about playing with an isolated queen's pawn: lessons that are transferable to other more respected openings. Openings, like childhood toys, are not necessarily for life. You might grow out of them and either replace or neglect them in favour of what you perceive to be more adult choices as you mature as a chess player.
But
Again, as with your white repertoire, the more positions you master, the stronger a player you become. Answering 1. c4 with 1... e5 is considered better than playing a version of a Dutch after say 1.c4 f5 2. Nf3 e6 3. g3 d5 4. Bg2 c6 5. d3 or Tarrasch after 1. Nf3 e6 2. g3 d5 3. c4 c5 4. cd ed 5. d4 Nc6 6. 0-0 Nf6 7. 0-0 Be7 8. dc Bxc5 9. Qc2.
Remember!
Listen to the advice of much stronger players. If they tell you that a line you are playing will not win you many games, you should bear that in mind.
But
Don't forget to think for yourself. Personal taste in openings and the positions resulting should not be denied. Chess should be fun, so ultimately pick an opening you enjoy playing. Study it. Study games in it. Learn the ideas. Play it. Test it. Enjoy.